Is it possible to apply the method to conventional LF image?

[jhss]

Dear Author

I read your paper with great interest.

You said that the method is used for FiMic images. I am wondering if it is possible to apply the method to conventional LF image such as an image from Lytro camera.

[freerafiki]

Thanks for the question.

Yes,

Actually in the paper I also used the method on the images from the Light-Field Microscopy with a Consumer Light-Field Camera Paper arxiv pdf, which are actually captured using a modification of a Lytro Camera (first generation).

The code now uses a read_input_image_v2 (at line 117 in the main.m file) method (defined in the corresponding file in the READ Folder), which assumes an image of the FiMiC type (so like a raw image of the sensor with all the lenslet images in a grid) and uses the parameters from our setup. The output of the functions here the code is commented, but I agree is not the best explanation are three:

• the focal stack (fs): the stack of images at different depths in a multi-dimensional array. It's a 4D array, so [h, w, c, l] with h=height, w=width, c=channel, l=layer (in the code is written [h,w,n], but images are RGB, so it should be 4D. Sorry for the confusion. here you can see the creation of the 4D array.
• a map which tells you the plane position (so the i-th image is focused at a distance contained in the i-th element of the map), for example if map[3] = 300 the image at index 3 in the array (fs[:,:,:,3]) will be focused at 300 (nanometer). In the Lytro case you would have to leave this information (maybe just values from A to B) or calculate through the calibration. We had this because of the setup of the microscope. This information is used to convert the disparity map into metrical information. But if you just need disparity (so relative values and not metrical values) you do not need this information.
• the elemental images (ei): these are the sub-aperture (or viewpoint) images. So each images is contained there. So it is also a big 4-D array, where ei[:,:,0] is the first image ( I think top-left) and ei[:,:,1] is the second and so on.

You could change the function by writing your own function read_input_image_lytro which returns the focal stack and the elemental images (as said, you do not necessarily need the map). It should not be too difficult, the Lytro Toolbox offers pre-made methods to extract the elemental image (or sub-aperture images, or viewpoint images, as you may wish to call them) and also filter to create the focal stack. The more planes you have, the better will be the results. Also the denser the planes are (the lower the distance between two planes), the better the results.

However, most of the parameters are tuned with respect to microscopy images, which are quite different from natural images. Since I do not know on which kind of images you want to apply the method to, I need to warn that you may need to fine-tune the parameters. I will be available for answering if you have specific questions.

How familiar are you with coding and Lytro images? I can imagine you can write your own function to read the images or also adapt the one that is already used and it should be doable. I can be here for helping, but I do not have the time at the moment to implement the whole function on my own.

If you create a new method for reading that works good (I will have to review the code), you are very welcome to create a pull-request to add your code to the repository. New contributions are welcomed.

I hope this helps.

[jhss]

I'm sorry for late reply.

Thank you for the detailed answer.

I have used the Lytro Toolbox to obtain sub-aperture images of Lytro images, but I'm not familiar with it. I can just read the manual, follow the guideline, and implement a small fraction of code.

I want to apply the method to Lytro Illum Light Field Dataset. As you said that all I need to implement is read_input_image_v2, I will try to implement the counterpart of Lytro Camera. If I have a problem while implementing the code, I will ask you again.

Thanks again for your sincere effort to answer the question.

[sky-fly97]

I'm sorry for late reply.

Thank you for the detailed answer.

I have used the Lytro Toolbox to obtain sub-aperture images of Lytro images, but I'm not familiar with it. I can just read the manual, follow the guideline, and implement a small fraction of code.

I want to apply the method to Lytro Illum Light Field Dataset. As you said that all I need to implement is read_input_image_v2, I will try to implement the counterpart of Lytro Camera. If I have a problem while implementing the code, I will ask you again.

Thanks again for your sincere effort to answer the question.

Hi, sorry to bother you, have you successfully used the light field data in Lytro format? I currently have traditional LFM data as (u,v,h,w,c) 5D mat，and would like to use this method to recover the structure.

[freerafiki]

Hi @sky-fly97 , yes, I used the LF data in the input format.

I have to say, I have not been actively working on this for a couple of years, so you would most likely need to adapt some parts of the code. I used the Lytro first-generation images from the 2013 paper on LF Microscopy with Lytro, L. Mignard-Debise and I. Ihrke, Light-Field Microscopy with a Consumer Light-Field Camera, 2015 International Conference on 3D Vision, 2015, pp. 335-343, doi: 10.1109/3DV.2015.45.

This code was in a private repository because it was never ready (never polished, never commented), but since you may benefit from it, I made it public. You can find it here. As the note says, please be aware that the code is old and may not work properly. You may want to check the code from the READ folder there and use it to read Lytro data into the format you need for the code you are using, then it should work.

Sorry about this, but I did not work further on the topic and I do not work on Matlab now, so I do not manage to fix it. If you add the reading of Lytro images and make it work, I would be very grateful if you submit a pull request and I will integrate your code here.

Thanks.