Closed max-zilla closed 7 years ago
I don't have much experience here but I found this Quora page which suggests using an ImageJ plugin - here's a list of cameras the plugin supports (Find "Supported Cameras"). Perhaps a place to start if there isn't already a solution at hand.
From @nfahlgren on February 15, 2016 20:24
The code we have used for converting images in RAW format from our system is here: https://github.com/danforthcenter/PhenoFront/tree/master/src/main/java/src/ddpsc/results. The Bayer2Rgb.java is probably the most relevant for example.
From @solmazhajmohammadi on February 15, 2016 20:28
Currently I am using JSON-input parser MATLAB toolbox to parse the meta-data files to find the format. Then simply you can open the file and save it in specific filetype.
From @yanliu-chn on February 15, 2016 22:12
From what I know in general: Droid from National Archives: http://digital-preservation.github.io/droid/ is a metadata identification and recording tool.
Commercial software https://www.safe.com/ provides conversion support for a lot of file formats, esp. GIS data.
From @robkooper on April 21, 2016 17:26
One command that used to convert some of the raw data is:
convert -size 640x480 -depth 16 gray:input_ir.bin -evaluate multiply 16 -depth 8 output.png
From @dlebauer on April 21, 2016 17:42
@robkooper thanks. That looks like a good first step, additional steps for this issue include:
convert input.png -resize 4096@ thumbnail.jpg
according to imagemagik resize docs.Would it be worth it to start with a non-georeferenced png (or TIFF) + thumbnail jpg? And / or have separate extractors for 1) converting binary file 2) generating thumbnail and 3) creating georeferenced geoTIFF?
From @robkooper on April 21, 2016 18:5
yes clowder uses imagemagick to do this. One option which makes it easier is to make an extractor that will look at the metadata for octet-stream files and convert them to an image that is attaches as preview to the file.
From @dlebauer on June 8, 2016 14:29
@TinoDornbusch
@LTBen says 'Tino finished a range of Octave scripts on the control PC such that operators on site can view raw data (Vis, FLIR, PS2, Sensors) and make simple analyses and visualizations.'
Could you please either send these to me or place them in the repository (you can copy and paste them individually here)?
From @TinoDornbusch on June 8, 2016 15:45
@dlebauer OK it is in gantry_data/Lemnatec/OctaveScripts
DO not laugh, just very simple loading procedures knitted with hot needle
From @dlebauer on June 8, 2016 16:11
@TinoDornbusch thanks! "if your code is good enough to do the job, then it is good enough to release"
From @TinoDornbusch on June 8, 2016 16:24
@dlebauer forgot to mention that Solmaz contributed to by providing the Demosaic algorithm, which is not implemented in Octave
From @rachelshekar on July 7, 2016 14:6
@abby621 can you please give an update for this issue?
From @pless on July 7, 2016 14:39
@rachelshekar @max-zilla @dlebauer @abby621 Hi, we have a new person onboarding to this project (Zongyang Li, who will shortly get access to this). While he is getting his clowder development/test environment setup, could we ask that @max-zilla integrate the demosaic script:
https://github.com/terraref/computing-pipeline/blob/master/scripts/stereoImager/demosaic.py
as a clowder extractor, and then provide that as an additional example script? The conference call on the clowder test environment was really good, but having an example to show our new guy how clowder can trigger on the appearance of 2 binary files and a json file in a folder would be helpful.
@pless @dlebauer @abby621 yes, I can prepare that extractor as an example.
@pless @abby621 I want to test a bit more, but initial version is here: https://github.com/terraref/computing-pipeline/tree/demosaic_extractor/scripts/stereoImager/extractor
Basically, you can run this extractor pointing at rabbitMQ, create a dataset and upload left/right/metadata to that dataset in Clowder pointing at same rabbitMQ, and extractor will add two demosaic'ed JPGs to the dataset.
I might clean this up slightly, but we could have a little phonecall to walk through the code if that would be helpful. fortunately it's fairly simple I think.
From @ZongyangLi on July 11, 2016 18:9
Hi, I am working on fix the GeoTIFF creation, because there is something un-regular with these GeoTIFF creations. I am trying to view images in a google map using tile coordinate. When I finish this, I will start working on a Clowder extrator.
From @ZongyangLi on July 14, 2016 21:0
@max-zilla Hi, I am trying to get my Clowder test environment setup. Here are some problems:
I think I might made a mistake in uploading files to dataset. Could you give me some additional examples to show me how could I provide inputs in Clowder.
@ZongyangLi yes, if you were able to access localhost:9000 you have things set up correctly.
That error I believe is due to the version of PyClowder you are using. Currently the pyclowder master branch does not support extractors that operate on datasets (such as the demosaic extractor). Instead you must install my branch: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse?at=refs%2Fheads%2Fbugfix%2FCATS-554-add-pyclowder-support-for-dataset
Easiest way to do this is probably to:
git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/pyclowder.git
cd pyclowder
git checkout bugfix/CATS-554-add-pyclowder-support-for-dataset
python setup.py install
So you pull down PyClowder, switch to my branch that supports datasets, install that version. Then try running demosaic and see if bug is fixed.
Soon my pyclowder branch should become part of main so this won't be necessary, but for now it is. Let me know if that helps!
From @ZongyangLi on July 14, 2016 22:28
@max-zilla Thank you! That's really helpful. Clowder starts to process when left/right/metadata are ready. But it choose an incorrect file as an input dataset. I got these output information in the terminal: "img_left: /var/folders/rw/4bn7m8x90c1_vjxlw94gxk6h0000gn/T/tmpHDehoV/1_left.bin_57880e72e4b07f52668dee40/_metadata.json
So how could I get the file path from "parameters" in "process_dataset" function?
From @ZongyangLi on July 20, 2016 15:31
@max-zilla I am still working on this problem. Could you please give me more examples that how to access all datasets since I can not run the demosaic extractor successfully.
From @ZongyangLi on July 22, 2016 18:5
This discussion has moved to #133 and the code is written. closing.
From @dlebauer on February 15, 2016 19:34
The field scanner sensors are exporting binary files (with .bin extension) along with .json files that with a field "Output Data format" that describes how the binary file was written, e.g. the FLIR camera files are
"Output data format": "Bayer GR8 3296x2472", stereo cameras are
"Output data format": "Bayer GR8 3296x2472",`Here are the datasets: https://uofi.box.com/s/c9ngkwi3xvtfr0ei5zfzgf4jya0dmrlc
The Feb 9 folder has one sample from each of the sensors. The feb 12 folder has many images from the sensors (FLIR thermal IR, Stereo Vis, and Fluorescence).
Is there a general way to parse the meta-data files to find the format, and then convert the .bin files to something that can be read by computer programs? Are there existing BrownDog converters that we can use for these files?
Update
Copied from original issue: terraref/computing-pipeline#64