Write code to convert binary image files to full resolution geoTIFF + jpg preview

[dlebauer]

The field scanner sensors are exporting binary files (with .bin extension) along with .json files that with a field "Output Data format" that describes how the binary file was written, e.g. the FLIR camera files are "Output data format": "Bayer GR8 3296x2472", stereo cameras are"Output data format": "Bayer GR8 3296x2472",`

Here are the datasets: https://uofi.box.com/s/c9ngkwi3xvtfr0ei5zfzgf4jya0dmrlc

The Feb 9 folder has one sample from each of the sensors. The feb 12 folder has many images from the sensors (FLIR thermal IR, Stereo Vis, and Fluorescence).

Is there a general way to parse the meta-data files to find the format, and then convert the .bin files to something that can be read by computer programs? Are there existing BrownDog converters that we can use for these files?

• [x] @pless will write scripts that read information from metadata.json and convert these files to a lossless TIFF + jpg thumbnail along with a polygon describing the field of view.
  • scripts will take input and output folders as arguments. We will work
• [x] At March 14 meeting in St Louis we can work with him to integrate these into Clowder.

  Update

• We have octave / matlab code to convert PS2, NDVI, FLIR from Lemnatec in scripts/
• next step: add extractors for these

[max-zilla]

I don't have much experience here but I found this Quora page which suggests using an ImageJ plugin - here's a list of cameras the plugin supports (Find "Supported Cameras"). Perhaps a place to start if there isn't already a solution at hand.

[nfahlgren]

The code we have used for converting images in RAW format from our system is here: https://github.com/danforthcenter/PhenoFront/tree/master/src/main/java/src/ddpsc/results. The Bayer2Rgb.java is probably the most relevant for example.

[solmazhajmohammadi]

Currently I am using JSON-input parser MATLAB toolbox to parse the meta-data files to find the format. Then simply you can open the file and save it in specific filetype.

[yanliu-chn]

From what I know in general: Droid from National Archives: http://digital-preservation.github.io/droid/ is a metadata identification and recording tool.

Commercial software https://www.safe.com/ provides conversion support for a lot of file formats, esp. GIS data.

[robkooper]

One command that used to convert some of the raw data is:

convert -size 640x480 -depth 16 gray:input_ir.bin -evaluate multiply 16 -depth 8 output.png

[dlebauer]

@robkooper thanks. That looks like a good first step, additional steps for this issue include:

1. creating geoTIFF.
   • presumably requires subsequent step with gdal or similar
   • also requires knowledge of image extent
2. thumbnail generation:
   • Does Clowder already have a thumbnail extractor that we can use?
   • Or, what resolution or max pixel count do you recommend? to handle sensors with different dimensions, we can set max pixel count as convert input.png -resize 4096@ thumbnail.jpg according to imagemagik resize docs.

Would it be worth it to start with a non-georeferenced png (or TIFF) + thumbnail jpg? And / or have separate extractors for 1) converting binary file 2) generating thumbnail and 3) creating georeferenced geoTIFF?

[robkooper]

yes clowder uses imagemagick to do this. One option which makes it easier is to make an extractor that will look at the metadata for octet-stream files and convert them to an image that is attaches as preview to the file.

[dlebauer]

@TinoDornbusch

@LTBen says 'Tino finished a range of Octave scripts on the control PC such that operators on site can view raw data (Vis, FLIR, PS2, Sensors) and make simple analyses and visualizations.'

Could you please either send these to me or place them in the repository (you can copy and paste them individually here)?

[TinoDornbusch]

@dlebauer OK it is in gantry_data/Lemnatec/OctaveScripts

DO not laugh, just very simple loading procedures knitted with hot needle

[dlebauer]

@TinoDornbusch thanks! "if your code is good enough to do the job, then it is good enough to release"

[TinoDornbusch]

@dlebauer forgot to mention that Solmaz contributed to by providing the Demosaic algorithm, which is not implemented in Octave

[ghost]

@abby621 can you please give an update for this issue?

[pless]

@rachelshekar @max-zilla @dlebauer @abby621 Hi, we have a new person onboarding to this project (Zongyang Li, who will shortly get access to this). While he is getting his clowder development/test environment setup, could we ask that @max-zilla integrate the demosaic script:

https://github.com/terraref/computing-pipeline/blob/master/scripts/stereoImager/demosaic.py

as a clowder extractor, and then provide that as an additional example script? The conference call on the clowder test environment was really good, but having an example to show our new guy how clowder can trigger on the appearance of 2 binary files and a json file in a folder would be helpful.

[max-zilla]

@pless @dlebauer @abby621 yes, I can prepare that extractor as an example.

[max-zilla]

@pless @abby621 I want to test a bit more, but initial version is here: https://github.com/terraref/computing-pipeline/tree/demosaic_extractor/scripts/stereoImager/extractor

• the demosaic.py script is unchanged from what is in /scripts, i just put it in extractor folder for simpler importing.
• listens on files being added to datasets and checks for metadata, _left.bin, _right.bin. the script expects only one set of left/right/metadata per dataset for now
• when it writes the output JPGs to the dataset and writes COMPLETED as extractor metadata when done, so the extractor won't regenerate the JPGs if more files are added to dataset
• extractor will attempt to check for metadata in 2 ways: attached to dataset as metadata (eventually we can use this) and for short term, as a metadata.json file in the dataset

Basically, you can run this extractor pointing at rabbitMQ, create a dataset and upload left/right/metadata to that dataset in Clowder pointing at same rabbitMQ, and extractor will add two demosaic'ed JPGs to the dataset.

I might clean this up slightly, but we could have a little phonecall to walk through the code if that would be helpful. fortunately it's fairly simple I think.

[ZongyangLi]

Hi, I am working on fix the GeoTIFF creation, because there is something un-regular with these GeoTIFF creations. I am trying to view images in a google map using tile coordinate. When I finish this, I will start working on a Clowder extrator.

[ZongyangLi]

@max-zilla Hi, I am trying to get my Clowder test environment setup. Here are some problems:

1. I try to get my machine ip by running "docker-machine ip" in Docker Quickstart Terminal, but I get this error message: "Error: No machine name(s) specified and no "default machine exists". So I simply use "localhost:9000" in my browser, then complete the sign up. Is that Okay for my test environment?
2. I have run that "wordcount" demo successfully, but when I try "terra.demosaic.py", I got this error message at the time I upload my left/right/metadata:"ERROR: pyclowder.extractors -[5787ec39e4b07f5213e28fc8] error processing Trace back(most recent call last): File "....../pyclowder/extractors.py", line 93, in on_message filename=jbody['filename'] KeyError: 'filename'".

I think I might made a mistake in uploading files to dataset. Could you give me some additional examples to show me how could I provide inputs in Clowder.

[max-zilla]

@ZongyangLi yes, if you were able to access localhost:9000 you have things set up correctly.

That error I believe is due to the version of PyClowder you are using. Currently the pyclowder master branch does not support extractors that operate on datasets (such as the demosaic extractor). Instead you must install my branch: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse?at=refs%2Fheads%2Fbugfix%2FCATS-554-add-pyclowder-support-for-dataset

Easiest way to do this is probably to:

git clone https://opensource.ncsa.illinois.edu/bitbucket/scm/cats/pyclowder.git
cd pyclowder
git checkout bugfix/CATS-554-add-pyclowder-support-for-dataset
python setup.py install

So you pull down PyClowder, switch to my branch that supports datasets, install that version. Then try running demosaic and see if bug is fixed.

Soon my pyclowder branch should become part of main so this won't be necessary, but for now it is. Let me know if that helps!

[ZongyangLi]

@max-zilla Thank you! That's really helpful. Clowder starts to process when left/right/metadata are ready. But it choose an incorrect file as an input dataset. I got these output information in the terminal: "img_left: /var/folders/rw/4bn7m8x90c1_vjxlw94gxk6h0000gn/T/tmpHDehoV/1_left.bin_57880e72e4b07f52668dee40/_metadata.json

So how could I get the file path from "parameters" in "process_dataset" function?

[ZongyangLi]

@max-zilla I am still working on this problem. Could you please give me more examples that how to access all datasets since I can not run the demosaic extractor successfully.

[ZongyangLi]

133

[max-zilla]

This discussion has moved to #133 and the code is written. closing.

[max-zilla]

This issue was moved to terraref/extractors-stereo-rgb#1