search

terraref / computing-pipeline

Pipeline to Extract Plant Phenotypes from Reference Data
BSD 3-Clause "New" or "Revised" License
24 stars 13 forks source link

hdf/netcdf extractor for clowder #145

Closed robkooper closed 7 years ago

robkooper commented 8 years ago

Given the netcdf files we receive we should have an extrctor that takes the properties of the netcdf/hdf file and inserts them as metadata in clowder.

Another user in the CyberGIS group has brought this up as well as an interesting item.

Also see https://opensource.ncsa.illinois.edu/jira/browse/CATS-628

ghost commented 8 years ago

can this issue be closed?

ghost commented 7 years ago

@max-zilla and @jterstriep - can this issue be closed?

max-zilla commented 7 years ago

@rachelshekar not yet. i'll try to get this updated this week - JD and I are dealing with some pipeline things today.

max-zilla commented 7 years ago

Just updated the NCO version in dockerfile and added a 3rd output, JSON format, to netCDF extractor. Closing this issue,

dlebauer commented 7 years ago

Is the extractor inserting these metadata into clowder?

In this sample dataset https://terraref.ncsa.illinois.edu/clowder/datasets?space=57e42cd44f0cff4b58dd3eea there is only gantry metadata, but the information from the netcdf header is not available.

max-zilla commented 7 years ago

The hyperspectral netCDFs are crashing the VM when it tries to download them, since the VM only has 8GB memory.

This won't happen when Roger filesystem is mounted but i need to make sure with @jdmaloney that we are ready to do that again without filesystem errors.

max-zilla commented 7 years ago

This is redeployed and running now.