Closed robkooper closed 7 years ago
Import is still in progress. The flirIrCamera is currently uploading and has been since Saturday afternoon, but this is the most complex/largest collection to recreate - this is one of the longest file lists, and each flirIr dataset only has a single file in it so there are more datasets created than, e.g. ps2Top which has a long file list but 100 files per dataset so it processes much more quickly.
RAW DATA
EVALUATE
LEVEL 1 DATA
@robkooper can any data be recovered from the database? For example, the UAV / Tractor data? Or the sample data that Max had prepared?
Repo IDs have changed Sample data in the database may be able to be restored
@robkooper we should look at:
...to see if we can recover them before trying to rebuild.
@craig-willis @dlebauer are eager to see if this is possible.
Space IDs from my local backup of Terra production:
collections:
{ "_id" : ObjectId("58035fa34f0c4a438cbb53dc"), "name" : "Device and Sensor Information", "description" : "Collection containing information about the LemnaTec Scanalyzer Field platform devices, sensors, and calibration targets. Each device is represented by a dataset containing programmatically-accessible metadata, data sheets, and calibration information." }
datasets:
{ "_id" : ObjectId("580361164f0c4a438cbb546c"), "name" : "Crop Circle", "description" : "Information about the Holland Scientific Crop Circle ACS-430 active crop canopy sensor." }
{ "_id" : ObjectId("5817870c4f0ce77b6655aecd"), "name" : "Headwall SWIR", "description" : "Information about the Headwall Photonics SWIR Hyperspectral imager." }
{ "_id" : ObjectId("581787264f0ce77b6655b125"), "name" : "Headwall VNIR", "description" : "Information about the Headwall Photonics VNIR Hyperspectral imager." }
{ "_id" : ObjectId("5817873d4f0ce77b6655b269"), "name" : "Prosilica RGB Camera", "description" : "Information about the Prosilica GT3300 8 Megapixel CCD camera. This device is used for stereo RGB imaging." }
{ "_id" : ObjectId("581787524f0ce77b6655b2c7"), "name" : "Skye NDVI", "description" : "Information about the Skye NDVI sensor. This sensor is used for nitrogen content assessment." }
{ "_id" : ObjectId("5817877a4f0ce77b6655b320"), "name" : "FLIR Thermal Camera", "description" : "Information about the FLIR A615 thermal infrared imager." }
{ "_id" : ObjectId("581787d94f0ce77b6655b819"), "name" : "Vaisala Carbocap", "description" : "Information for the Vaisala Carbocap CO2 Probe" }
{ "_id" : ObjectId("581788344f0ce77b6655bf23"), "name" : "Apogee PAR", "description" : "Information for the Apogee Instruments photosynthetically active radiation (PAR) sensor. " }
{ "_id" : ObjectId("581789524f0ce77b6655ccf9"), "name" : "Skye PRI", "description" : "Information about the Skye PRI sensor." }
{ "_id" : ObjectId("581789834f0ce77b6655cee4"), "name" : "LemnaTec PSII", "description" : "Information about the LemnaTec PSII camera. This camera is used to measure red light emitted by chlorophyll fluorescence." }
{ "_id" : ObjectId("581789af4f0ce77b6655d094"), "name" : "Fraunhofer 3D Scanner", "description" : "Information about the Fraunhofer 3D scanner. This device is used to measure 3D surface topology." }
{ "_id" : ObjectId("581789ca4f0ce77b6655d0f5"), "name" : "Ocean Optics Spectrometer", "description" : "Information about the OceanOptics spectrometer." }
{ "_id" : ObjectId("58178a744f0ce77b6655d38a"), "name" : "Thies Clima Weather Station", "description" : "Information about the Thies Clima weather station." }
{ "_id" : ObjectId("58178ee54f0ce77b66561602"), "name" : "LemnaTec Scanalyzer Field", "description" : "General information about the LemnaTec Scanalyzer Field platform. Please see individual device and sensor datasets." }
{ "_id" : ObjectId("5817c7c84f0c63f2a8ca5e6c"), "name" : "Labsphere Spectralon Diffuse Color Standards", "description" : "Information about the Labsphere Spectralon Diffuse Color standards (8 targets)" }
{ "_id" : ObjectId("5818bb544f0c63f2a8d20963"), "name" : "SphereOptics Zenith Polymer", "description" : "Information about the SphereOptics Zenith Polymer diffuse reflectance target." }
uploads:
{ "_id" : ObjectId("58178c304f0ce77b6655eb4d"), "filename" : "datasheet2012.pdf", "length" : 2449914}
{ "_id" : ObjectId("58178cfd4f0ce77b6655f4d5"), "filename" : "09001-SensorDocumentation_V4-CropCircle.pdf", "length" : 346721}
{ "_id" : ObjectId("58178cfe4f0ce77b6655f4d8"), "filename" : "cropcircle-gantry-position.jpg", "length" : 212439}
{ "_id" : ObjectId("58178df64f0ce77b665606da"), "filename" : "G4-383.384.Pixel.to.wavelength.xlsx", "length" : 72434}
{ "_id" : ObjectId("58178df74f0ce77b665606e3"), "filename" : "09001-SensorDocumentation_V4_SWIR.PDF", "length" : 354890}
{ "_id" : ObjectId("58178df74f0ce77b665606ed"), "filename" : "G4-383 SWIR Wavelength_Calibration.pdf", "length" : 105777}
{ "_id" : ObjectId("58178df84f0ce77b6656071f"), "filename" : "SWIR.pdf", "length" : 542599}
{ "_id" : ObjectId("581794924f0ce77b66564809"), "filename" : "G4-383.384.Pixel.to.wavelength.xlsx", "length" : 72434}
{ "_id" : ObjectId("581794934f0ce77b6656481e"), "filename" : "09001-SensorDocumentation_V4_vnir.PDF", "length" : 434151}
{ "_id" : ObjectId("5817949e4f0ce77b6656492b"), "filename" : "G4-384 VNIR Wavelength_Calibration.pdf", "length" : 127133}
{ "_id" : ObjectId("581794a14f0ce77b665649c8"), "filename" : "VNIR 2015.pdf", "length" : 573097}
{ "_id" : ObjectId("58178e574f0ce77b6656108f"), "filename" : "09001-SensorDocumentation_V4_Stereo.PDF", "length" : 698854}
{ "_id" : ObjectId("58178e584f0ce77b66561095"), "filename" : "Prosilica_GT_DataSheet_3300_v3.0.0_en.pdf", "length" : 409878}
{ "_id" : ObjectId("58178e5b4f0ce77b665610c7"), "filename" : "Stereocamera_Specification_V06.pdf", "length" : 1876775}
{ "_id" : ObjectId("58178d784f0ce77b6655f9b9"), "filename" : "ndvi-gantry-position.jpg", "length" : 171028}
{ "_id" : ObjectId("58178d784f0ce77b6655f9c4"), "filename" : "45951-45952.xlsx", "length" : 26626}
{ "_id" : ObjectId("58178d7a4f0ce77b6655f9ee"), "filename" : "09001-SensorDocumentation_V4-ndvi.PDF", "length" : 491839}
{ "_id" : ObjectId("58178d804f0ce77b6655fa78"), "filename" : "Skye_NDVI_PRI_Protokolle.pdf", "length" : 2147651}
{ "_id" : ObjectId("58178ddb4f0ce77b66560437"), "filename" : "flir-gantry-position.jpg", "length" : 188612}
{ "_id" : ObjectId("58178ddc4f0ce77b6656044d"), "filename" : "09001-SensorDocumentation_V4-FLIR.PDF", "length" : 349823}
{ "_id" : ObjectId("58178dde4f0ce77b66560480"), "filename" : "FLIR-SC655-Datasheet.pdf", "length" : 657530}
{ "_id" : ObjectId("58178de04f0ce77b66560491"), "filename" : "FLIR-calibration-certificate.PDF", "length" : 1546303}
{ "_id" : ObjectId("58178eb44f0ce77b665613a2"), "filename" : "Vaisala-GMP343-CO-Probe-Data-Sheet.pdf", "length" : 267557}
{ "_id" : ObjectId("58178eb54f0ce77b665613b3"), "filename" : "09001-SensorDocumentation_V4_co2sensor.PDF", "length" : 370267}
{ "_id" : ObjectId("58178eb64f0ce77b665613bd"), "filename" : "Vaisala.pdf", "length" : 1504886}
{ "_id" : ObjectId("581794464f0ce77b66564067"), "filename" : "SQ-110.and.SQ-500.Spectral.response.xlsx", "length" : 61348}
{ "_id" : ObjectId("581794464f0ce77b6656407d"), "filename" : "09001-SensorDocumentation_V4_par.PDF", "length" : 371327}
{ "_id" : ObjectId("5817942c4f0ce77b66563e8b"), "filename" : "09001-SensorDocumentation_V4_pri.pdf", "length" : 499436}
{ "_id" : ObjectId("581793a54f0ce77b66563454"), "filename" : "09001-SensorDocumentation_V4_ps2.pdf", "length" : 670982}
{ "_id" : ObjectId("581793394f0ce77b66562ff9"), "filename" : "09001-SensorDocumentation_V4_scanner3D.pdf", "length" : 546740}
{ "_id" : ObjectId("5817924b4f0ce77b66562844"), "filename" : "09001-SensorDocumentation_V4_spectrometer.PDF", "length" : 341238}
{ "_id" : ObjectId("581792694f0ce77b665628cd"), "filename" : "Ocean Optics.pdf", "length" : 7622747}
{ "_id" : ObjectId("58178f044f0ce77b665617d3"), "filename" : "Thies CLIMA SENSOR US.pdf", "length" : 275092}
{ "_id" : ObjectId("58178f414f0ce77b665618ef"), "filename" : "09001-SensorDocumentation_V4_weather.PDF", "length" : 465347}
{ "_id" : ObjectId("58178ef44f0ce77b6656165c"), "filename" : "09001-SensorDocumentation_V4_oK.PDF", "length" : 2034611}
{ "_id" : ObjectId("5817cbd44f0c63f2a8caba3f"), "filename" : "PUAA01-0715-2343.pdf", "length" : 107087}
{ "_id" : ObjectId("5817cbd44f0c63f2a8caba40"), "filename" : "RDAA01-0815-2498.pdf", "length" : 153072}
{ "_id" : ObjectId("5817cbd44f0c63f2a8caba47"), "filename" : "ORAA01-0416-4051.pdf", "length" : 106940}
{ "_id" : ObjectId("5817cbd44f0c63f2a8caba46"), "filename" : "GNAA01-0416-4066.pdf", "length" : 107215}
{ "_id" : ObjectId("5817cbd44f0c63f2a8caba4a"), "filename" : "BLAA01-0815-2506.pdf", "length" : 107186}
{ "_id" : ObjectId("5817cbd64f0c63f2a8caba67"), "filename" : "VIAA01-0715-2351.pdf", "length" : 107015}
{ "_id" : ObjectId("5817cbd64f0c63f2a8caba6c"), "filename" : "YWAA01-0416-4034.pdf", "length" : 107044}
{ "_id" : ObjectId("5817cbf24f0c63f2a8cabb4d"), "filename" : "CYAA01-0216-3605.pdf", "length" : 146999}
{ "_id" : ObjectId("5818b4374f0c63f2a8d1dbc6"), "filename" : "spectralon_color_standards_data_sheet.pdf", "length" : 490821}
{ "_id" : ObjectId("5818bbaa4f0c63f2a8d20b12"), "filename" : "SphereOptics-Diffuse-Reflectance-Standards-Targets-Materials-Zenith-Polymer.pdf", "length" : 1168327}
{ "_id" : ObjectId("5818bbb34f0c63f2a8d20b4b"), "filename" : "SphereOptics.Panel.pdf", "length" : 3034143}
flirIrCamera finally finished at 3AM, and we determined from memory patterns we should be safe to run 2 threads on this at once. cropCircle and ndviSensor running now.
cropCircle and ndviSensor still running, but I updated extractors to be more robust in recognizing when their outputs already exist on Roger. This meant I could start re-deploying extractors without waiting for the raw_data rebuild to be totally done. I've started doing so.
code updated; just need to run on VM
more work to do:
note: make sure old LAS files have been merged according to new extractor logic from #237
@max-zilla please update.
I've been updating my first comment as this has progressed, although at some points I prioritized ingesting new data so others could access before getting back to this rebuild.
priSensor is about 1/3 done, then we are 99% done. the last few sensors have a small amount of data to restore comparatively, and we might restore parts of the Level_1 data but others have been subsequently deprecated by updated extractors or workflows (e.g the plot level stuff)
The clowder instance for terraref had a catastrophic disk failure.