restore clowder instance for terraref

[robkooper]

The clowder instance for terraref had a catastrophic disk failure.

• [x] create new vm
• [x] install required software
• [x] install clowder
• [x] copy backup of spaces/users
• [ ] import existing data

[max-zilla]

Import is still in progress. The flirIrCamera is currently uploading and has been since Saturday afternoon, but this is the most complex/largest collection to recreate - this is one of the longest file lists, and each flirIr dataset only has a single file in it so there are more datasets created than, e.g. ps2Top which has a long file list but 100 files per dataset so it processes much more quickly.

RAW DATA

• [x] VNIR
• [x] SWIR
• [x] scanner3DTop
• [x] ps2Top
• [x] flirIrCamera
• [x] cropCircle
• [x] weather
• [x] Danforth
• [x] EnvironmentLogger
• [x] stereoTop
• [x] ndviSensor
• [x] co2Sensor
• [x] priSensor
• [x] lightning

EVALUATE

• [x] scanner3DEast, scanner3DWest
• [x] stereoEast, stereoWest

LEVEL 1 DATA

• [x] Danforth
• [ ] demosaic
• [x] EnvironmentLogger
• [ ] flirIrCamera
• [ ] scanner3DTop

[dlebauer]

• [x] re-create sample data space
• [x] re-import sensor metadata

[dlebauer]

@robkooper can any data be recovered from the database? For example, the UAV / Tractor data? Or the sample data that Max had prepared?

[ghost]

Repo IDs have changed Sample data in the database may be able to be restored

[max-zilla]

@robkooper we should look at:

• [x] restoring sensor metadata space & data
• [ ] restoring UAV space & data
• [x] restoring my Sample Data space

...to see if we can recover them before trying to rebuild.

• [x] make a separate issue for Mongo database duplication mode?

@craig-willis @dlebauer are eager to see if this is possible.

[max-zilla]

Space IDs from my local backup of Terra production:

• Sample Data 57e42cd44f0cff4b58dd3eea
• UAV 5769760ee4b0058ffbb11c8c
• Sensor Metadata unfortunately I don't have this one in my backup. But @robkooper might've found this one already when you sent this to @craig-willis in TERRA Hipchat:

  collections:
  { "_id" : ObjectId("58035fa34f0c4a438cbb53dc"), "name" : "Device and Sensor Information", "description" : "Collection containing information about the LemnaTec Scanalyzer Field platform devices, sensors, and calibration targets. Each device is represented by a dataset containing programmatically-accessible metadata, data sheets, and calibration information." }
  datasets:
  { "_id" : ObjectId("580361164f0c4a438cbb546c"), "name" : "Crop Circle", "description" : "Information about the Holland Scientific Crop Circle ACS-430 active crop canopy sensor." }
  { "_id" : ObjectId("5817870c4f0ce77b6655aecd"), "name" : "Headwall SWIR", "description" : "Information about the Headwall Photonics SWIR Hyperspectral imager." }
  { "_id" : ObjectId("581787264f0ce77b6655b125"), "name" : "Headwall VNIR", "description" : "Information about the Headwall Photonics VNIR Hyperspectral imager." }
  { "_id" : ObjectId("5817873d4f0ce77b6655b269"), "name" : "Prosilica RGB Camera", "description" : "Information about the Prosilica GT3300 8 Megapixel CCD camera. This device is used for stereo RGB imaging." }
  { "_id" : ObjectId("581787524f0ce77b6655b2c7"), "name" : "Skye NDVI", "description" : "Information about the Skye NDVI sensor. This sensor is used for nitrogen content assessment." }
  { "_id" : ObjectId("5817877a4f0ce77b6655b320"), "name" : "FLIR Thermal Camera", "description" : "Information about the FLIR A615 thermal infrared imager." }
  { "_id" : ObjectId("581787d94f0ce77b6655b819"), "name" : "Vaisala Carbocap", "description" : "Information for the Vaisala Carbocap CO2 Probe" }
  { "_id" : ObjectId("581788344f0ce77b6655bf23"), "name" : "Apogee PAR", "description" : "Information for the Apogee Instruments photosynthetically active radiation (PAR) sensor. " }
  { "_id" : ObjectId("581789524f0ce77b6655ccf9"), "name" : "Skye PRI", "description" : "Information about the Skye PRI  sensor." }
  { "_id" : ObjectId("581789834f0ce77b6655cee4"), "name" : "LemnaTec PSII", "description" : "Information about the LemnaTec PSII camera. This camera is used to measure red light emitted by chlorophyll fluorescence." }
  { "_id" : ObjectId("581789af4f0ce77b6655d094"), "name" : "Fraunhofer 3D Scanner", "description" : "Information about the Fraunhofer 3D scanner. This device is used to measure 3D surface topology." }
  { "_id" : ObjectId("581789ca4f0ce77b6655d0f5"), "name" : "Ocean Optics Spectrometer", "description" : "Information about the OceanOptics spectrometer." }
  { "_id" : ObjectId("58178a744f0ce77b6655d38a"), "name" : "Thies Clima Weather Station", "description" : "Information about the Thies Clima weather station." }
  { "_id" : ObjectId("58178ee54f0ce77b66561602"), "name" : "LemnaTec Scanalyzer Field", "description" : "General information about the LemnaTec Scanalyzer Field platform. Please see individual device and sensor datasets." }
  { "_id" : ObjectId("5817c7c84f0c63f2a8ca5e6c"), "name" : "Labsphere Spectralon Diffuse Color Standards", "description" : "Information about the Labsphere Spectralon Diffuse Color standards (8 targets)" }
  { "_id" : ObjectId("5818bb544f0c63f2a8d20963"), "name" : "SphereOptics Zenith Polymer", "description" : "Information about the SphereOptics Zenith Polymer diffuse reflectance target." }
  uploads:
  { "_id" : ObjectId("58178c304f0ce77b6655eb4d"), "filename" : "datasheet2012.pdf", "length" : 2449914}
  { "_id" : ObjectId("58178cfd4f0ce77b6655f4d5"), "filename" : "09001-SensorDocumentation_V4-CropCircle.pdf", "length" : 346721}
  { "_id" : ObjectId("58178cfe4f0ce77b6655f4d8"), "filename" : "cropcircle-gantry-position.jpg", "length" : 212439}
  { "_id" : ObjectId("58178df64f0ce77b665606da"), "filename" : "G4-383.384.Pixel.to.wavelength.xlsx", "length" : 72434}
  { "_id" : ObjectId("58178df74f0ce77b665606e3"), "filename" : "09001-SensorDocumentation_V4_SWIR.PDF", "length" : 354890}
  { "_id" : ObjectId("58178df74f0ce77b665606ed"), "filename" : "G4-383 SWIR Wavelength_Calibration.pdf", "length" : 105777}
  { "_id" : ObjectId("58178df84f0ce77b6656071f"), "filename" : "SWIR.pdf", "length" : 542599}
  { "_id" : ObjectId("581794924f0ce77b66564809"), "filename" : "G4-383.384.Pixel.to.wavelength.xlsx", "length" : 72434}
  { "_id" : ObjectId("581794934f0ce77b6656481e"), "filename" : "09001-SensorDocumentation_V4_vnir.PDF", "length" : 434151}
  { "_id" : ObjectId("5817949e4f0ce77b6656492b"), "filename" : "G4-384 VNIR Wavelength_Calibration.pdf", "length" : 127133}
  { "_id" : ObjectId("581794a14f0ce77b665649c8"), "filename" : "VNIR 2015.pdf", "length" : 573097}
  { "_id" : ObjectId("58178e574f0ce77b6656108f"), "filename" : "09001-SensorDocumentation_V4_Stereo.PDF", "length" : 698854}
  { "_id" : ObjectId("58178e584f0ce77b66561095"), "filename" : "Prosilica_GT_DataSheet_3300_v3.0.0_en.pdf", "length" : 409878}
  { "_id" : ObjectId("58178e5b4f0ce77b665610c7"), "filename" : "Stereocamera_Specification_V06.pdf", "length" : 1876775}
  { "_id" : ObjectId("58178d784f0ce77b6655f9b9"), "filename" : "ndvi-gantry-position.jpg", "length" : 171028}
  { "_id" : ObjectId("58178d784f0ce77b6655f9c4"), "filename" : "45951-45952.xlsx", "length" : 26626}
  { "_id" : ObjectId("58178d7a4f0ce77b6655f9ee"), "filename" : "09001-SensorDocumentation_V4-ndvi.PDF", "length" : 491839}
  { "_id" : ObjectId("58178d804f0ce77b6655fa78"), "filename" : "Skye_NDVI_PRI_Protokolle.pdf", "length" : 2147651}
  { "_id" : ObjectId("58178ddb4f0ce77b66560437"), "filename" : "flir-gantry-position.jpg", "length" : 188612}
  { "_id" : ObjectId("58178ddc4f0ce77b6656044d"), "filename" : "09001-SensorDocumentation_V4-FLIR.PDF", "length" : 349823}
  { "_id" : ObjectId("58178dde4f0ce77b66560480"), "filename" : "FLIR-SC655-Datasheet.pdf", "length" : 657530}
  { "_id" : ObjectId("58178de04f0ce77b66560491"), "filename" : "FLIR-calibration-certificate.PDF", "length" : 1546303}
  { "_id" : ObjectId("58178eb44f0ce77b665613a2"), "filename" : "Vaisala-GMP343-CO-Probe-Data-Sheet.pdf", "length" : 267557}
  { "_id" : ObjectId("58178eb54f0ce77b665613b3"), "filename" : "09001-SensorDocumentation_V4_co2sensor.PDF", "length" : 370267}
  { "_id" : ObjectId("58178eb64f0ce77b665613bd"), "filename" : "Vaisala.pdf", "length" : 1504886}
  { "_id" : ObjectId("581794464f0ce77b66564067"), "filename" : "SQ-110.and.SQ-500.Spectral.response.xlsx", "length" : 61348}
  { "_id" : ObjectId("581794464f0ce77b6656407d"), "filename" : "09001-SensorDocumentation_V4_par.PDF", "length" : 371327}
  { "_id" : ObjectId("5817942c4f0ce77b66563e8b"), "filename" : "09001-SensorDocumentation_V4_pri.pdf", "length" : 499436}
  { "_id" : ObjectId("581793a54f0ce77b66563454"), "filename" : "09001-SensorDocumentation_V4_ps2.pdf", "length" : 670982}
  { "_id" : ObjectId("581793394f0ce77b66562ff9"), "filename" : "09001-SensorDocumentation_V4_scanner3D.pdf", "length" : 546740}
  { "_id" : ObjectId("5817924b4f0ce77b66562844"), "filename" : "09001-SensorDocumentation_V4_spectrometer.PDF", "length" : 341238}
  { "_id" : ObjectId("581792694f0ce77b665628cd"), "filename" : "Ocean Optics.pdf", "length" : 7622747}
  { "_id" : ObjectId("58178f044f0ce77b665617d3"), "filename" : "Thies CLIMA SENSOR US.pdf", "length" : 275092}
  { "_id" : ObjectId("58178f414f0ce77b665618ef"), "filename" : "09001-SensorDocumentation_V4_weather.PDF", "length" : 465347}
  { "_id" : ObjectId("58178ef44f0ce77b6656165c"), "filename" : "09001-SensorDocumentation_V4_oK.PDF", "length" : 2034611}
  { "_id" : ObjectId("5817cbd44f0c63f2a8caba3f"), "filename" : "PUAA01-0715-2343.pdf", "length" : 107087}
  { "_id" : ObjectId("5817cbd44f0c63f2a8caba40"), "filename" : "RDAA01-0815-2498.pdf", "length" : 153072}
  { "_id" : ObjectId("5817cbd44f0c63f2a8caba47"), "filename" : "ORAA01-0416-4051.pdf", "length" : 106940}
  { "_id" : ObjectId("5817cbd44f0c63f2a8caba46"), "filename" : "GNAA01-0416-4066.pdf", "length" : 107215}
  { "_id" : ObjectId("5817cbd44f0c63f2a8caba4a"), "filename" : "BLAA01-0815-2506.pdf", "length" : 107186}
  { "_id" : ObjectId("5817cbd64f0c63f2a8caba67"), "filename" : "VIAA01-0715-2351.pdf", "length" : 107015}
  { "_id" : ObjectId("5817cbd64f0c63f2a8caba6c"), "filename" : "YWAA01-0416-4034.pdf", "length" : 107044}
  { "_id" : ObjectId("5817cbf24f0c63f2a8cabb4d"), "filename" : "CYAA01-0216-3605.pdf", "length" : 146999}
  { "_id" : ObjectId("5818b4374f0c63f2a8d1dbc6"), "filename" : "spectralon_color_standards_data_sheet.pdf", "length" : 490821}
  { "_id" : ObjectId("5818bbaa4f0c63f2a8d20b12"), "filename" : "SphereOptics-Diffuse-Reflectance-Standards-Targets-Materials-Zenith-Polymer.pdf", "length" : 1168327}
  { "_id" : ObjectId("5818bbb34f0c63f2a8d20b4b"), "filename" : "SphereOptics.Panel.pdf", "length" : 3034143}

[max-zilla]

flirIrCamera finally finished at 3AM, and we determined from memory patterns we should be safe to run 2 threads on this at once. cropCircle and ndviSensor running now.

[max-zilla]

cropCircle and ndviSensor still running, but I updated extractors to be more robust in recognizing when their outputs already exist on Roger. This meant I could start re-deploying extractors without waiting for the raw_data rebuild to be totally done. I've started doing so.

• [x] stereoTop.demosaic
• [x] stereoTop.canopycover
• [x] hyperspectral
• [x] netcdf

code updated; just need to run on VM

• [x] ply2las
• [x] flirIrCamera
• [x] psii
• [x] NDVI/PRI
• [x] EnvironmentLogger
• [x] plantCV

more work to do:

• [x] sensorposition (store by plot, not by site)
• [x] meteo

[max-zilla]

note: make sure old LAS files have been merged according to new extractor logic from #237

[ghost]

@max-zilla please update.

[max-zilla]

I've been updating my first comment as this has progressed, although at some points I prioritized ingesting new data so others could access before getting back to this rebuild.

priSensor is about 1/3 done, then we are 99% done. the last few sensors have a small amount of data to restore comparatively, and we might restore parts of the Level_1 data but others have been subsequently deprecated by updated extractors or workflows (e.g the plot level stuff)