Extractors for meteorological data

[dlebauer]

Description

We have one script to process the environmental logger data.

• We need scripts to insert data from MAC, Kansas, and UIUC, into the geostreams database
• need to insert data from environmental logger into the geostreams database
• Would be good to update these files daily.

  Further Suggestions / Request for Feedback

@robkooper Can we use BrownDog / PEcAn infrastructure?

How to upload these files? They are small, not necessarily worth setting up Globus endpoint just for these, if they can be downloaded via FTP.

---

(Appended below the task list and useful information)

Tasks

• [x] Input
  • [x] Subscribe related messages (*.dataset.files.added)
  • [x] Process condition
  • [x] All input ready (24 .dat files present)
  • [x] Process not done yet (check metadata)
• [x] Process
  • [x] Record property transformation (for mapping details see https://github.com/terraref/computing-pipeline/issues/156#issuecomment-252131358)
  • [x] "TIMESTAMP" - Convert to ISO 8601 (and use as start_time and end_time)
  • [x] "RECORD" - Discard
  • [x] "BattV" - Discard
  • [x] "PTemp_C" - Discard
  • [x] "AirTC" - Convert to Kelvin
  • [x] "RH" - Ensure unit is percent
  • [x] "Pyro" - Direct use
  • [x] "PAR_ref" - Direct use
  • [x] "WindDir" - Given WS_ms is also present, convert into eastward_wind and northward_wind.
  • [x] "WS_ms" - Ensure unit is meters per second
  • [x] "Rain_mm_Tot" - Direct use
  • [x] Assign proper time stamps
  • [x] Assign coordinates
  • "can be SW corner of Gantry … we can start with 4 digits precision (10m?) and update this later"?
  • [x] Assign stream_id
  • [x] Assign sensor_id
  • [ ] Assign sensor_name
  • [x] Add source info (dataset and file)
  • [x] Add unit and sample method
• [x] Output
  • [x] Save result (GeoStream API)
  • [x] Assert record of sensor, station and stream
  • [x] Mark input data as processed (use metadata).

    Raw data structure (.dat files)

{
   "WindDir":{
      "unit":"degrees",
      "sample_method":"Smp"
   },
   "PAR_ref":{
      "unit":"umol/s/m^2",
      "sample_method":"Smp"
   },
   "BattV":{
      "unit":"Volts",
      "sample_method":"Smp"
   },
   "TIMESTAMP":{
      "unit":"TS",
      "sample_method":""
   },
   "Rain_mm_Tot":{
      "unit":"mm",
      "sample_method":"Tot"
   },
   "Pyro":{
      "unit":"W/m^2",
      "sample_method":"Smp"
   },
   "RECORD":{
      "unit":"RN",
      "sample_method":""
   },
   "AirTC":{
      "unit":"Deg C",
      "sample_method":"Smp"
   },
   "WS_ms":{
      "unit":"meters/second",
      "sample_method":"Smp"
   },
   "RH":{
      "unit":"%",
      "sample_method":"Smp"
   },
   "PTemp_C":{
      "unit":"Deg C",
      "sample_method":"Smp"
   }
}

[max-zilla]

@Zodiase i have not seen the UIUC/Kansas datasets yet.... I put that comment there as a note during the meeting, but i think the netCDF note was not for this specific extractor but for other met data we might see.

[dlebauer]

Data flow should be raw --> geostreams --> netCDF

The raw--> netCDF developed alongside the hyperspectral extractor is a special case

The raw --> geostreams extractor may need special handling for each data source. We should open separate issues for each additional source. later we can write the geostreams to netCDF extractor and we will only need one.

[max-zilla]

@robkooper @dlebauer this extractor raises a question given our discussions yesterday. If we want to store the geostream info by plot as discussed, we'll probably want a "plot' that covers the entire field as well somehow (or a marker to the side of the field) to indicate the met data is not specific to a plot, but instead to the entire location.

I thought about adding the met datapoint to EVERY plot, but that would make the visualiztion of that metric busy and confusing I think.

Having a synthetic "full field" plot that is not in the lookup shapefile (so things are only assigned to it if we engineer them to do so) could be handy for other reasons eventually also.

[robkooper]

I agree, having a plat that is the whole site for the met data should work.

[dlebauer]

we have a plot that is the whole site that we can use.

On Thu, Jan 26, 2017 at 4:31 PM, Rob Kooper notifications@github.com wrote:

I agree, having a plat that is the whole site for the met data should work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/156#issuecomment-275534814, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX5ydvJ2aAlQnh9os_MIECe-eIBs-Kks5rWR66gaJpZM4JtN81 .

[max-zilla]

This is complete and running with geostream component & 5-minute aggregations.

We can use #173 to discuss the netCDF portion.