Compute plot level information from UAV data

dlebauer commented 7 years ago

Description

Use polygons here: to subset calibrated UAV data to plot level data that can be inserted into BETYdb: https://terraref.ncsa.illinois.edu/bety/api/beta/sites?key=
Add methods as necessary to
- github.com/terraref/documentation
- cross reference these in https://terraref.ncsa.illinois.edu/bety/methods
Add variables as necessary to https://terraref.ncsa.illinois.edu/bety/variables/new
- first search https://betydb.org/variables; we can move existing variables as needed
Create a csv called 'traits.csv' that includes the following columns
- sitename (this is the name of the plots, format is MAC Field Scanner Field Plot *** Season 2)
- trait name (e.g. NDVI etc. once you have)
- n (optional, can be 'effective n' if appropriate)
- SD (optional, if available, can be defined in methods documentation)
- methods.name
  Context

This can be used to insert UAV data into BETYdb.

Further Suggestions / Request for Feedback

Documentation is here: https://pecan.gitbooks.io/betydbdoc-dataentry/content/trait_insertion_api.html
- note that the most of the other metadata (site, cultivar, treatments, etc) is already in the database or will be entered separately.
Data can be added to the traits table using either the csv or json format (@gsrohde can help with this)
I (@dlebauer) can help show you how to define the variables and methods

Mamatemenrs commented 7 years ago

Sounds great ! we'll work on this once we have the finalized shapefile and orthomosaicked UAV data.

dlebauer commented 7 years ago

@Mamatemenrs and @remotesensinglab could you please create a separate issue defining the deliverables for Rick and his employee what they need to provide (e.g. georeferenced geotiff files for each sensor x flight combination?

dlebauer commented 7 years ago

Is this waiting on #188 ?

remotesensinglab commented 7 years ago

@dlebauer: My understanding on this issue is that Rick Ward wanted to do all processing and upload the summary statistics to the server. If this is not the case, please advise, we can do it. Maybe, we should coordinate a meeting with Rick and see what needs to get done and what he prefers in terms of our involvement. This is an interesting area for my research and pubs as well, so good to talk to Rick on this.

ghost commented 7 years ago

@rickw-ward - are you planning to do the processing and uploads? Do we need a teleconference to discuss?

ghost commented 7 years ago

Rick has confirmed that he will do processing and upload

NewcombMaria commented 6 years ago

Update August 8: the 2016 Maricopa UAV data needs to be re-run in Pix4D to correct some geo-referencing alignment issues before accurate orthomosaics for each flight are ready to go to ROGER. This is in process and Rick thinks that corrected sorghum season 2 orthomosaics for 4 spectral flights and 1 thermal flight can be uploaded to a Google directory later today or tomorrow.

Plot-level summary statistics are still the end-goal, but the first near-future goal is to complete the full-field orthomosaics that can go to ROGER then to Clowder.

ghost commented 6 years ago

@dlebauer please follow up

NewcombMaria commented 6 years ago

Rick Ward and Sara Harders have re-processed 2016 UAV data in Pix4D and to my understanding orthomosaics are ready. Is there any standardized way of naming UAV orthomosaics and associated metadata info? Did KSU establish a method?

dlebauer commented 6 years ago

Metadata

Make sure this page is up to date: https://terraref.gitbooks.io/terraref-documentation/content/user/protocols-UAV.html

(it can be edited here: https://github.com/terraref/documentation/edit/master/user/protocols-UAV.md)

File Organization

The most important is that everything is organized consistently. It is easy to reorganize and rename as long as the same basic conventions are kept across dates / sensors / data products.

Good enough

organize into directories like

{level}/{sensor}/{date}/{filename}

where

{level} is optional, the important part is that the raw data should be separate from the derived datasets
{sensor} can be the sensor name (e.g. rededge) or data product (e.g. ndvi)
{date} should be in the format YYYY-MM-DD
{filename} can be whatever you have, as long as it is consistent

Not Good Enough

Hopefully it is clear where the current organization can be improved:

The convention we are using for the Field Scanner

This is what we are using for the Field Scanner data. The following would be consistent with our approach to the Field Scanner data, which is defined in https://github.com/terraref/terrautils/blob/master/terrautils/sensors.py (but needs to be documented somewhere, consider this the first draft!)

The format of the directory and filenames is:

directory: {Level}/{sensor}/{date}/{timestamp}/',
filename: {sensor}_{L}_{station}_{timestamp}_{dataproduct}{opts}.{ext}'

where:

{Level} is one of "raw_data", "Level_1", Level_2, ...
{sensor} is the name of the sensor or data product
{date} should be in YYYY-MM-DD format
{timestamp} in the format YYYY-MM-DD_HH-MM-SS. This is optional
{L} is level, but shortened to L1, L2, ...
{station} would be one of ua-mac, ksu, or ...
{data product}
{opts} might be left or right

In the case of the RGB sensor, one of the fullfield stitched files is called:

Level_1/fullfield/2017-05-29/fullfield_L1_ua-mac_2017-05-29_rgb.tif

Level_1 includes the georeferenced stitched images / point clouds. Level 2 would be any further derived products, for example, an NDVI map computed from a Level 1 product that contains Red and Near IR bands or a height map derived from a Level 1 point cloud.

gpmorrisksu commented 6 years ago

Hi David,

Can you provide a link to the Gdrive folder in the screen shot?

Thanks, Geoff

Geoff Morris, Assistant Professor Department of Agronomy | Kansas State University 3004 Throckmorton Plant Science Center | Manhattan KS, 66506 E-mail: gpmorris@k-state.edumailto:gpmorris@k-state.edu | Web: http://www.morrislab.org Office: 785-532-3397 | Cell: 312-909-1330 | Skype/Google ID: morris.geoff.p

On Sep 21, 2017, at 4:27 PM, David LeBauer notifications@github.com<mailto:notifications@github.com> wrote:

Metadata

Make sure this page is up to date: https://terraref.gitbooks.io/terraref-documentation/content/user/protocols-UAV.html

(it can be edited here: https://github.com/terraref/documentation/edit/master/user/protocols-UAV.md)

File Organization

The most important is that everything is organized consistently. It is easy to reorganize and rename as long as the same basic conventions are kept across dates / sensors / data products.

Good enough

organize into directories like

{level}/{sensor}/{date}/{filename}

where

{level} is optional, the important part is that the raw data should be separate from the derived datasets
{sensor} can be the sensor name (e.g. rededge) or data product (e.g. ndvi)
{date} should be in the format YYYY-MM-DD
{filename} can be whatever you have, as long as it is consistent

Not Good Enough

Hopefully it is clear where the current organization can be improved:

[screen shot 2017-09-21 at 4 25 04 pm]https://user-images.githubusercontent.com/464871/30719555-918d899a-9ee9-11e7-9843-8a8a415465af.png

The convention we are using for the Field Scanner

This is what we are using for the Field Scanner data. The following would be consistent with our approach to the Field Scanner data, which is defined in https://github.com/terraref/terrautils/blob/master/terrautils/sensors.py (but needs to be documented somewhere, consider this the first draft!)

The format of the directory and filenames is:

directory: {Level}/{sensor}/{date}/{timestamp}/', filename: {sensor}{L}{station}{timestamp}{dataproduct}{opts}.{ext}'

where:

{Level} is one of "raw_data", "Level_1", Level_2, ...
{sensor} is the name of the sensor or data product
{date} should be in YYYY-MM-DD format
{timestamp} in the format YYYY-MM-DD_HH-MM-SS. This is optional
{L} is level, but shortened to L1, L2, ...
{station} would be one of ua-mac, ksu, or ...
{data product}
{opts} might be left or right

In the case of the RGB sensor, one of the fullfield stitched files is called:

Level_1/fullfield/2017-05-29/fullfield_L1_ua-mac_2017-05-29_rgb.tif

Level_1 includes the georeferenced stitched images / point clouds. Level 2 would be any further derived products, for example, an NDVI map computed from a Level 1 product that contains Red and Near IR bands or a height map derived from a Level 1 point cloud.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/terraref/computing-pipeline/issues/186#issuecomment-331286922, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATLjZMwsvc36eoXUJD-WOie_BjTN-eqtks5sktSsgaJpZM4KXSFZ.

dlebauer commented 6 years ago

It's not a public folder so I sent you an invite. These data are Rick's so please don't share Beyond the team without his permission.

rickw-ward commented 6 years ago

@dlebauer please extract shape files I can use in Qgis to generate plot level means for MAC Field Scanner Seasons 1 and 2.

dlebauer commented 6 years ago

exporting shapefiles covered in this issue: https://github.com/terraref/computing-pipeline/issues/361; the files I generated are here: mac_plots.zip

terraref / computing-pipeline