Create tall format derived datasets from all seasons

[MagicMilly]

As requested by Ryan, retain tall format from traits data queried from betydb, but apply same cleaning functions as the wide derived datasets. Will be easier for him to work with using R for machine learning.

[rbartelme]

@MagicMilly could you please clarify what the wide derived datasets are? Are these the metrics calculated in your notebooks like days to flowering, flag leaf emergence, gdd, etc?

[MagicMilly]

Yes, any of the single traits like the ones you listed are wide, with one row per plot. Canopy height is a bit different, but when I originally queried in R using the traits package, it was tall format with all traits.

[rbartelme]

@MagicMilly Thanks for the clarification!!

[dlebauer]

here is the code to take the tall format trait_data.zip from dryad and put into a single tall file

https://github.com/terraref/data-publication/blob/5bef8f4b8b834c00cb24a0d96f7562976a9535bd/content/phenotypes.Rmd#L24

[MagicMilly]

Tall format tables will be uploaded to this Google Drive folder for feedback until they're ready to be shared on CyVerse. Please let me know if you cannot access.

[MagicMilly]

@rbartelme Just an update: all of the raw, tall datasets for four seasons (including MAC 4 and 6 with the additional info like lat and lon) are now in the same folder that I shared above.

• I do not think that blocking height was labeled in all four seasons / experiments - only in the MAC experiments, so I was planning on leaving those out.
• I'm assuming you are dealing with the null values on your end, but let me know if I need to do that.
• I am still working on adding GDD values because every dataset contains at least days_to_flowering as one of the traits. This is when I'll change the datetime objects and populate the other columns where possible.

[MagicMilly]

@rbartelme The raw tall formats in the tall_format_data folder have been updated with

• MAC Season 4
  • growing degree days to flowering
  • growing degree days to flag leaf emergence
• MAC Season 6
  • no update due to no days_to_ traits
• KSU
  • growing degree days to flowering
• Clemson
  • growing degree days to flowering

I matched the new data with the original where I could, but I did not modify any of the date values. Hopefully that can be more easily done in R, though it may not be needed for this particular trait.

The code I used can be found on github in the create_tall_formats notebook. Let me know if you notice any errors or have any other feedback.