Functions to convert NEON to ecocomDP

clnsmth commented 6 years ago

Develop functions to display available NEON community data products and convert selections to the ecocomDP. Resultant ecocomDPs may be written to file, but not archived in the EDI data repo.

sokole commented 6 years ago

I have some functions to pull NEON data. I'll try to post to the repo or put in a pull request (if I can push directly) today.

sokole commented 6 years ago

functions to pull NEON data in the dev branch

https://github.com/EDIorg/ecocomDP/commit/6e92c9866bcb364e145334ce51a3614f21841309

clnsmth commented 5 years ago

Conversions are specific to each NEON data product and should be added to reformat_neon. Add each conversion within an if/then statement below the section "Workflows for NEON data products". Use the data product identifier for the logical constraint (e.g. see section "Workflow for macro invertebrate (DP1.20120.001).

clnsmth commented 4 years ago

Hi @sokole. Do you have any updates on this front? Or a prioritized list of NEON data products to be converted?

sokole commented 4 years ago

@clnsmth, I don't remember where we left this, but we should update functions to use neonUtilities to pull in data. I can also come up with a list of priority data products to map. It's relatively easy to do, just need to prioritize it and find some time if you're actively working on this part of ecocomDP now.

clnsmth commented 4 years ago

Hey @sokole, we're not working on this right now but may be in the not too distant future. Would be great to see that prioritized list when you have the time!

clnsmth commented 4 years ago

@sokole, querying for the full "DP1.20166.001" returns a lot of noise (see below). What could be done in map_neon_data_to_ecocomDP() to reduce it?

my_result <- ecocomDP::map_neon_data_to_ecocomDP(
  neon.data.product.id = "DP1.20166.001")
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No files found for site PRPO and month 2018-07
No files found for site PRPO and month 2018-09
No files found for site PRPO and month 2019-07
No files found for site PRPO and month 2019-09
No files found for site REDB and month 2015-10
No files found for site REDB and month 2017-04
No files found for site REDB and month 2017-07
No files found for site REDB and month 2017-10
No files found for site REDB and month 2018-04
No files found for site REDB and month 2018-10
No files found for site REDB and month 2019-04
No files found for site REDB and month 2019-07
No files found for site SUGG and month 2014-07
No files found for site SUGG and month 2014-10
No files found for site SUGG and month 2015-02
No files found for site SUGG and month 2015-11
No files found for site SUGG and month 2016-03
No files found for site SUGG and month 2016-06
No files found for site SUGG and month 2017-02
No files found for site SUGG and month 2017-07
No files found for site SUGG and month 2017-11
No files found for site SUGG and month 2018-07
No files found for site SUGG and month 2018-11
No files found for site SUGG and month 2019-02
No files found for site SUGG and month 2019-11
No files found for site SUGG and month 2020-02
No files found for site SYCA and month 2017-07
No files found for site SYCA and month 2018-02
No files found for site SYCA and month 2018-03
No files found for site SYCA and month 2018-06
No files found for site SYCA and month 2019-02
No files found for site SYCA and month 2019-03
No files found for site SYCA and month 2019-06
No files found for site SYCA and month 2020-01
No files found for site TECR and month 2019-09
No files found for site TOMB and month 2016-11
No files found for site TOMB and month 2017-02
No files found for site TOMB and month 2017-11
No files found for site TOMB and month 2018-03
No files found for site TOMB and month 2018-07
No files found for site TOMB and month 2019-04
No files found for site TOMB and month 2019-07
No files found for site TOMB and month 2019-11
No files found for site TOOK and month 2016-08
No files found for site TOOK and month 2017-07
No files found for site TOOK and month 2017-09
No files found for site TOOK and month 2018-06
No files found for site TOOK and month 2018-09
No files found for site TOOK and month 2019-06
No files found for site TOOK and month 2019-07
No files found for site WALK and month 2015-07
No files found for site WALK and month 2015-11
No files found for site WALK and month 2016-03
No files found for site WALK and month 2016-11
No files found for site WALK and month 2017-03
No files found for site WALK and month 2017-10
No files found for site WALK and month 2018-03
No files found for site WALK and month 2018-07
No files found for site WALK and month 2018-10
No files found for site WALK and month 2019-07
No files found for site WALK and month 2019-10
No files found for site WLOU and month 2017-05
No files found for site WLOU and month 2017-07
No files found for site WLOU and month 2017-09
No files found for site WLOU and month 2018-05
No files found for site WLOU and month 2018-09
No files found for site WLOU and month 2019-05
No files found for site WLOU and month 2019-09
Continuing will download files totaling approximately 26.0 MiB. Do you want to proceed y/n: y
Downloading 325 files
  |=========================================================| 100%

Unpacking zip files using 1 cores.
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=04s  
Stacking operation across a single core.
Stacking table alg_biomass
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=05s  
Stacking table alg_fieldData
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=05s  
Stacking table alg_taxonomyProcessed
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=05s  
Copied the most recent publication of validation file to /stackedFiles
Copied the most recent publication of categoricalCodes file to /stackedFiles
Copied the most recent publication of variable definition file to /stackedFiles
Finished: Stacked 3 data tables and 3 metadata tables!
Stacking took 17.03703 secs
Creating the ecocomDP location table

sokole commented 4 years ago

All those warnings are from neonUtilities. I'll see if there's a way to silence it. Although, those warnings are meaningful. It seems there are no files for site/months where the metadata suggests we should expect data. I might need to look into that.

...well, I'm not sure if the encoding warning is useful.

clnsmth commented 4 years ago

Yeah @sokole some of these are meaningful but others are not . For example:

A user downloading all available data will understand where and when data are not available by their absence in the returned object or by reading the product metadata ahead of time. Thus the "No files found for ... " messages could be suppressed.
There doesn't seem to be an argument to neonUtilities::loadByProduct() that allows specification of an encoding so the "No encoding supplied: defaulting to UTF-8." message is not useful.

clnsmth commented 3 years ago

read_data() calls NEON mapping functions and returns L1 results to the user.

EDIorg / ecocomDP

Functions to convert NEON to ecocomDP #31