search

terraref / reference-data

Coordination of Data Products and Standards for TERRA reference data
https://terraref.org
BSD 3-Clause "New" or "Revised" License
9 stars 2 forks source link

Upload 2016 data from KSU to terraref/bety #85

Open dlebauer opened 7 years ago

dlebauer commented 7 years ago

Data from KSU 2016 field season is in here:

The most straightforward way to get these data into BETYdb will be to use the API. The web interface is also handy, especially for information like defining traits and agronomic managements where there are only a handful of records. (you could also use SQL but the API hides a lot of complexity).

If you would prefer to upload to BMS, then we can work on the BMS-->BETYdb importer. This assumes BMS exports BRAPI, so this importer would also work with FieldBook and any other data source that exports BRAPI.

Completion Criteria

References

Templates for the data we need are here: https://docs.google.com/spreadsheets/d/1WmWyVhGfKHEL-i9E2JOEt1txiAgTL4n3lbyQnqsUbNA/edit#gid=1497549722);

There are some examples of measurements we have uploaded from MAC in the google drive folder (Maricopa Experiments/Field Seasons 1 and 2/Data for uploading/). This one is fairly comprehensive

dlebauer commented 7 years ago

@gpmorrisksu what is the plan?

gpmorrisksu commented 7 years ago

Hi David,

Zhenbin has already sent the 2016 manual phenotyping data, no? If the question is fieldbook to BrAPI format conversion, I think that better addressed in your group. I think our responsibility lies in producing the data and analyzing G-to-P.

Best, Geoff

Geoff Morris, Assist. Professor Kansas State Univ., Agronomy 3004 Throckmorton<x-apple-data-detectors://6/0> Plant Sci. Manhattan KS, 66506<x-apple-data-detectors://6/0> E-mail: gpmorris@k-state.edumailto:gpmorris@k-state.edu Web:http://www.morrislab.orghttp://www.morrislab.org/ Office: 785-532-3397 Cell: 312-909-1330 Skype/Google: morris.geoff.p

Sent from my mobile device

On Mar 8, 2017, at 10:47 PM, David LeBauer notifications@github.com<mailto:notifications@github.com> wrote:

@gpmorrisksuhttps://github.com/gpmorrisksu what is the plan?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/terraref/reference-data/issues/85#issuecomment-285195213, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATLjZBi3yVk7_oZ-54cPbCHLNUQlrVr3ks5rjy_vgaJpZM4L0JSX.

dlebauer commented 7 years ago

@gpmorrisksu sorry if I was not clear. When entering data, we need to make sure that we have all of the information that would be required for someone to make use of the primary data.

This includes the definitions of phenotypes and methods used to collect them, the agronomic managements (planting date and density, fertilization, tillage, experimental design, plot layouts), and the genotypes, used,

My groups responsibility is to provide a place to store this data and metadata and make it available. We can help with the entire process. But it is not our responsibility to interpret any set of tables that are emailed to us.

My understanding was that fieldbook was already capable of exporting to the BRAPI format, which includes all of this information. Supporting import (and export) of data via community supported formats such as the BRAPI api is our responsibility. So if it is easier to supply your information in that format than via our web pages and bulk upload interface or API then that would also work.

dlebauer commented 7 years ago

@zhenbinHU can you please update this issue with a list of the steps that you have accomplished with respect to preparing data and metadata.

max-zilla commented 7 years ago

@zhenbinHU please check / update completion criteria and let us know if you need input

zhenbinHU commented 7 years ago

@dlebauer PG::RaiseException: ERROR: The value of cultivar_id (6000000456) is not consistent with the value 6000000937 specified for site_id 6000004483. : INSERT INTO "traits" ("access_level", "checked", "citation_id", "created_at", "cultivar_id", "date", "date_day", "date_month", "date_year", "dateloc", "entity_id", "mean", "method_id", "n", "notes", "site_id", "specie_id", "stat", "statname", "time", "time_hour", "time_minute", "timeloc", "treatment_id", "updated_at", "user_id", "variable_id") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27) RETURNING "id"

After I click the insert data for uploading data to betydb, I have this error.

NewcombMaria commented 7 years ago

Is this issue the same topic as #119 (Upload hand measured traits from KSU 2016). There might be separate conversations going on about betydb data upload errors. See comment by @gsrohde in issue 119.

zhenbinHU commented 7 years ago

Thanks, @NewcombMaria Yes, I just remove the data (five plots) with errors from the uploading data.

gsrohde commented 7 years ago

@NewcombMaria , @zhenbinHU In this case, it seems the error does have to do with the sites_cultivars table. The sites_cultivars table says that for site 6000004483 ("Ashland Bottoms KSU 2016 Season Range 25 Pass 22"), the cultivar there is 6000000937 ("PI569459"), but the upload file specifies cultivar 6000000456 ("Blade"). If you believe the sites_cultivars table is correct, you can simply remove the cultivar column from the upload file and the correct cultivar will be supplied automatically.