search

terraref / brapi

Breeder's API implementation for TERRA-REF
https://terraref.org/brapi/v1/calls
BSD 3-Clause "New" or "Revised" License
0 stars 2 forks source link

{WIP}Issue 537/cultivar treatment siteclear #10

Closed tcnichol closed 5 years ago

tcnichol commented 5 years ago

This pull request is part of issue 537 - add treatments and cultivar data to site metadata.

A new endpoint has been added to brapi where you can either request the treatments (id, name, definition) by a single experiment id, or by a list of experiment ids in a POST method.

An endpoint for the cultivars still needs to be added.

The plan to finish this issue requires 2 pull requests in 2 repositories.

  1. pull request in brapi.
  2. changes made in terrautils.lemnatec to take advantage of these new routes.

I'm also not sure how to proceed with updating existing datasets with this information.

robkooper commented 5 years ago

I think we should star to use logging statements. I think queries can be DEBUG. Add to the main function the initialization of the logging system, and set the default to be WARNING but allow this to be overriden with an environment variable (like we do for database) and you can set it on your own machine to be DEBUG.

tcnichol commented 5 years ago

Still need to fix logging, but can someone check the phenotypes-search changes I made? This is what would allow us to look up treatments by the experimentId. The other 2 methods I have implemented or changed I feel more confident about.

tcnichol commented 5 years ago

Here is the result of this route : http://0.0.0.0:8080/brapi/v1/studies/6000000012

// 20190302125411
// http://0.0.0.0:8080/brapi/v1/studies/6000000012

{
  "metadata": {
    "datafiles": [

    ],
    "pagination": {
      "currentPage": 0,
      "pageSize": 1000,
      "totalCount": 1,
      "totalPages": 1
    }
  },
  "result": {
    "experiment": [
      {
        "description": "# MaterialsProMix BRK20 + 14-14-14 Osmocote pots; pre-filled by Hummert Sorghum seedConviron Growth HouseLemnaTec moving field conveyor belt systemScanalyzer 3D platform# Procedures## PlantingPlant directly into phenotyping pots ## Chamber ConditionsPre-growth (11 days) and Phenotying (11 days)14 hour photoperiod32 C day / 22 C night temperature60% relative humidity700 umol/m2/s lightWatering ConditionsPrior to phenotyping, plants watered dailyThe first night after loading, plants watered 1× by treatment group to 100% field capacity (fc)Days 2 – 12, plants watered 2× daily by treatment group (100% or 30% FC) to target weightAutomationLeft shift lane rotation within each GH, during overnight watering jobsVIS (TV and 2 x SV), NIR (TV and 2 x SV) imaging dailyRecipesField capacity = 200% GWC (200 g water/100 g soil), based upon extensive GWC testing done by Skyler MitchellTarget weight (fc) = [(water weight at % fc) + [(average weight of carrier/saucer) + (dry soil weight) + (pot weight)]Water weight at 100% fc = dry soil weight * (%GWC/100)Water weight at 30% fc = water weight at 100% fc * 0.30",
        "end_date": "2014-06-24",
        "experiment_id": 6000000012,
        "experiment_name": "Danforth Sorghum Pilot: Drought Tolerance",
        "site": {
          "site_id": 6000000866,
          "site_name": "Danforth Plant Science Center Bellweather Phenotyping Facility"
        },
        "start_date": "2014-05-27"
      }
    ]
  }
}
tcnichol commented 5 years ago

for the /germplasm, this is one of the results (I did not post whole json, as it was too long)

"result": {
    "experiment": [
      {
        "description": "",
        "end_date": "2016-07-14",
        "experiment_id": 6000000002,
        "experiment_name": "MAC Season 1: Buffer",
        "site": {
          "cultivar": {
            "common_name": "sorghum",
            "name": "RIL-CS3_(TX2910/(Macia/R07007)-CS44)-CSF1-PRF2-CS3",
            "scientific_name": "Sorghum bicolor"
          },
          "site_id": 6000000862,
          "site_name": "MAC Field Scanner Season 1 Field Plot 861"
        },
        "start_date": "2016-04-19"
tcnichol commented 5 years ago

http://0.0.0.0:8080/brapi/v1/phenotypes-search?studyDbId=6000000012

"result": {
    "data": {
      "observations": [
        {
          "experimentid": 6000000012,
          "observationdbid": 6000075861,
          "observationtimestamp": "2014-06-14T21:04:04.406000Z",
          "observationvariabledbid": 6000000005,
          "observationvariablename": "plant_height",
          "sitename": "Danforth Plant Science Center Bellweather Phenotyping Facility",
          "treatmentdefinition": "",
          "treatmentid": 6000000010,
          "treatmentname": "100%: 217 ml water (47.6% VWC)",
          "value": 671.0
        },
        {
          "experimentid": 6000000012,
          "observationdbid": 6000075860,
          "observationtimestamp": "2014-06-14T21:04:04.406000Z",
          "observationvariabledbid": 6000000004,
          "observationvariablename": "solidity",
          "sitename": "Danforth Plant Science Center Bellweather Phenotyping Facility",
          "treatmentdefinition": "",
          "treatmentid": 6000000010,
          "treatmentname": "100%: 217 ml water (47.6% VWC)",
          "value": 0.153547903005
tcnichol commented 5 years ago

changes made to jsons : below

// http://0.0.0.0:8080/brapi/v1/phenotypes-search?studyDbId=6000000012

{
  "metadata": {
    "datafiles": [

    ],
    "pagination": {
      "currentPage": 0,
      "pageSize": 1000,
      "totalCount": 21960,
      "totalPages": 22
    }
  },
  "result": {
    "data": {
      "observations": [
        {
          "location_abbreviation": "Danforth Plant Science Center Bellweather Phenotyping Facility",
          "observationdbid": 6000075861,
          "observationtimestamp": "2014-06-14T21:04:04.406000Z",
          "observationvariabledbid": 6000000005,
          "observationvariablename": "plant_height",
          "studydbid": 6000000012,
          "treatment_definition": "",
          "treatment_factor": "100%: 217 ml water (47.6% VWC)",
          "treatments": 6000000010,
          "value": 671.0
        },
tcnichol commented 5 years ago 

// 20190304103138
// http://0.0.0.0:8080/brapi/v1/studies/6000000001

{
  "metadata": {
    "datafiles": [

    ],
    "pagination": {
      "currentPage": 0,
      "pageSize": 1000,
      "totalCount": 144,
      "totalPages": 1
    }
  },
  "result": {
    "study": [
      {
        "end_date": "2016-07-14",
        "location": {
          "location_abbreviation": "MAC Field Scanner Season 1 Field Plot 572",
          "location_name": 6000000080
        },
        "start_date": "2016-04-19",
        "studyDbId": 6000000001,
        "studyDescription": "",
        "studyName": "MAC Season 1: Biomass"
      },
tcnichol commented 5 years ago 

// 20190304103209
// http://0.0.0.0:8080/brapi/v1/studies/6000000001/germplasm

{
  "metadata": {
    "datafiles": [

    ],
    "pagination": {
      "currentPage": 0,
      "pageSize": 1000,
      "totalCount": 166481,
      "totalPages": 167
    }
  },
  "result": {
    "study": [
      {
        "end_date": "2016-07-14",
        "location": {
          "germplasm": {
            "common_name": "sorghum",
            "germplasmName": "RIL-CS3_(TX2910/(Macia/R07007)-CS44)-CSF1-PRF2-CS3",
            "scientific_name": "Sorghum bicolor"
          },
          "location_abbreviation": "MAC Field Scanner Season 1 Field Plot 580 W",
          "location_name": 6000001498
        },
        "start_date": "2016-04-19",
        "studyDbId": 6000000001,
        "studyDescription": "",
        "studyName": "MAC Season 1: Biomass"
      },
tcnichol commented 5 years ago

Here are current jsons. Please provide feedback on which keys are wrong. Part of the json included, but should have every key

brapi/v1/studies/6000000002

"result": {
    "study": [
      {
        "endDate": "2016-07-14",
        "location": {
          "abbreviation": "MAC Field Scanner Season 1 Field Plot 1",
          "name": 6000000002
        },
        "startDate": "2016-04-19",
        "studyDbId": 6000000002,
        "studyDescription": "",
        "studyName": "MAC Season 1: Buffer"
      },

brapi/v1/studies/6000000002/germplasm

result": {
    "study": [
      {
        "endDate": "2016-07-14",
        "location": {
          "abbreviation": "MAC Field Scanner Season 1 Field Plot 861",
          "germplasm": {
            "common_name": "sorghum",
            "germplasmName": "RIL-CS3_(TX2910/(Macia/R07007)-CS44)-CSF1-PRF2-CS3",
            "scientific_name": "Sorghum bicolor"
          },
          "name": 6000000862
        },
        "startDate": "2016-04-19",
        "studyDbId": 6000000002,
        "studyDescription": "",
        "studyType": "MAC Season 1: Buffer"
      },

brapi/v1/phenotypes-search?studyDbId=6000000012

"result": {
    "data": {
      "observations": [
        {
          "location_abbreviation": "Danforth Plant Science Center Bellweather Phenotyping Facility",
          "observationdbid": 6000075861,
          "observationtimestamp": "2014-06-14T21:04:04.406000Z",
          "observationvariabledbid": 6000000005,
          "observationvariablename": "plant_height",
          "studydbid": 6000000012,
          "treatment_definition": "",
          "treatment_factor": "100%: 217 ml water (47.6% VWC)",
          "treatments": 6000000010,
          "value": 671.0
        },
tcnichol commented 5 years ago

I changed 'study' to 'data' - will remove the other deprecated endpoint.

tcnichol commented 5 years ago

There is now a test instance up that matches whatever the latest branch is for this pull request at

https://brapi-dev.workbench.terraref.org/brapi/v1

@dlebauer hopefully this will make checking the jsons easier for you.

in the future the plan will be to implement each endpoint on its own pull request, so that they are easier to check and merge. for this pull request, how about we consider it 'done' when the 'studies/studyDbId' and the 'studies/studyDbId/germplasm and the 'studies/studyDbId/layout' endpoints are compliant.

i will implement the /layout endpoint later today or tomorrow.

tcnichol commented 5 years ago

I am trying to fix the endpoints one at a time.

@dlebauer could you check the dev brapi endpoints for 'studyDbId' and also the 'germplasm' one. I made some changes to germplasm trying to match the brapi specs as I had included what appear to be unnecessary keys there.

dlebauer commented 5 years ago

@tcnichol ... the website says ‘your service is still initializing, please wait’

And if you don’t mind, it is easier for me as well as less ambiguous so I know I am hitting the correct endpoint if you can provide a full URL such as https://brapi-dev.workbench.terraref.org/brapi/v1/studies/studydbid/6000000012

tcnichol commented 5 years ago

I will check why this isn't loading, and once it's up, I will get links to an endpoint for each

tcnichol commented 5 years ago

links to endpoints on the test/dev brapi (should be up and running now)

https://brapi-dev.workbench.terraref.org/brapi/v1/studies/6000000012

https://brapi-dev.workbench.terraref.org/brapi/v1/studies/6000000002/germplasm

tcnichol commented 5 years ago

i'm seeing an error in the server logs for the /germplasm endpoint. this does not happen locally, and did not occur previously 

"GET /brapi/v1/studies/6000000002/germplasm HTTP/1.1" 500 -
[2019-03-20 15:33:37,630] DEBUG in StudiesController_impl: SELECT experiments.id as studyDbId,    experiments.name as studyName,    experiments.start_date as startDate,    experiments.end_date as endDate,    experiments.description as studyDescription,    experiments_sites.site_id as location_name,    sites.sitename as location_abbreviation,    sites_cultivars.cultivar_id as germPlasmDbId,    cultivars.specie_id as species,    cultivars.id as cultivarid,    cultivars.name as germplasmName,    species.scientificname as scientificname,    species.commonname as commonname FROM experiments, experiments_sites, sites, sites_cultivars, cultivars, species WHERE experiments.id = experiments_sites.experiment_id AND sites.id = experiments_sites.site_id AND sites_cultivars.site_id = experiments_sites.site_id AND species.id = cultivars.specie_id  and experiments.id = %s 
[2019-03-20 15:33:37,635] ERROR in app: Exception on /brapi/v1/studies/6000000002/germplasm [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: permission denied for relation sites_cultivars
dlebauer commented 5 years ago

for studies I think location / abbreviation should be locationName, and location / name should be locationDbId, right?

David LeBauer Director of Data Sciences Arizona Experiment Station THE UNIVERSITY OF ARIZONA

Bioscience Research Labs, 207 1230 N Cherry Ave | Tucson, AZ 85721 Office: 520-621-4381 dlebauer@email.arizona.edu

(sent from my phone - please pardon brevity and typos)

From: Todd Nicholson notifications@github.com Sent: Wednesday, March 20, 2019 8:38 AM To: terraref/brapi Cc: LeBauer, David Shaner - (dlebauer); Mention Subject: Re: [terraref/brapi] Issue 537/cultivar treatment siteclear (#10)

i'm seeing an error in the server logs for the /germplasm endpoint. this does not happen locally, and did not occur previously

"GET /brapi/v1/studies/6000000002/germplasm HTTP/1.1" 500 - [2019-03-20 15:33:37,630] DEBUG in StudiesController_impl: SELECT experiments.id as studyDbId, experiments.name as studyName, experiments.start_date as startDate, experiments.end_date as endDate, experiments.description as studyDescription, experiments_sites.site_id as location_name, sites.sitename as location_abbreviation, sites_cultivars.cultivar_id as germPlasmDbId, cultivars.specie_id as species, cultivars.id as cultivarid, cultivars.name as germplasmName, species.scientificname as scientificname, species.commonname as commonname FROM experiments, experiments_sites, sites, sites_cultivars, cultivars, species WHERE experiments.id = experiments_sites.experiment_id AND sites.id = experiments_sites.site_id AND sites_cultivars.site_id = experiments_sites.site_id AND species.id = cultivars.specie_id and experiments.id = %s [2019-03-20 15:33:37,635] ERROR in app: Exception on /brapi/v1/studies/6000000002/germplasm [GET] Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context cursor, statement, parameters, context File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute cursor.execute(statement, parameters) psycopg2.ProgrammingError: permission denied for relation sites_cultivars

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/terraref/brapi/pull/10#issuecomment-474890045, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAcX55knKnDOVn_F_Rqpf6cxllJIZzAUks5vYlXmgaJpZM4bDpXh.

tcnichol commented 5 years ago

@dlebauer

i can fix those. for some reason the germplasm endpoint is not working, but not sure why that is yet.

tcnichol commented 5 years ago

https://brapi-dev.workbench.terraref.org/brapi/v1/studies/6000000002/layout

layout and germplasm endpoints now work.