Summarize trait data from current season

[dlebauer]

Create a web page that summarizes:

• current experiments: records in experiments table for current season (or most recent season)
  • select records in experiments table w/ no end_date, or if none exist the most recent end_date
• records from managements table that occurred during currently running experiments
  • select all records from managements where date between experiments.start_date and experiments.end_date
• trait data collected during currently running experiments
• for each value of variables.name (or variable_id in traits table), plot:
  • histogram
  • time series of boxplots (or similar, including median, quantiles, outliers max/min) (time is (traits.raw_date)
  • time series of records per day observed (traits.raw_date)
  • time series of records per day created (traits.created_at)
  • these should be colored by the value in the 'methods' field

Generate a table with the following

• number of NULLs for each of the following fields in the traits table: site_id, citation_id, cultivar_id, treatment_id, variable_id, user_id, method_id, cultivar_id,

For the purposes of developing a prototype (and until the experiments api is deployed) please start with Sorghum Season 2 which ran from 2016-08-03 to 2016-12-02

Completion Criteria

• [x] code to generate web page in the reference-data repository
• [ ] live web page for each season with above listed summaries.

[ghost]

@nickheyek - can you please give an update on this below?

[nheyek]

Sure! At this point I have implemented code to pull data from the BETYdb API and create box plots for data associated with each variable measured in a given experiment. This is written in Python using the Pygal library. Additionally, I have implemented a web application using the Flask web framework that displays the box plots mentioned above. Also on this web application is a timeline of managements data associated with a given experiment. Currently I am in the process of making corrections and improvements to the existing code. I am also waiting for an 'experiments' endpoint to be added to the beta BETYdb API so that my application can function independently (right now it requires just plugging in a date range). My code is in the reference-data repository within experiment-trait-data-visualizer under the 'feature' branch. Let me know if there's anything else you need.

[dlebauer]

Have you submitted a pull request? You can visualize html checked into GitHub with htmlpreview.github.io/ or http://rawgit.com/

Perhaps put the html in a separate repository or gist for review. On Tue, Apr 4, 2017 at 1:40 PM Nick Heyek notifications@github.com wrote:

Sure! At this point I have implemented code to pull data from the BETYdb API and create box plots for data associated with each variable measured in a given experiment. This is written in Python using the Pygal library. Additionally, I have implemented a web application using the Flask web framework that displays the box plots mentioned above. Also on this web application is a timeline of managements data associated with a given experiment. Currently I am in the process of making corrections and improvements to the existing code. I am also waiting for an 'experiments' endpoint to be added to the beta BETYdb API so that my application can function independently (right now it requires just plugging in a date range). My code is in the reference-data repository within experiment-trait-data-visualizer under the 'feature' branch. Let me know if there's anything else you need.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/261#issuecomment-291593831, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX5ycU2Jin7S1P7tFzXRVFwIbD1iMcks5rso6fgaJpZM4MH5fS .

[nheyek]

I have not submitted a pull request, I will do so today. Because the HTML is not static it won't render without the backend, but I can create a static HTML equivalent with placeholder data. I'll also do that today.

[ghost]

@max-zilla where will we put html files like this? Please advise.

[max-zilla]

talked with @robkooper @craig-willis @jterstriep

Initial plan is to create a new /html directory under /sites/ua-mac alongside raw_data and Level_1. We can deploy the python code to pull from BetyDB and generate the HTML page with a cron job running in a VM that has Roger mounted, maybe the terra.org VM even. That cron job will run python script to generate latest HTML. Then we serve the static HTML file from where it is.

Craig had mentioned running in a container as another option.

@nickheyek let us know once the pull request is created and whether I have the general idea correct (i.e. your python pulls from Bety and generates HTML page? or is HTML a web app that queries bety dynamically?)

[dlebauer]

@max-zilla here is the pull request: https://github.com/terraref/reference-data/pull/110

[dlebauer]

This is a flask application. It has a cache. @nickheyek is planning to have it update the cache each day. It could be dynamic except that the API call is too slow.

[dlebauer]

@gsrohde is there an experiments API endpoint ready to be deployed?

[dlebauer]

@gsrohde actually I realized that I submitted a PR a few weeks ago https://github.com/PecanProject/bety/pull/484 that --should-- add the experiments endpoint.

[gsrohde]

OK, I pulled that request. Now it just needs to be deployed.

[robkooper]

So since this is a flask app, we should either create a docker container, or a VM that hosts the machine and add a link in the terraref web app.

@nickheyek is this a docker container already?

[nheyek]

It is not. Unfortunately I do not have any experience with Docker.

[robkooper]

We will need to create you a VM and give you access to the VM so you can deploy your code there. @dlebauer can you work with me to get @nickheyek access to NCSA accounts.

[gsrohde]

@robkooper Can you figure out why Travis is failing after I pulled this? It complains 'relation "experiments" does not exist'.

[robkooper]

@gsrohde can you create a new issue for that. That is not related to this issue.

[dlebauer]

Indirectly related because the app will eventually use the experiments table

I think Nick already has an NCSA account as a guest. Is that correct Nick? On Fri, Apr 7, 2017 at 11:12 AM Rob Kooper notifications@github.com wrote:

@gsrohde https://github.com/gsrohde can you create a new issue for that. That is not related to this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/terraref/computing-pipeline/issues/261#issuecomment-292580142, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcX5-mwXSRd5GwEzPxtYkfDlMl2DHWzks5rtmBcgaJpZM4MH5fS .

[nheyek]

Not that I know of.

[dlebauer]

@robkooper I've submitted a guest account request for Nick. Is it possible for him to use terraref.ndslabs.org to deploy the VM/docker image?

[ghost]

@dlebauer - this is waiting for review.

[dlebauer]

The first draft is here: http://141.142.211.143:5000/

But it has a few issues:

• does not dynamically update
• only has data from one season.

@nickheyek and I will work on this this summer. A good target would be the July Sprint. I'll convert this to an epic.

[ghost]

@nickheyek do you have a more recent draft with a different URL than http://141.142.211.143:5000/ I think David showed me something a bit different...

[nheyek]

That is the most recent draft. I had planned to begin working on creating a higher quality R/Shiny version when the SPIN internship period begins on the 12th.

[dlebauer]

@nickheyek has this been replaced by a new epic?

[nheyek]

Yes

[dlebauer]

Can you link to the new epic and close this one?

[nheyek]

https://github.com/terraref/reference-data/issues/142