Traitvis webapp does not reload properly after reboot - Githubissues

terraref / reference-data

Coordination of Data Products and Standards for TERRA reference data

https://terraref.org

BSD 3-Clause "New" or "Revised" License

9 stars 2 forks source link

Traitvis webapp does not reload properly after reboot #252

Closed dlebauer closed 5 years ago

dlebauer commented 5 years ago

After the traitvis webapp is rebooted, it does not have a complete dataset.

I am unable to reproduce this issue locally. I suspect that the docker logs may have some additional information as to what is causing this issue to occur.

What I expect is what happens when I run this locally, and what @craig-willis gets after he restarts the docker image (not sure if he just reruns the cache-refresh.R script, or ?)

But what typically occurs after the traitvis.terrraref.workbench.org is rebooted is that only the data from one experiment shows up. This is the present state:

@tcnichol

could you please check the logs on this machine to see what is the error?
can you let me know the size of the file cache.RData?
can you see if running cat cache-refresh.R | R --vanilla fixes the issue?

max-zilla commented 5 years ago

cache file ~400 MB if it works properly? this does not seem to happen consistently. on workbench R studio it works fine.

max-zilla commented 5 years ago

so we saw when we loaded the app that only Season 1 was available.

got onto machine, ran R, source('cache-refresh.R'), now we have up through Season 4 and it's still running, but it seems like data is being loaded.

more sleuthing is required but maybe we can rule out:

running out of memory - didn't happen here
R script failing - if it did, it failed on a loop through Season 2 b/c Season 1 properly loaded.

no errors were obvious in the logs so far.

max-zilla commented 5 years ago

We're currently thinking shiny might be timing out on the BETY pull:

Adding cronjob:
---------------

## cronR job
## id:   cache-update
## tags: 
## desc: daily update of BETYdb cache
0 0 * * * /usr/local/lib/R/bin/Rscript '/srv/shiny-server/cache-refresh.R'  >> '/srv/shiny-server/cache-refresh.log' 2>&1
Downloading data for Danforth Sorghum Pilot
Downloading data for MAC Season 1
[2018-09-28T18:37:38.303] [INFO] shiny-server - Error getting worker: Error: The application took too long to respond.
[2018-09-28T18:37:38.304] [INFO] shiny-server - Error getting worker: Error: The application took too long to respond.
[2018-09-28T18:37:38.304] [INFO] shiny-server - Error getting worker: Error: The application took too long to respond.

default seems to be 60 seconds. increased 5 minutes and running a test.

max-zilla commented 5 years ago

Updating timeout seems promising:

Adding cronjob:
---------------

## cronR job
## id:   cache-update
## tags: 
## desc: daily update of BETYdb cache
0 0 * * * /usr/local/lib/R/bin/Rscript '/srv/shiny-server/cache-refresh.R'  >> '/srv/shiny-server/cache-refresh.log' 2>&1
Downloading data for Danforth Sorghum Pilot
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 1
Downloading data for MAC Season 2
Downloading data for KSU 2016
Downloading data for KSU 2016
Downloading data for MAC Season 3
Downloading data for MAC Season 3
Downloading data for MAC Season 3
Downloading data for MAC Season 3
Downloading data for MAC Season 3
Downloading data for MAC Season 4
Downloading data for MAC Season 4
Downloading data for MAC Season 4

max-zilla commented 5 years ago

OK, - @tcnichol as we hoped restarting the container and not trying to load the webpage before the cache is done worked with a 20 minute timeout:

Calling cache-refresh.R to create cache.RData for the first time.
Downloading data for Danforth Sorghum Pilot : Drought Tolerance
Downloading data for MAC Season 1 : Biomass
Downloading data for MAC Season 1 : Buffer
Downloading data for MAC Season 1 : Density
Downloading data for MAC Season 1 : RILs
Downloading data for MAC Season 1 : Row#
Downloading data for MAC Season 1 : Uniform
Downloading data for MAC Season 1 : nighthv
Downloading data for MAC Season 1 : BAP
Downloading data for MAC Season 2 : Stay-green RILs F10
Downloading data for KSU 2016 : Biomass Association Panel
Downloading data for KSU 2016 : Ashland Bottoms BAP & RIL
Downloading data for MAC Season 3 : Winter Wheat
Downloading data for MAC Season 3 : Winter Wheat Border
Downloading data for MAC Season 3 : Winter Wheat Uniformity Block
Downloading data for MAC Season 3 : Winter Wheat North
Downloading data for MAC Season 3 : Winter Wheat South
Downloading data for MAC Season 4 : Border Plots
Downloading data for MAC Season 4 : Short BAP Accessions
Downloading data for MAC Season 4 : Medium BAP Accessions
Downloading data for MAC Season 4 : Tall BAP Accessions
Downloading data for MAC Season 4 : All BAP Accessions
Downloading data for Danforth BAP Drought LT1b : Drought Experiment (TM016_F_052716)
Downloading data for Danforth BAP Drought LT1a :  Drought Experiment (TM015_F_051616)
Downloading data for MAC Season 5 : Winter Wheat
Downloading data for MAC Season 6 : Sorghum BAP
Completed cache refresh properly.

Listening on http://127.0.0.1:33329

On Monday we can see if loading the page mid-cache refresh is causing problems itself or if the long 20 min timeout resolves it. In the meantime, app is up with all data.

dlebauer commented 5 years ago

make sure to add shiny-server.conf file to repo (and document in readme)

max-zilla commented 5 years ago

can we run cache-refresh.R as a background process in the docker init outside the app?

dlebauer commented 5 years ago

Seems reasonable to me

max-zilla commented 5 years ago

This issue appears to be resolved by fixes in this PR: https://github.com/terraref/traitvis-webapp/pull/18/files

Was unable to recreate bug running that version.

tcnichol commented 5 years ago

Since this shows merged i will close this issue