Job per graph - Githubissues

opengridcc / opengrid-dev

Open source building monitoring, analysis and control

Apache License 2.0

26 stars 21 forks source link

Job per graph #38

Closed saroele closed 8 years ago

saroele commented 9 years ago

Our graphs are developed in IPython notebooks. We have to create cron jobs for automatically generating the new graphs every day.

converting notebook to .py (maybe not needed if this works: http://stackoverflow.com/a/17913858/566942
creating a crontab job

For the crontab file, the best will be to put the file in our opengrid repo so we can track changes to the file and deploy a new file easily with crontab ourfile after pulling a new version on the droplet.

dirkdevriendt commented 9 years ago

Indeed, nbconvert is the secret sauce; Either use it to do the conversion to a python script and run the script:

ipython nbconvert SourceNotebook.ipynb --to python --output TargetScript.py ipython TargetScript.py

more info: http://ipython.org/ipython-doc/stable/notebook/nbconvert.html

Or use the ExecutePreprocessor preprocessor to execute the notebook during conversion and throw away the conversion result:

ipython nbconvert SourceNotebook.ipynb --ExecutePreprocessor.enabled=True --to notebook --output ToBeThrownAway.ipynb

more info: http://ipython.org/ipython-doc/dev/whatsnew/development.html

saroele commented 9 years ago

Seems like the second solution is the more elegant.

wrt the crontab job: @icarus75 proposed to have a single python job that syncs houseprint, then syncs tempo, and then calls each of the scripts we want to run for generating graphs. The syncing part has been implemented already in 5ba5e03916d923b95b58d1bd3921e85b1791cab6

saroele commented 8 years ago

I currently use crontab to schedule the jobs individually. The crontab setup is stored in crontab.txt. Currently, there are 3 jobs:

sync tmpo database (hourly)
create the timeseries plots (hourly, starts 5 minutes later)
create the electricity standby plots (every morning 5h30 UTC).

We can easily add graphs in this framework, so I'll close this issue. We can create a more SMART issue if the current approach is not sufficient any longer.

JrtPec commented 8 years ago

In connection with #99 and opengridcc/website#24, I have a feeling that there is some other script running which is permanently connected to the tmpo-database, or one of these three processes doesn't terminate correctly.

When I wrote the download to CSV functionality I fully expected that it wouldn't work whenever another process was running in the background, which is why I added the error message please try again later. I haven't been able to connect to tmpo from the website, even once.

Let's address this in the next meeting.