Fix cron jobs

[Aneoshun]

It seems that the Cron jobs are not running properly (or at all) on master.

First, I changed the jobs so that they save some logs in /var/log/.

Then, from what I see, the main problem is that /usr/bin/python is actually python2, while everything has been install with pip3. So, this PR replaces the calls to python, with /usr/local/bin/python (which is python3.5)

Also, it seems that an import is missing in the reset_container_limits.py.

---

However, this PR does not solve all the problems that I encountered: generate_tree does not work on my side (I don't see the nice visualisation on the web interface):

cat generate_tree.log                                                                                          
Unknown command: 'generate_tree'. Did you mean generate_secret_key?                                                                        
Type 'manage.py help' for usage.  

I can't find the corresponding command definition (while I do for reset_container_limits and cleanup_dummy).

Also, I get this kind of warnings with reset_container_limits, cleanup_dummy, and backup_db:

cat backup_db.log
Backing up users                                                                                                                           
WARNING Database disabled. Install sqlalchemy for full functionality                                                                       
Backing up api                                                                                                                             
WARNING Database disabled. Install sqlalchemy for full functionality                                                                       
Backing up main                                                                                                                            
WARNING Database disabled. Install sqlalchemy for full functionality                                                                       
Backing up logs                                                                                                                            
WARNING Database disabled. Install sqlalchemy for full functionality

Do you think the DB operations are conducted properly and it is safe to ignore, or is there an additional package to install?

[Aneoshun]

Hum... the formatting ci check is failing, while I ran make lint. Interestingly, the only two files that make lint changed in my PR are the same two files that the CI complains about.

[vsoch]

Taking a look! For the linter, it's fussy with respect to version, so you'd want to install the same version as CI. The one I have locally (that if I remember correctly coincides with CI) is

$ pip freeze | grep black
black==19.3b0

[Aneoshun]

I see, on my side I have

pip3 freeze | grep black
black==19.10b0

[vsoch]

To check cron I shell into uwsgi and do:

root@63a129740eba:/code# service cron status
[ ok ] cron is running.
root@63a129740eba:/code# crontab -l
0 0 * * * /usr/bin/python /code/manage.py generate_tree
0 * * * Mon /usr/bin/python /code/manage.py reset_container_limits
0 1 * * * /bin/bash /code/scripts/backup_db.sh

You are correct that generate tree is missing - let me see what might have happened there.

[Aneoshun]

As far as I know, this won't give you proper logs on the jobs. Or are there some cron magic that I don't know? (by default cron saves logs in /var/log/syslog, but this requires the installation of a specific package according to https://askubuntu.com/questions/56683/where-is-the-cron-crontab-log).

[vsoch]

Yes this is true, it doesn't generate logs!

I'm looking at the project history, and it looks like I removed generation of the tree because it was using an external library (singularity) and I felt that I wasn't following the KISS (keep it simple, stupid) principle. So what I'd suggest we do here is remove the function from the cron, and the links under tools, It's not the prime function of the registry, and can just become a burden if it becomes very big. So - for this particular issue don't worry about it, after we finish up here I can open a PR to clean out the tree views.

[vsoch]

I'm not getting the error about sqlalchemy not installed when I run inside my container - but I'll tell you where that comes from! sregistry (client) is used for the push/pull endpoints (before the library) and a "full" install of sregistry (pip install sregistry that is) would include a local database using sqlalchemy. If it's not installed, you get a warning. But in the uwsgi container you shouldn't because the MESSAGELEVEL is explicitly set to quiet, try:

# env | grep MESSAGELEVEL
MESSAGELEVEL=QUIET

Are you running this elsewhere other than the uwsgi container? Or is the envar not set?

[Aneoshun]

Regarding the MESSAGELEVEL: I am not running the scripts myself, I just changed the frequency of cron jobs (every 10 minutes) to be able to check the logs more frequently. From what I know, the cron environment variables is different from the user environment, that might be why MESSAGELEVEL is not defined properly. Anyway, if the warning is not critical, I don't think it is an issue.

[vsoch]

Oh, that's a very good point!

Were you able to install the same version of black (and it's still failing?)

[Aneoshun]

No sorry, I did not try to install the right version of black (so that's the same error as before).

[vsoch]

If you want to adjust the cron time (or chat about it) after that this looks good to me, although it has to look good to you too (and remove the WIP label of course).

[vsoch]

One small detail (and sorry for not mentioning before) but in the VERSION file we need to bump 1.1.14 to 1.1.15 and add a comment about changes to the CHANGELOG. This will determine the container tag that gets built when we merge. It's not a great versioning system, but it's better than nothing!

[Aneoshun]

Sure! I do that right now. Thanks for approving the changes.

[vsoch]

okay we're good! Sorry for the delay - I keep getting really absorbed / distracted and then see the tab open! Let's merge this and I'll get started on removing the tools / treemap view.