Scheduled update

[csomh]

Add a command and set-up a cron job to execute it, in order to have it periodically checked whether source-git repos are up to date and create update tasks in case they are not.

TODO:

• [x] Monitor with Pushgateway
• [x] Unit tests

[softwarefactory-project-zuul[bot]]

Build succeeded.

• pre-commit : SUCCESS in 3m 24s
• dist2src-tests : SUCCESS in 31m 01s
• dist2src-deployment-tests : SUCCESS in 9m 35s

[softwarefactory-project-zuul[bot]]

Build succeeded.

• pre-commit : SUCCESS in 5m 54s
• dist2src-tests : SUCCESS in 27m 34s
• dist2src-deployment-tests : SUCCESS in 8m 49s

[softwarefactory-project-zuul[bot]]

Build succeeded.

• pre-commit : SUCCESS in 1m 55s
• dist2src-tests : SUCCESS in 25m 14s
• dist2src-deployment-tests : SUCCESS in 9m 08s

[csomh]

One suggestion I'd make would be to do more logging, e.g. log the expected git-tags or hashes:

I've added more log messages, increased the log-level in the CronJob and added an option to also print some timestamps (this is not required when we run our code through Celery, as workers attach their own timestamps).

Example output:

[2020-12-08 08:45:38,779 updater.py DEBUG] Source-git API: 'https://git.stg.centos.org/api/0/'
[2020-12-08 08:45:38,779 updater.py DEBUG] Source-git namespace: 'forks/csomh/source-git'
[2020-12-08 08:45:38,869 updater.py DEBUG] Dist-git API: 'https://git.centos.org/api/0/'
[2020-12-08 08:45:38,870 updater.py DEBUG] Dist-git namespace: 'forks/csomh/rpms'
[2020-12-08 08:45:38,870 updater.py DEBUG] Dist-git branches watched: ['c8s', 'c8', 'c8s-stream-rhel']
[2020-12-08 08:45:39,745 updater.py DEBUG] Next page: None. Total pages: 1.
[2020-12-08 08:45:39,745 updater.py DEBUG] Checking project 'acl'...
[2020-12-08 08:45:41,402 updater.py DEBUG] Tags expected in source-git: {'convert/c8/fa4074d481e8088a1b9167f1b3d2318dd29604a0'}
[2020-12-08 08:45:41,977 updater.py DEBUG] Current tags in source-git: {'imports/test/20201124101134', 'convert/c8/fa4074d481e8088a1b9167f1b3d2318dd29604a0', 'convert/c8/a4c738fb2879285555a7c31e8c26ab50e0ed6d75', 'sg-start', 'imports/test/20201124100929'}
[2020-12-08 08:45:41,977 updater.py DEBUG] Tags missing from source-git: set()
[2020-12-08 08:45:41,977 updater.py DEBUG] Checking project 'packit-hello-world'...
[2020-12-08 08:45:42,460 updater.py WARNING] 'fork/csomh/rpms/packit-hello-world' does not exist in 'git.centos.org'
[2020-12-08 08:45:42,466 updater.py DEBUG] Checking project 'rpm'...
[2020-12-08 08:45:42,994 updater.py WARNING] 'fork/csomh/rpms/rpm' does not exist in 'git.centos.org'
[2020-12-08 08:45:42,999 updater.py DEBUG] Checking project 'rsync'...
[2020-12-08 08:45:43,676 updater.py DEBUG] Tags expected in source-git: {'convert/c8s/e7be5025dc4ed6d41a587d36df4584e134300112', 'convert/c8/043c57dad5c7665b0cdb553e561dc55b3c676999'}
[2020-12-08 08:45:44,232 updater.py DEBUG] Current tags in source-git: {'convert/c8s/e7be5025dc4ed6d41a587d36df4584e134300112', 'convert/c8/043c57dad5c7665b0cdb553e561dc55b3c676999', 'c8-source-git', 'c8s-source-git', 'sg-start'}
[2020-12-08 08:45:44,232 updater.py DEBUG] Tags missing from source-git: set()

I've also renamed tests/test_updater.py to tests/test_worker_updater.py to avoid confusion with tests/test_update.py and changed the default of the PUSHGATEWAY_ADDRESS to None, so that it's disabled unless explicitly set (handy for local usage).

Please take another look.

[softwarefactory-project-zuul[bot]]

Build succeeded.

• pre-commit : SUCCESS in 2m 21s
• dist2src-tests : SUCCESS in 32m 47s
• dist2src-deployment-tests : SUCCESS in 11m 51s

[softwarefactory-project-zuul[bot]]

Build succeeded (gate pipeline).

• pre-commit : SUCCESS in 2m 55s