search

CentOS / container-pipeline-service

Code, infrastructure and deployment backend for the CentOS Container Pipeline backing up build system for registry.centos.org
https://registry.centos.org
GNU General Public License v3.0
49 stars 27 forks source link

Processes the index projects in batches #612

Closed navidshaikh closed 6 years ago

navidshaikh commented 6 years ago

This changeset updates index reader to process index-projects in defined (default=5) length of batches. Index reader, splits the index projects in batches and processes each batch one by one.

In between processing batches of projects, it checks if there are any running builds. If it finds a running build, it waits for given poll_cyle (default=120 seconds) and checks again. As it finds, openshift idle (not processing any builds ATM), it queues the batch and waits for its completion as above.

The weekly scan projects are processed at the end after processing build jobs completely. Every weekly scan job is processed with 5 seconds delay in between.

Sample output first run of the index

[pipeline-service] Running shell script
+ python ccp/index_reader.py /tmp/container-index/index.d 172.29.33.53:5000 cccp nshaikh@redhat.com smtp://mail.centos.org
Number of projects in index 9
Number of projects in OpenShift 0
Number of projects to be updated/created: 9
Processing projects batch: ['test-anomaly-latest', 'test-python-release', 'dharmit-freegeoip-latest', 'dharmit-nsq-latest', 'mattermost-mattermost-github-integration-92394f2']
buildconfig.build.openshift.io "test-anomaly-latest" created

buildconfig.build.openshift.io "test-python-release" created

buildconfig.build.openshift.io "dharmit-freegeoip-latest" created

buildconfig.build.openshift.io "dharmit-nsq-latest" created

buildconfig.build.openshift.io "mattermost-mattermost-github-integration-92394f2" created

Processing projects batch: ['arrfab-nanoc-latest', 'openshift-origin-base-latest', 'openshift-origin-source-latest', 'centos-centos-latest']
buildconfig.build.openshift.io "arrfab-nanoc-latest" created

buildconfig.build.openshift.io "openshift-origin-base-latest" created

buildconfig.build.openshift.io "openshift-origin-source-latest" created

buildconfig.build.openshift.io "centos-centos-latest" created

Processing weekly scan projects..
buildconfig.build.openshift.io "wscan-test-anomaly-latest" created

buildconfig.build.openshift.io "wscan-test-python-release" created

buildconfig.build.openshift.io "wscan-dharmit-freegeoip-latest" created

buildconfig.build.openshift.io "wscan-dharmit-nsq-latest" created

buildconfig.build.openshift.io "wscan-mattermost-mattermost-github-integration-92394f2" created

buildconfig.build.openshift.io "wscan-arrfab-nanoc-latest" created

buildconfig.build.openshift.io "wscan-openshift-origin-base-latest" created

buildconfig.build.openshift.io "wscan-openshift-origin-source-latest" created

buildconfig.build.openshift.io "wscan-centos-centos-latest" created
navidshaikh commented 6 years ago

This PR is rebased on #617, which needs to go first.

navidshaikh commented 6 years ago

Rebased and fixed the review comments.