Hard 1 hour time limit even if no workers are available

ipfs-inactive / dev-team-enablement

[ARCHIVED] Dev Team Enablement Working Group

3 stars 1 forks source link

Hard 1 hour time limit even if no workers are available #140

Closed alanshaw closed 6 years ago

alanshaw commented 6 years ago

Is there a way to enforce the time limit after all the jobs in the pipeline have been assigned a worker? When CI is busy it can take ages to get a worker and so the run fails. It wastes resources and my time having to retstart it again.

magik6k commented 6 years ago

Jobs shouldn't timeout on waiting for workers, where does it happen?

magik6k commented 6 years ago

We could also try implementing dynamic worker scaling, trading lack of queues for slightly longer job startup (likely using https://wiki.jenkins.io/display/JENKINS/JClouds+Plugin)

alanshaw commented 6 years ago

@magik6k I can't really send you a link to an example as the worker becomes available and the tests start but then get killed after the run takes more than an hour.

alanshaw commented 6 years ago

Ok, well this is probably an instance where this happened: https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs-ipfs/detail/PR-1576/1/pipeline/16