search

ros2 / ci

ROS 2 CI Infrastructure
http://ci.ros2.org/
Apache License 2.0
48 stars 30 forks source link

Start nightly jobs at 4:00 UTC (21:00 PDT). #671

Closed nuclearsandwich closed 2 years ago

nuclearsandwich commented 2 years ago

Currently nightly jobs start at 7:30 UTC (00:30 PDT) but as the job counts and runtimes grow, this has started to spill over into the early morning.

Additionally, the build farmer report currently runs at 17:00 UTC (10:00 PDT) and the last builds are frequently still in progress when the report generates, meaning that feedback for a job can be delayed up to 24 hours since it won't be scanned until the next build report.

Moving the jobs 3.5 hours earlier should enable them to complete before the build report is generated, improving its utility.

nuclearsandwich commented 2 years ago

I had originally proposed this for starting an hour earlier at 3:00 UTC (20:00 PDT) but with US Daylight Savings Time that would start nightly builds at 19:00 PST, which I think cuts too close to the late shift for the US team.

gbiggs commented 2 years ago

21:00 PDT is OK with me, it's midday for east Asia and I've usually finished my package maintenance work by then.

clalancette commented 2 years ago

I've heard no objections in 2 days, so I would say go ahead with merging and deploying this.

tfoote commented 2 years ago

Historically there was a router reset on the HQ network that we made sure all the nightlies started after. I think that has been resolved with our improved networking hardware.

Edit: since @nuclearsandwich proposed this he's probably ok with it.

nuclearsandwich commented 2 years ago

Historically there was a router reset on the HQ network that we made sure all the nightlies started after. I think that has been resolved with our improved networking hardware.

Yeah we discussed that in the ROS 2 meeting this week as one of the reasons but the infrastructure we've had since this past winter no longer requires a nightly reset.

nuclearsandwich commented 2 years ago

In an extremely Friday turn of events, although I updated the PR description to use 4am UTC I did not push the updated commit so the change that I just merged, and subsequently deployed, was for 3am UTC.

nuclearsandwich commented 2 years ago

Fixed with https://github.com/ros2/ci/pull/673