I will pause the normal schedule_and_publish pipeline schedule, and kick off a Buildkite build of this pipeline on this branch with no global timeout. It should try to go through each item in the queue, one at a time. If it can't get a compare result after 5 minutes (probably 4 tries of 60 seconds each) it will mark that notification as "finished" without posting a message, and say so in the logs.
Hopefully there's only a few problematic notification runs, and the rest of them work.
I forgot that we don't build and push an image for branches in this repo. I think it'll be easier to merge this, kick off the backfill, and then revert it.
This implements option (3) of https://github.com/voltrondata-labs/arrow-benchmarks-ci/issues/148#issuecomment-1644208839.
I will pause the normal
schedule_and_publish
pipeline schedule, and kick off a Buildkite build of this pipeline on this branch with no global timeout. It should try to go through each item in the queue, one at a time. If it can't get a compare result after 5 minutes (probably 4 tries of 60 seconds each) it will mark that notification as "finished" without posting a message, and say so in the logs.Hopefully there's only a few problematic notification runs, and the rest of them work.