search

JuliaCI / julia-buildkite

Buildkite configuration files for Base Julia CI
8 stars 15 forks source link

Don't allow failure in FreeBSD test jobs #384

Closed ararslan closed 2 months ago

ararslan commented 2 months ago

The issues with the CI setup that had made it flaky for a while have been fixed. But because the tests are allowed to fail, multiple regressions have snuck in. Once https://github.com/JuliaLang/julia/pull/55594 lands, things will once again be in a stable, working state, so we should require tests to pass to avoid further regressions.

giordano commented 2 months ago

Is this segfault

[1053] signal 11 (1): Segmentation fault
in expression starting at /usr/home/julia/.buildkite-agent/builds/freebsd13-amdci6-3/julialang/julia-master/julia-185c2206dc/share/julia/test/ccall.jl:1059
Allocations: 31845368 (Pool: 31844755; Big: 613); GC: 309

fixed by https://github.com/JuliaLang/julia/pull/55594 or it's another flaky failure?

ararslan commented 2 months ago

Should be fixed by the linked PR. (EDIT: Not should be, is)

ararslan commented 2 months ago

The PR that fixed CI has been merged, so FreeBSD tests are once again green. I don't understand the status reported by the julia-buildkite CI job here though, since AFAICT the only failures are ones that are explicitly allowed. Or am I missing something?

ararslan commented 2 months ago

Bump, FreeBSD CI has been green consistently since that PR was merged. Not sure what to do about the failing required CI check here though; it looks unrelated.

staticfloat commented 2 months ago

@DilumAluthge can you figure out why this can't be merged? TBH I find myself fighting the "required status checks" quite often.

DilumAluthge commented 2 months ago

Looks like a new CI job is running. I can take a look when that job finishes.

DilumAluthge commented 2 months ago

TBH I find myself fighting the "required status checks" quite often.

Yeah, it is definitely annoying.

But at this point, we've had multiple instances of PRs on this repo being merged with true CI failures, which instantly breaks all CI jobs on the JuliaLang/julia repo. So, because of the blast radius, I think we unfortunately have to live with the required status checks here, as the lesser of two evils.

DilumAluthge commented 2 months ago

Alright, I see one test job failing. I have manually clicked the "retry" job.

DilumAluthge commented 2 months ago

The red X here is indeed a true failure, and it represents a bug in this PR.

From https://buildkite.com/julialang/julia-buildkite/builds/1773#0192023a-0926-4fa2-a1df-ba440f88c184:

Step dependencies not found

The following step dependencies were not found during the build:

test_x86_64-unknown-freebsd Check the pipeline is configured correctly, or > see the documentation.

So this is a true failure, and we shouldn't merge this PR until we fix it.

@staticfloat Can you take a look? Basically, our job dependency graph is incorrect, so one of the jobs (the FreeBSD upload job) has an invalid dependency graph, and thus can't run.

ararslan commented 2 months ago

Ah my b, I think I need to rearrange where things are listed in the pipeline launching.

DilumAluthge commented 2 months ago

Okay, looks like you've fixed it!

DilumAluthge commented 2 months ago

The Build, Check, and Test status checks are all green.

And, importantly, the buildkite/julia-buildkite status check is now green.