Closed smithfarm closed 4 years ago
@kshtsk tells me that this failure has been keeping the devel-storage-5.0 CI red for a month now. Can we raise the priority?
I just ran salt-run state.orch ceph.functests.1node.remove
successfully. Is this environment with the failure accessible?
@swiftgist The CI always destroys the environment. Reproducing would involve using the same command (salt-run state.orch ceph.functests.1node
) in the same environment (SLE-15-SP1, deepsea 0.8.12, single-node cluster). Is that what you did?
I ran salt-run state.orch ceph.functests.1node.remove
. I do have a multinode cluster. How many OSDs do you have in your single node cluster? The multiple OSD removal test removes both OSD 0 and 1.
we actually try and deploy the system
@swiftgist Possibly you are not only running on a multinode cluster, but you might also be running on a newer version of DeepSea - I see the same test succeeds in the CI when run on the tip of the SES5
branch (v0.8.12-12-g162e1139
).
@jschmid1 Can we get an incremental release pushed to Devel:Storage:5.0 to fix the breakage there?
found the fix after experimenting on the teuthology VM. It's already in #1724.
$ git describe 6dff6ca
v0.8.12-2-g6dff6ca9
@swiftgist Right. Devel:Storage:5.0 still has DeepSea version v0.8.12-0-e885d513e
, hence the CI breakage and this bug report. Thanks for fixing it and looking into it.
The moral of this story, I guess, is to make sure the CI is passing on the tip of the release branch before cutting a release.
Also, after pushing to Devel:Storage:*
I do not submit to maintenance immediately, but rather wait for a CI run in the devel CI and only submit once the CI is green with the new release.
This functest (
remove.osd
) is failing reproducibly in the SES5 CI (DeepSea 0.8.12 in SLE-12-SP3).Relevant portion of the log DeepSea
The functests fail like so:
Full log: http://ci.ses.suse.de:8080/job/storage-ci-teuthology-suite-devel-5.0/292/testReport/teuthology/Devel/suse_tier1_functional__0_salt_yaml_1_deepsea_yaml_2_services_and_test_yaml_ceph_cm_salt_yaml_cluster__4disks_yaml_roles_yaml__deepsea_cli_off_yaml_distros_sle_12_3_yaml_/attachments/teuthology-suse-1311.log