CI failure: Module 'mirroring' is not enabled

phlogistonjohn commented 3 years ago

We're seeing the following test flake somewhat regularly now:

=== RUN   TestParsePeerList/ok
--- PASS: TestParsePeerList (0.00s)
    --- PASS: TestParsePeerList/error (0.00s)
    --- PASS: TestParsePeerList/statusSet (0.00s)
    --- PASS: TestParsePeerList/ok (0.00s)
=== RUN   TestMirroring
    mirror_workflow_test.go:62: 
            Error Trace:    mirror_workflow_test.go:62
            Error:          Received unexpected error:
                            rados: ret=-95, Operation not supported: "Module 'mirroring' is not enabled (required by command 'fs snapshot mirror enable'): use `ceph mgr module enable mirroring` to enable it"
            Test:           TestMirroring
--- FAIL: TestMirroring (12.85s)

I suspect that the API that enables modules doesn't guarantee callers will see the module as enabled at return time, thus producing a race between us activating the module and then immediately trying to use it before the mgr is ready to serve requests that make use of the modue.

phlogistonjohn commented 3 years ago

This bug keeps biting our ci and causing test flakes. This should be a high priority task for the v0.13.0 cycle.

phlogistonjohn commented 2 years ago

https://github.com/ceph/go-ceph/runs/4128633714?check_suite_focus=true

This just occurred again in the daily CI run. This means the workaround was not sufficient to avoid the problem.

phlogistonjohn commented 2 years ago

PR #610 resolves this.

ceph / go-ceph

CI failure: Module 'mirroring' is not enabled #574