Input/Output Errors When Reading/Writing Files After Successfully Mounting CephFS Using go-ceph/cephfs

[herenli]

Hello, I am encountering input/output errors when trying to read and write files after successfully mounting the Ceph File System (CephFS) using the go-ceph/cephfs library.

Context: I have successfully mounted the CephFS manually, and I can create directories and files without any issues. However, when using the go-ceph/cephfs interface to perform read and write operations, I encounter input/output errors.

Error Output: go run -tags luminous test.go Success creating CephFS mount Success reading default config file Success mounting CephFS Directory created successfully. Error writing to file: cephfs: ret=5, Input/output error

Details: go-ceph v0.4.0 ceph luminous

Could anyone provide insights or suggestions on why this might be happening? Any help would be greatly appreciated! Thank you!

the code is attached. code.md

[nixpanic]

go-ceph v0.4.0 ceph luminous

Wow, these are both quite old and not maintained anymore.

There is a very similar existing test:

https://github.com/ceph/go-ceph/blob/2c8601001720606bf444453aa2448065d64ada87/cephfs/file_test.go#L80-L96

I have added an extra mount.MakeDir() and put the fname in the new directory, like this:

    t.Run("writeAndReadInDir", func(t *testing.T) {
        dirname := "/TestDir"
        dfname := dirname + "/" + fname

        err := mount.MakeDir(dirname, 0777)
        assert.NoError(t, err)
        defer func() { assert.NoError(t, mount.RemoveDir(dirname)) }()

        // idempotent open for read and write
        f1, err := mount.Open(dfname, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
        n, err := f1.Write([]byte("yello world!"))
        assert.NoError(t, err)
        assert.EqualValues(t, 12, n)
        err = f1.Close()
        assert.NoError(t, err)
        defer func() { assert.NoError(t, mount.Unlink(dfname)) }()

        buf := make([]byte, 1024)
        f2, err := mount.Open(dfname, os.O_RDONLY, 0)
        n, err = f2.Read(buf)
        assert.NoError(t, err)
        assert.EqualValues(t, 12, n)
        assert.Equal(t, "yello world!", string(buf[:n]))
    })

After that, running make test-container passes the new TestFileReadWrite/writeAndReadInDir test.

=== RUN   TestFileReadWrite
=== RUN   TestFileReadWrite/writeAndRead
=== RUN   TestFileReadWrite/writeAndReadInDir
=== RUN   TestFileReadWrite/openForWriteOnly
=== RUN   TestFileReadWrite/openForReadOnly
=== RUN   TestFileReadWrite/uninitializedFile
--- PASS: TestFileReadWrite (0.01s)
    --- PASS: TestFileReadWrite/writeAndRead (0.00s)
    --- PASS: TestFileReadWrite/writeAndReadInDir (0.00s)
    --- PASS: TestFileReadWrite/openForWriteOnly (0.00s)
    --- PASS: TestFileReadWrite/openForReadOnly (0.00s)
    --- PASS: TestFileReadWrite/uninitializedFile (0.00s)

This probably indicates a permission or other configuration mistake on the Ceph cluster. Maybe logs from the MDS or OSD processes give a hint on what could be wrong?

Have you used the same credentials for mounting with kernel or fuse CephFS, and did that work?

[phlogistonjohn]

I'll 2nd the "wow that's really old" comment from @nixpanic - it's so old that I hesitate to invest a lot of effort here. Please consider upgrading to something more recent & supported by the upstream projects.

I will add that ceph has various debugging option that can be loaded via a ceph.conf or config set functions. See https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/ The Ceph libraries will then emit debug logging (to the standard error IIRC).

[phlogistonjohn]

Oh, I should add I generally find it simpler to turn them all on to level 10. It produces a huge amount of output but I never seem to be able to guess the proper subsystem name for what I am looking for and so I just prefer to cast a wide net.

[herenli]

@nixpanic Thank you for your reply. I did a similar existing test, but still have an error

--- FAIL: TestFileReadWrite (0.07s)
    --- FAIL: TestFileReadWrite/writeAndRead (0.01s)
    --- PASS: TestFileReadWrite/openForWriteOnly (0.00s)
    --- PASS: TestFileReadWrite/openForReadOnly (0.00s)
    --- PASS: TestFileReadWrite/uninitializedFile (0.00s)

And I used ceph-fuse to mount the cephfs manually and can read/write the file successfully. So, I don't know the reasons.

[herenli]

@phlogistonjohn Thank you for your reply. I updated the go-ceph version to v0.11 for using of nautilus. But I still have the input/output errors when reading/writing files after successfully mounting cephfs using go-ceph/cephfs.

=== RUN   TestFileReadWrite/writeAndRead
    cephfs_test.go:336: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:336
                Error:          Received unexpected error:
                                cephfs: ret=-5, Input/output error
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:337: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:337
                Error:          Not equal: 
                                expected: 12
                                actual  : 0
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:345: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:345
                Error:          Received unexpected error:
                                cephfs: ret=-5, Input/output error
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:346: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:346
                Error:          Not equal: 
                                expected: 12
                                actual  : 0
                Test:           TestFileReadWrite/writeAndRead
    cephfs_test.go:347: 
                Error Trace:    /home/lhrtest/rclone/rclone/vendor/github.com/ceph/go-ceph/cephfs/cephfs_test.go:347
                Error:          Not equal: 
                                expected: "yello world!"
                                actual  : ""

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -yello world!
                                +
                Test:           TestFileReadWrite/writeAndRead