search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.27k stars 541 forks source link

Cannot create nomad volume with ceph csi #3668

Closed acziryak closed 1 year ago

acziryak commented 1 year ago

Describe the bug

Creating a nomad volume with ceph csi results in state inconsistent, omap names mismatch error:

https://github.com/ceph/ceph-csi/blob/devel/internal/journal/voljournal.go#L369-L375

Error creating volume: Unexpected response code: 500 (1 error occurred:
        * controller create volume: rpc error: controller create volume: CSI.ControllerCreateVolume: controller plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (prometheus-us-ind-test) volume UUID (c11c65d7-a58f-11ed-b2a6-1e383fc3db06) volume omap name ())
I0213 17:13:58.665398       7 utils.go:206] ID: 43834 Req-ID: prometheus-us-ind-test GRPC request: {"accessibility_requirements":{},"capacity_range":{"limit_bytes":20000000000,"required_bytes":10000000000},"name":"prometheus-us-ind-test","parameters":{"clusterID":"******************","imageFeatures":"layering","pool":"ind-nonprod2"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["noatime"]}},"access_mode":{"mode":1}}]}
I0213 17:13:58.665928       7 rbd_util.go:1279] ID: 43834 Req-ID: prometheus-us-ind-test setting disableInUseChecks: false image features: [layering] mounter: rbd
I0213 17:13:58.669757       7 omap.go:88] ID: 43834 Req-ID: prometheus-us-ind-test got omap values: (pool="ind-nonprod2", namespace="", name="csi.volumes.ind-test-nomad-worker11-controller"): map[csi.volume.prometheus-us-ind-test:c11c65d7-a58f-11ed-b2a6-1e383fc3db06]
E0213 17:13:58.672481       7 omap.go:79] ID: 43834 Req-ID: prometheus-us-ind-test omap not found (pool="ind-nonprod2", namespace="", name="csi.volume.c11c65d7-a58f-11ed-b2a6-1e383fc3db06"): rados: ret=-2, No such file or directory
W0213 17:13:58.672920       7 voljournal.go:708] ID: 43834 Req-ID: prometheus-us-ind-test unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0213 17:13:58.673324       7 utils.go:210] ID: 43834 Req-ID: prometheus-us-ind-test GRPC error: rpc error: code = Internal desc = internal state inconsistent, omap names mismatch, request name (prometheus-us-ind-test) volume UUID (c11c65d7-a58f-11ed-b2a6-1e383fc3db06) volume omap name ()

Environment details

Steps to reproduce

Steps to reproduce the behavior:

  1. Setup details: Nomad cluster and ceph cluster.
  2. Deployment to trigger the issue: a nomad volume create
  3. See error

At this point I'm trying to figure out where in the stack this is a problem; if it's a user permission problem, or if ceph is corrupted, or if Nomad is having issues.

FWIW, I see this at the beginning of the node's logs:

E0208 15:34:58.994796       7 cephcsi.go:208] Failed to get the PID limit, can not reconfigure: open /sys/fs/cgroup//pids.max: no such file or directory
I0208 15:34:58.995028       7 cephcsi.go:241] Starting driver type: rbd with name: rbd.csi.ceph.com
I0208 15:34:58.995598       7 server.go:114] listening for CSI-Addons requests on address: &net.UnixAddr{Name:"/tmp/csi-addons.sock", Net:"unix"}
I0208 15:34:58.998531       7 mount_linux.go:283] Detected umount with safe 'not mounted' behavior
W0208 15:34:59.001081       7 rbd_attach.go:226] nbd modprobe failed (an error (exit status 1) occurred while running modprobe args: [nbd]): "modprobe: FATAL: Module nbd not found in directory /lib/modules/5.15.0-30-generic\n"

I'm not sure if that's related or not, especially because I see the volume get created on the Ceph cluster. However, I can't seem to get the omap values from that volume:

root@ind-ceph-mon1:~# rbd ls ind-nonprod2
csi.volume.c11c65d7-a58f-11ed-b2a6-1e383fc3db06
prometheus-us-ind-test
test_image
root@ind-ceph-mon1:~# rados -p ind-nonprod2 listomapkeys csi.volume.c11c65d7-a58f-11ed-b2a6-1e383fc3db06
error getting omap key set ind-nonprod2/csi.volume.c11c65d7-a58f-11ed-b2a6-1e383fc3db06: (2) No such file or directory

The comment by the error that I'm seeing is not promising: 

// NOTE: This should never be possible
Madhu-1 commented 1 year ago

@acziryak not sure how you ended in this case. Do you have any clear reproducer and complete logs to understand what went wrong?

acziryak commented 1 year ago

I can rebuild the Nomad cluster, but cannot rebuild the Ceph cluster. However, I can create a new pool/client for the ceph cluster.

Is there a way that I can manually test retrieving the omap values?

Madhu-1 commented 1 year ago

See if https://www.mrajanna.com/tracking-pv-rados-omap-in-cephcsi/ helps to track the omap values. some commands are kubernetes specific I think you can replace it with nomad

acziryak commented 1 year ago

So I tried to create a new volume which succeeded:

I0217 17:24:19.174991       7 omap.go:88] ID: 78467 Req-ID: agnostic2-us-ind-test got omap values: (pool="ind-nonprod2", namespace="", name="csi.volume.9f0760b0-aee6-11ed-96f6-ae765b511692"): map[csi.imageid:208269a29eef81 csi.imagename:csi-vol-9f0760b0-aee6-11ed-96f6-ae765b511692 csi.volname:agnostic2-us-ind-test]
I0217 17:24:19.216688       7 rbd_journal.go:337] ID: 78467 Req-ID: agnostic2-us-ind-test found existing volume (0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-9f0760b0-aee6-11ed-96f6-ae765b511692) with image name (csi-vol-9f0760b0-aee6-11ed-96f6-ae765b511692) for request (agnostic2-us-ind-test)
I0217 17:24:19.217186       7 utils.go:212] ID: 78467 Req-ID: agnostic2-us-ind-test GRPC response: {"volume":{"capacity_bytes":10737418240,"volume_context":{"clusterID":"1e35f6bc-1257-45b6-aa9d-16f9ecd30652","imageFeatures":"layering","imageName":"csi-vol-9f0760b0-aee6-11ed-96f6-ae765b511692","journalPool":"ind-nonprod2","pool":"ind-nonprod2"},"volume_id":"0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-9f0760b0-aee6-11ed-96f6-ae765b511692"}}

However, I cannot find the omap keys for that volume on my monitor node:

rados listomapkeys csi.volume.9f0760b0-aee6-11ed-96f6-ae765b511692
usage: rados [options] [commands]

This is per the troubleshooting page linked.

EDIT: It looks like it requires the pool name:

rados -p ind-nonprod2 listomapkeys csi.volume.9f0760b0-aee6-11ed-96f6-ae765b511692
csi.imageid
csi.imagename
csi.volname
Madhu-1 commented 1 year ago

yes it require pool name and also you can list values as well listomapvals if no problem exists can we close this one?

acziryak commented 1 year ago

Yes.

I will note down here that after I did a nomad system gc and nomad system reconcile summaries and a full stop and start of the CSI nodes and controllers, I was able to have the volume created. So whatever that inconsistent state error was must have been something that Nomad was caching somewhere that a garbage collection and restart of the jobs solved. Hope that might help whoever might find their way here later. And thank you for your willingness to help @Madhu-1 .

acziryak commented 1 year ago

This is happening again, and the above commands aren't fixing it.

I wonder if there's some cache somewhere that's not being cleared.