search

openebs / mayastor

Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.
Apache License 2.0
753 stars 109 forks source link

Csi attacher error: children of nexus have mixed block sizes #1773

Open marcinpro1 opened 4 days ago

marcinpro1 commented 4 days ago

Cannot attach created pvc volume to pod.

Storage made on 3 nodes with 1 nvme (same model, same vendor) disk/diskpool per node

Logs from agent core:

2024-11-21T12:53:24.517428Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:24 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(d0c0f79c-e849-4b4a-9b36-2e43e027eb56, "d0c0f79c-e849-4b4a-9b36-2e43e027eb56"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 11184177593679604566, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:24.524510Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:24 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:25.519938Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:25 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(69a3ef6e-d724-415f-991e-79d8bf13e952, "69a3ef6e-d724-415f-991e-79d8bf13e952"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 11033390108929943890, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:25.526347Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:25 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:29.671206Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:29 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(1d278a42-e0fc-4be3-9f05-46bdce0594f1, "1d278a42-e0fc-4be3-9f05-46bdce0594f1"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 11458642607933330673, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:29.677982Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:29 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:37.834329Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:37 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(5bd13b04-1fce-48a2-92d3-f21449ffda8a, "5bd13b04-1fce-48a2-92d3-f21449ffda8a"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 10580066118578199178, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:37.841643Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:37 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:53.992407Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:53 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(c456c8dc-034e-45d9-b23e-73312d447a9e, "c456c8dc-034e-45d9-b23e-73312d447a9e"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 12843829842357287582, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:53:53.999512Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:53:53 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:54:22.523985Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:54:22 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(823bb4b1-1983-47e3-a00e-6cf379381ce5, "823bb4b1-1983-47e3-a00e-6cf379381ce5"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 11533275488708992229, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:54:22.530406Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:54:22 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:54:26.153607Z ERROR core::controller::io_engine::v1::nexus: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:54:26 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:89
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::create with request: CreateNexus { node: NodeId("nvme-3"), uuid: NexusId(aefa892f-a885-4136-951a-c15ae38c0bef, "aefa892f-a885-4136-951a-c15ae38c0bef"), size: 53687091200, children: [Replica(ReplicaUri { uuid: ReplicaId(aaea006f-cdee-45e0-a4e2-e8c5cf88bc26, "aaea006f-cdee-45e0-a4e2-e8c5cf88bc26"), share_uri: ChildUri("bdev:///aaea006f-cdee-45e0-a4e2-e8c5cf88bc26?uuid=aaea006f-cdee-45e0-a4e2-e8c5cf88bc26") }), Replica(ReplicaUri { uuid: ReplicaId(83537695-938f-4e6c-a69d-cb00a2c202be, "83537695-938f-4e6c-a69d-cb00a2c202be"), share_uri: ChildUri("nvmf://10.32.46.68:8420/nqn.2019-05.io.openebs:83537695-938f-4e6c-a69d-cb00a2c202be?uuid=83537695-938f-4e6c-a69d-cb00a2c202be") }), Replica(ReplicaUri { uuid: ReplicaId(17301ff8-885f-4a6e-93f6-65413314f917, "17301ff8-885f-4a6e-93f6-65413314f917"), share_uri: ChildUri("nvmf://10.32.46.73:8420/nqn.2019-05.io.openebs:17301ff8-885f-4a6e-93f6-65413314f917?uuid=17301ff8-885f-4a6e-93f6-65413314f917") })], managed: true, owner: Some(VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5")), config: Some(NexusNvmfConfig { controller_id_range: NvmfControllerIdRange(1..=2), reservation_key: 10744112457154563055, reservation_type: ExclusiveAccess, preempt_policy: Holder }) }
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:54:26.160537Z ERROR core::volume::service: error: gRPC request 'create_nexus' for 'Nexus' failed with 'status: InvalidArgument, message: "Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 21 Nov 2024 12:54:26 GMT", "content-length": "0"} }'
    at control-plane/agents/src/bin/core/volume/service.rs:379
    in core::volume::service::publish_volume with request: PublishVolume { uuid: VolumeId(6678e69e-f60a-4b18-8e2d-6092cbfae6e5, "6678e69e-f60a-4b18-8e2d-6092cbfae6e5"), target_node: Some(NodeId("nvme-3")), share: Some(Nvmf), publish_context: {}, frontend_nodes: ["nvme-3"] }, volume.uuid: 6678e69e-f60a-4b18-8e2d-6092cbfae6e5
  2024-11-21T12:55:29.483001Z  WARN core::controller::reconciler::nexus::garbage_collector: Attempting to destroy nexus
    at control-plane/agents/src/bin/core/controller/reconciler/nexus/garbage_collector.rs:161
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: 039cba86-e042-47ac-98eb-b7204682def0, request.reconcile: true
  2024-11-21T12:55:29.488463Z  WARN core::controller::io_engine::v1::nexus: Trying to destroy nexus which is already destroyed, nexus.uuid: 039cba86-e042-47ac-98eb-b7204682def0
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:108
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::destroy with request: DestroyNexus { node: NodeId("nvme-3"), uuid: NexusId(039cba86-e042-47ac-98eb-b7204682def0, "039cba86-e042-47ac-98eb-b7204682def0"), disowners: All, lazy: false }
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: 039cba86-e042-47ac-98eb-b7204682def0, request.reconcile: true
  2024-11-21T12:55:29.488532Z  INFO core::controller::resources::operations_helper: complete_destroy, val: ()
    at control-plane/agents/src/bin/core/controller/resources/operations_helper.rs:382
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: 039cba86-e042-47ac-98eb-b7204682def0, request.reconcile: true
  2024-11-21T12:55:29.491095Z  INFO core::controller::reconciler::nexus::garbage_collector: Successfully destroyed nexus
    at control-plane/agents/src/bin/core/controller/reconciler/nexus/garbage_collector.rs:168
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: 039cba86-e042-47ac-98eb-b7204682def0, request.reconcile: true
  2024-11-21T12:55:29.491163Z  WARN core::controller::reconciler::nexus::garbage_collector: Attempting to destroy nexus
    at control-plane/agents/src/bin/core/controller/reconciler/nexus/garbage_collector.rs:161
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: d0c0f79c-e849-4b4a-9b36-2e43e027eb56, request.reconcile: true
  2024-11-21T12:55:29.494665Z  WARN core::controller::io_engine::v1::nexus: Trying to destroy nexus which is already destroyed, nexus.uuid: d0c0f79c-e849-4b4a-9b36-2e43e027eb56
    at control-plane/agents/src/bin/core/controller/io_engine/v1/nexus.rs:108
    in core::controller::io_engine::v1::nexus::rpc::v1::nexus::destroy with request: DestroyNexus { node: NodeId("nvme-3"), uuid: NexusId(d0c0f79c-e849-4b4a-9b36-2e43e027eb56, "d0c0f79c-e849-4b4a-9b36-2e43e027eb56"), disowners: All, lazy: false }
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: d0c0f79c-e849-4b4a-9b36-2e43e027eb56, request.reconcile: true
  2024-11-21T12:55:29.494723Z  INFO core::controller::resources::operations_helper: complete_destroy, val: ()
    at control-plane/agents/src/bin/core/controller/resources/operations_helper.rs:382
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: d0c0f79c-e849-4b4a-9b36-2e43e027eb56, request.reconcile: true
  2024-11-21T12:55:29.497227Z  INFO core::controller::reconciler::nexus::garbage_collector: Successfully destroyed nexus
    at control-plane/agents/src/bin/core/controller/reconciler/nexus/garbage_collector.rs:168
    in core::controller::reconciler::nexus::garbage_collector::destroy_deleting_nexus with nexus.uuid: d0c0f79c-e849-4b4a-9b36-2e43e027eb56, request.reconcile: true
  2024-11-21T12:55:29.497289Z  WARN core::controller::reconciler::nexus::garbage_collector: Attempting to destroy nexus
    at control-plane/agents/src/bin/core/controller/reconciler/nexus/garbage_collector.rs:161

Describe from pod:

  Warning  FailedAttachVolume  3m6s                 attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6678e69e-f60a-4b18-8e2d-6092cbfae6e5" : rpc error: code = InvalidArgument desc = error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_nexus::status: InvalidArgument, message: \"Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 21 Nov 2024 12:53:24 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
  Warning  FailedAttachVolume  3m3s (x2 over 3m5s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6678e69e-f60a-4b18-8e2d-6092cbfae6e5" : rpc error: code = InvalidArgument desc = error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_nexus::status: InvalidArgument, message: \"Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 21 Nov 2024 12:53:25 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
  Warning  FailedAttachVolume  2m56s (x2 over 3m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6678e69e-f60a-4b18-8e2d-6092cbfae6e5" : rpc error: code = InvalidArgument desc = error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_nexus::status: InvalidArgument, message: \"Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 21 Nov 2024 12:53:29 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
  Warning  FailedAttachVolume  2m47s                attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6678e69e-f60a-4b18-8e2d-6092cbfae6e5" : rpc error: code = InvalidArgument desc = error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_nexus::status: InvalidArgument, message: \"Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 21 Nov 2024 12:53:37 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
  Warning  FailedAttachVolume  2m31s                attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6678e69e-f60a-4b18-8e2d-6092cbfae6e5" : rpc error: code = InvalidArgument desc = error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_nexus::status: InvalidArgument, message: \"Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 21 Nov 2024 12:53:53 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
  Warning  FailedMount         63s                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume kube-api-access-6h6n4]: timed out waiting for the condition
  Warning  FailedAttachVolume  53s (x2 over 118s)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-6678e69e-f60a-4b18-8e2d-6092cbfae6e5" : rpc error: code = InvalidArgument desc = error in response: status code '400 Bad Request', content: 'RestJsonError { details: "create_nexus::status: InvalidArgument, message: \"Children of nexus 6678e69e-f60a-4b18-8e2d-6092cbfae6e5 have mixed block sizes\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 21 Nov 2024 12:54:26 GMT\", \"content-length\": \"0\"} }", message: "SvcError::GrpcRequestError", kind: InvalidArgument }'
tiagolobocastro commented 4 days ago

hmm that error message is not very helpful tbh

@dsharma-dc the page-size we get from the blockbstore lvols is 4k right? Not sure how we'd get different block-size on the replicas...

@marcinpro1, could you please run this command on each node's backend device, example: sudo blockdev --report /dev/nvme0n1

marcinpro1 commented 4 days ago

nvme-1:

[root@nvme-1 ~]# blockdev --report /dev/nvme2n1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  8192   512  4096          0   1920383410176   /dev/nvme2n1

nvme-2:

[root@nvme-2 ~]# blockdev --report /dev/nvme3n1
RO    RA   SSZ   BSZ        StartSec            Size   Device
rw  8192   512  4096               0   1920383410176   /dev/nvme3n1

nvme-3:

[root@nvme-3 ~]# blockdev --report /dev/nvme2n1
RO    RA   SSZ   BSZ        StartSec            Size   Device
rw  8192   512  4096               0   1920383410176   /dev/nvme2n1
dsharma-dc commented 3 days ago

@tiagolobocastro The block size here is taken what's set in the replica namespace apparently. So perhaps replica namespace hasn't got populated correctly here. Could you provide the complete logs, primarily io-engine logs? Is the issue still persisting or was it transient?

marcinpro1 commented 3 days ago

Issue still persists. Here you go: openebs_dump.tar.gz