search

ceph / ceph-nvmeof

Service to provide Ceph storage over NVMe-oF/TCP protocol
GNU Lesser General Public License v3.0
85 stars 46 forks source link

[HA] All nvmeof GWs crash while setting ana_state failed to all ana_groups #539

Closed rahullepakshi closed 6 months ago

rahullepakshi commented 7 months ago

Build details

ceph - [quay.io/roysahar-ibm/ceph:bf9505fb569e9b95a78f9700ed8c4bd20508ef55](http://quay.io/roysahar-ibm/ceph:bf9505fb569e9b95a78f9700ed8c4bd20508ef55)
nvmeof-gw - [quay.io/barakda1/nvmeof:qe_ceph_devel_21e59b2](http://quay.io/barakda1/nvmeof:qe_ceph_devel_21e59b2)
[quay.io/barakda1/nvmeof-cli:qe_ceph_devel_21e59b2](http://quay.io/barakda1/nvmeof-cli:qe_ceph_devel_21e59b2)

This is scale setup hence I allowed maximum namespaces for each subsystem to 5000 but I only created 4 while I observed this GW failure.

What is strange is that we are setting ana_state for 5000 ana_groups though we have only 2 GWs. Why should we do this? As per logs issue caused while setting ana_stae at _ana_state='inaccessible' anagrpid=1322

Earlier below is the state

[ceph: root@tala014 /]# ceph nvme-gw show nvmeof ''
{
    "pool": "nvmeof",
    "group": "",
    "num gws": 2,
    "Anagrp list": "[ 2 1 ]"
}
{
    "gw-id": "client.nvmeof.nvmeof.argo023.lxayph",
    "anagrp-id": 2,
    "Availability": "AVAILABLE",
    "ana states": " 2: ACTIVE , 1: STANDBY ,"
}
{
    "gw-id": "client.nvmeof.nvmeof.argo024.ivonyw",
    "anagrp-id": 1,
    "Availability": "AVAILABLE",
    "ana states": " 2: STANDBY , 1: ACTIVE ,"
}

[root@argo024 ~]# nvmf --server-address 10.8.128.224 --server-port 5500 subsystem list
Subsystems:
╒═══════════╤════════════════════════════╤════════════╤════════════════════╤══════════════════╤═════════════╤══════════════╕
│ Subtype   │ NQN                        │ HA State   │ Serial             │ Controller IDs   │   Namespace │          Max │
│           │                            │            │ Number             │                  │       Count │   Namespaces │
╞═══════════╪════════════════════════════╪════════════╪════════════════════╪══════════════════╪═════════════╪══════════════╡
│ NVMe      │ nqn.2016-06.io.spdk:cnode1 │ enabled    │ Ceph97367748967584 │ 1-2040           │           3 │         5000 │
├───────────┼────────────────────────────┼────────────┼────────────────────┼──────────────────┼─────────────┼──────────────┤
│ NVMe      │ nqn.2016-06.io.spdk:cnode2 │ enabled    │ Ceph8951247352807  │ 1-2040           │           0 │         5000 │
╘═══════════╧════════════════════════════╧════════════╧════════════════════╧══════════════════╧═════════════╧══════════════╛

Later, all GWs failed

[28-Mar-2024 06:04:15] INFO grpc.py:2068: create_listener nvmf_subsystem_listener_set_ana_state response ret=True
[28-Mar-2024 06:04:15] INFO grpc.py:2056: using ana_map: set listener on nqn : nqn.2016-06.io.spdk:cnode1  ana state : inaccessible for group : 1322
[28-Mar-2024 06:04:15] INFO grpc.py:2058: create_listener nvmf_subsystem_listener_set_ana_state request=nqn: "nqn.2016-06.io.spdk:cnode1"
host_name: "argo024"
traddr: "10.8.128.224"
adrfam: ipv4
trsvcid: 4420
 _ana_state='inaccessible' anagrpid=1322
[28-Mar-2024 06:04:15] INFO grpc.py:2068: create_listener nvmf_subsystem_listener_set_ana_state response ret=True
[28-Mar-2024 06:04:15] INFO grpc.py:2056: using ana_map: set listener on nqn : nqn.2016-06.io.spdk:cnode1  ana state : inaccessible for group : 1323
[28-Mar-2024 06:04:15] INFO grpc.py:2058: create_listener nvmf_subsystem_listener_set_ana_state request=nqn: "nqn.2016-06.io.spdk:cnode1"
host_name: "argo024"
traddr: "10.8.128.224"
adrfam: ipv4
trsvcid: 4420
 _ana_state='inaccessible' anagrpid=1323
[2024-03-28 06:04:15.492787] tcp.c:2329:nvmf_tcp_req_parse_sgl: *ERROR*: SGL length 0x2b110 exceeds max io size 0x20000
[2024-03-28 06:04:15.492866] tcp.c:2329:nvmf_tcp_req_parse_sgl: *ERROR*: SGL length 0x2b110 exceeds max io size 0x20000
[2024-03-28 06:04:15.492889] tcp.c:1452:nvmf_tcp_qpair_set_recv_state: *ERROR*: The recv state of tqpair=0x296bac0 is same with the state(5) to be set
nvmf_tgt: /src/include/spdk_internal/sock.h:118: spdk_sock_request_queue: Assertion `req->internal.curr_list == NULL' failed.
[28-Mar-2024 06:04:15] ERROR grpc.py:2072: Failure adding nqn.2016-06.io.spdk:cnode1 listener at 10.8.128.224:4420: Error setting ANA state
Traceback (most recent call last):
  File "/src/control/grpc.py", line 2059, in create_listener_safe
    ret = rpc_nvmf.nvmf_subsystem_listener_set_ana_state(
  File "/usr/lib/python3.9/site-packages/spdk/rpc/nvmf.py", line 349, in nvmf_subsystem_listener_set_ana_state
    return client.call('nvmf_subsystem_listener_set_ana_state', params)
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
    response = self.recv()
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 165, in recv
    newdata = self.sock.recv(4096)
ConnectionResetError: [Errno 104] Connection reset by peer
[28-Mar-2024 06:04:15] ERROR server.py:40: GatewayServer: SIGCHLD received signum=17
[28-Mar-2024 06:04:15] INFO grpc.py:2287: Received request to get subsystems, context: <grpc._server._Context object at 0x7f994a19a3a0>
[28-Mar-2024 06:04:15] ERROR grpc.py:2293: get_subsystems failed
Traceback (most recent call last):
  File "/src/control/grpc.py", line 2290, in get_subsystems_safe
    ret = rpc_nvmf.nvmf_get_subsystems(self.spdk_rpc_client)
  File "/usr/lib/python3.9/site-packages/spdk/rpc/nvmf.py", line 177, in nvmf_get_subsystems
    return client.call('nvmf_get_subsystems', params)
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 185, in call
    req_id = self.send(method, params)
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 144, in send
    self.flush()
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 140, in flush
    self.sock.sendall(reqstr.encode("utf-8"))
BrokenPipeError: [Errno 32] Broken pipe
.
.
.

2024-03-28T06:04:15.929+0000 7fe27463f640  0 nvmeofgw void NVMeofGwMonitorClient::handle_nvmeof_gw_map(ceph::ref_t<MNVMeofGwMap>) GRPC set_ana_state failed
2024-03-28T06:04:15.933+0000 7fe27463f640  0 nvmeofgw void NVMeofGwMonitorClient::handle_nvmeof_gw_map(ceph::ref_t<MNVMeofGwMap>) GRPC set_ana_state failed
2024-03-28T06:04:15.936+0000 7fe27463f640  0 nvmeofgw void NVMeofGwMonitorClient::handle_nvmeof_gw_map(ceph::ref_t<MNVMeofGwMap>) GRPC set_ana_state failed
2024-03-28T06:04:15.939+0000 7fe27463f640  0 nvmeofgw void NVMeofGwMonitorClient::handle_nvmeof_gw_map(ceph::ref_t<MNVMeofGwMap>) GRPC set_ana_state failed
.
[28-Mar-2024 06:04:16] INFO server.py:377: Aborting (client.nvmeof.nvmeof.argo024.ivonyw) pid 93...
[28-Mar-2024 06:04:16] INFO server.py:106: Stopping the server...
[28-Mar-2024 06:04:16] INFO server.py:409: Terminating discovery service...
[28-Mar-2024 06:04:16] INFO server.py:416: Discovery service terminated
[28-Mar-2024 06:04:16] INFO server.py:113: Exiting the gateway process.
[28-Mar-2024 06:04:16] INFO utils.py:391: Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvmeof.argo024.ivonyw/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvmeof.argo024.ivonyw/nvmeof-log.gz
Gateway subprocess terminated pid=93 exit_code=-6
podman[890242]: 2024-03-28 06:04:16.149102295 +0000 UTC m=+0.034092956 container died 35b3c43fb64c373df8b0fbd38f62402003afe006c951626890ab27343962246d (image=quay.io/barakda1/nvmeof:qe_ceph_devel_21e59b2, name=ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458-nvmeof-nvmeof-argo024-ivonyw, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, version=1.1.0, name=ceph-nvmeof, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers <dev@ceph.io>, io.ceph.nvmeof=None, io.spdk.build.pkgdep-args=--rbd, io.spdk.build.target-arch=, distribution-scope=public, release=, io.spdk.build.ceph-release=18.2.2, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.version=23.01.1, io.spdk.git.repo=https://github.com/ceph/spdk.git, vcs-ref=21e59b2fb3a6ffd41c9aa2b981db0415f3d8768c, io.ceph.maintainer=Ceph Developers <dev@ceph.io>, org.centos.version=9.0-21.el9, io.ceph.component=ceph-nvmeof, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.version=1.1.0, vcs-type=git, io.ceph.git.commit=21e59b2fb3a6ffd41c9aa2b981db0415f3d8768c, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, url=https://github.com/ceph/ceph-nvmeof, architecture=x86_64, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.branch=ceph-nvmeof, build-date=2024-03-26 07:40:14 UTC, io.openshift.wants=ceph,rbd, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Ceph NVMe over Fabrics Gateway, io.buildah.version=1.29.0, vendor=Red Hat, Inc., io.spdk.git.commit=85a10c98e70a1d3bc6c166e2f2939b3e5bae37dc, io.ceph.git.branch=devel, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.makeflags=, com.redhat.component=ubi9-container)
podman[890242]: 2024-03-28 06:04:16.169668612 +0000 UTC m=+0.054659247 container remove 35b3c43fb64c373df8b0fbd38f62402003afe006c951626890ab27343962246d (image=quay.io/barakda1/nvmeof:qe_ceph_devel_21e59b2, name=ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458-nvmeof-nvmeof-argo024-ivonyw, io.openshift.wants=ceph,rbd, io.spdk.build.ceph-release=18.2.2, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.maintainer=Ceph Developers <dev@ceph.io>, vendor=Red Hat, Inc., io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.repo=https://github.com/ceph/spdk.git, distribution-scope=public, vcs-type=git, release=, io.ceph.nvmeof=None, io.spdk.git.branch=ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.git.commit=85a10c98e70a1d3bc6c166e2f2939b3e5bae37dc, io.buildah.version=1.29.0, com.redhat.component=ubi9-container, vcs-ref=21e59b2fb3a6ffd41c9aa2b981db0415f3d8768c, io.ceph.component=ceph-nvmeof, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2024-03-26 07:40:14 UTC, io.spdk.build.target-arch=, version=1.1.0, io.spdk.build.pkgdep-args=--rbd, io.ceph.git.commit=21e59b2fb3a6ffd41c9aa2b981db0415f3d8768c, url=https://github.com/ceph/ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.version=1.1.0, architecture=x86_64, io.ceph.git.branch=devel, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, summary=Ceph NVMe over Fabrics Gateway, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., maintainer=Ceph Developers <dev@ceph.io>, io.spdk.version=23.01.1, io.spdk.build.makeflags=, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, org.centos.version=9.0-21.el9, name=ceph-nvmeof)
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Failed with result 'exit-code'.
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Consumed 56.156s CPU time.
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Scheduled restart job, restart counter is at 6.
systemd[1]: Stopped Ceph nvmeof.nvmeof.argo024.ivonyw for 3f3f4c74-eb6c-11ee-afa3-b4835101e458.
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Consumed 56.156s CPU time.
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Start request repeated too quickly.
systemd[1]: ceph-3f3f4c74-eb6c-11ee-afa3-b4835101e458@nvmeof.nvmeof.argo024.ivonyw.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Ceph nvmeof.nvmeof.argo024.ivonyw for 3f3f4c74-eb6c-11ee-afa3-b4835101e458.

Adding both gw logs as an attachment

caroav commented 6 months ago

Should not happen following https://github.com/ceph/ceph-nvmeof/pull/543