search

ceph / ceph-nvmeof

Service to provide Ceph storage over NVMe-oF/TCP protocol
GNU Lesser General Public License v3.0
88 stars 46 forks source link

Warn/fail during Gateway startup if actual huge pages count is less than requested hugepages #458

Open rahullepakshi opened 8 months ago

rahullepakshi commented 8 months ago

There are cases where we deploy nvmeof service/ start GW though actual huge page count is less than requested huge pages to startup GW. Though GW might start successfully, it may error out during large scale due to memory crunch. So we need take a decision to warn or fail GW startup when huge page count is less

In below GW, nvmeof service is succesfully deployed when actual huge page count is 346 which is way less than requested i.e. 2048 but somehow GW started

[root@ceph-mytest-4lb9mq-node3 ~]# cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
346

[ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 -->
TELEMETRY: No legacy callbacks, legacy socket not created
[2024-02-22 11:48:17.172326] app.c: 712:spdk_app_start: *NOTICE*: Total cores available: 1
[2024-02-22 11:48:17.249185] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2024-02-22 11:48:17.276564] accel_sw.c: 681:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
[2024-02-22 11:48:17.459755] tcp.c: 629:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
INFO:nvmeof:Discovery service process id: 25
INFO:nvmeof:Using NVMeoF gateway version 1.0.0
INFO:nvmeof:Using SPDK version 23.01.1
INFO:nvmeof:Using vstart cluster version based on 18.2.1
INFO:nvmeof:NVMeoF gateway built on: 2024-01-31 13:36:14 UTC
INFO:nvmeof:NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof
INFO:nvmeof:NVMeoF gateway Git branch: tags/1.0.0
INFO:nvmeof:NVMeoF gateway Git commit: d08860d3a1db890b2c3ec9c8da631f1ded3b61b6
INFO:nvmeof:SPDK Git repository: https://github.com/ceph/spdk.git
INFO:nvmeof:SPDK Git branch: undefined
INFO:nvmeof:SPDK Git commit: 668268f74ea147f3343b9f8136df3e6fcc61f4cf
INFO:nvmeof:Starting ceph nvmeof discovery service
INFO:nvmeof:Connected to Ceph with version "18.2.1-629-g329eaff9 (329eaff91af228c8469648b22c12e1f4608e7b45) reef (stable)"
INFO:nvmeof:Requested huge pages count is 2048
INFO:nvmeof:Actual huge pages count is 346
WARNING:nvmeof:The actual huge page count 346 differs from the requested value of 2048
INFO:nvmeof:nvmeof.None.state omap object already exists.
INFO:nvmeof:log pages info from omap: nvmeof.None.state
INFO:nvmeof:Prometheus endpoint is enabled
ERROR:control.prometheus:Unable to start prometheus exporter - missing cert/key file(s)
INFO:nvmeof:discovery addr: 0.0.0.0 port: 8009
INFO:nvmeof:Received request to create subsystem nqn.2016-06.io.spdk:test_cli, enable_ha: False, ana reporting: False, context: <grpc._server._Context object at 0x7ff31e70>
INFO:nvmeof:create_subsystem nqn.2016-06.io.spdk:test_cli: True
INFO:nvmeof:Received request to create client.nvmeof.nvme_pool.ceph-mytest-4lb9mq-node3.lcojwd TCP ipv4 listener for nqn.2016-06.io.spdk:test_cli at 10.0.209.126:4420, aut>
[2024-02-22 12:15:54.732434] tcp.c: 850:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 10.0.209.126 port 4420 ***
INFO:nvmeof:create_listener: True
INFO:nvmeof:enable_ha: False
INFO:nvmeof:Received request to allow any host access for nqn.2016-06.io.spdk:test_cli, context: <grpc._server._Context object at 0x7ff31e730370>
INFO:nvmeof:add_host *: True
INFO:nvmeof:Received request to add a namespace to nqn.2016-06.io.spdk:test_cli, context: <grpc._server._Context object at 0x7ff31e6d4f40>
INFO:nvmeof:Received request to create bdev bdev_0c245ba1-7615-4ca7-a553-4e2c5ea27bfc from iscsi_test/iscsi_image (size 0 MiB) with block size 512, will not create image i>
INFO:nvmeof:Allocating cluster name='cluster_context_0'
[2024-02-22 12:17:38.872555] bdev_rbd.c:1199:bdev_rbd_create: *NOTICE*: Add bdev_0c245ba1-7615-4ca7-a553-4e2c5ea27bfc rbd disk to lun
INFO:nvmeof:bdev_rbd_create: bdev_0c245ba1-7615-4ca7-a553-4e2c5ea27bfc
INFO:nvmeof:Received request to add bdev_0c245ba1-7615-4ca7-a553-4e2c5ea27bfc to nqn.2016-06.io.spdk:test_cli with ANA group id 1 using UUID 0c245ba1-7615-4ca7-a553-4e2c5e>
INFO:nvmeof:subsystem_add_ns: 1

In another GW on same setup, nvmeof service deployment failed when hug page count is 149 which is way less than requested and GW did not start

[root@ceph-mytest-4lb9mq-node4 ~]# cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
149

[ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 -->
TELEMETRY: No legacy callbacks, legacy socket not created
[2024-02-22 11:48:05.967645] app.c: 712:spdk_app_start: *NOTICE*: Total cores available: 1
[2024-02-22 11:48:06.040587] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2024-02-22 11:48:06.068086] accel_sw.c: 681:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.
[2024-02-22 11:48:06.103353] bdev.c:1675:spdk_bdev_initialize: *ERROR*: could not allocate spdk_bdev_io pool
[2024-02-22 11:48:06.103434] subsystem.c: 143:spdk_subsystem_init_next: *ERROR*: Init subsystem bdev failed
[2024-02-22 11:48:06.103446] app.c: 819:spdk_app_stop: *WARNING*: spdk_app_stop'd on non-zero
ERROR:nvmeof:GatewayServer: SIGCHLD received signum=17
ERROR:nvmeof:GatewayServer exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 94, in _connect
    raise socket.error("Invalid or non-existing address: '%s'" % addr)
OSError: Invalid or non-existing address: '/var/tmp/spdk.sock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 60, in __init__
    self._connect(addr, port)
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 96, in _connect
    raise JSONRPCException("Error while connecting to %s\n"
spdk.rpc.client.JSONRPCException: Error while connecting to /var/tmp/spdk.sock
Is SPDK application running?
Error details: Invalid or non-existing address: '/var/tmp/spdk.sock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/src/control/__main__.py", line 44, in <module>
    gateway.serve()
  File "/src/control/server.py", line 123, in serve
    self._start_spdk(omap_state)
  File "/src/control/server.py", line 264, in _start_spdk
    self.spdk_rpc_client = rpc_client.JSONRPCClient(
  File "/usr/lib/python3.9/site-packages/spdk/rpc/client.py", line 64, in __init__
    time.sleep(0.2)
  File "/src/control/server.py", line 50, in sigchld_handler
    raise SystemExit(f"Gateway subprocess terminated {pid=} {exit_code=}")
SystemExit: Gateway subprocess terminated pid=22 exit_code=255
INFO:nvmeof:Aborting SPDK(client.nvmeof.nvme_pool.ceph-mytest-4lb9mq-node4.xeqhwb) pid 22...
INFO:nvmeof:Exiting the gateway process.
INFO:nvmeof:Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvme_pool.ceph-mytest-4lb9mq-node4.xeqhwb/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvme_pool.>
Gateway subprocess terminated pid=22 exit_code=255
caroav commented 8 months ago

@oritwas @rahullepakshi I believe that ceph adm should not start the GW in case that the number of hugepages allocated is less than what we try to allocate. Currently in ceph\src\cephadm\cephadmlib\daemons\nvmeof.py we ask to allocate 4096 hugepages.

rahullepakshi commented 8 months ago

@caroav but as you can see in above logs, at v1.0.0, we are requesting only 2048, right?

INFO:nvmeof:NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof
INFO:nvmeof:NVMeoF gateway Git branch: tags/1.0.0
INFO:nvmeof:NVMeoF gateway Git commit: d08860d3a1db890b2c3ec9c8da631f1ded3b61b6
INFO:nvmeof:SPDK Git repository: https://github.com/ceph/spdk.git
INFO:nvmeof:SPDK Git branch: undefined
INFO:nvmeof:SPDK Git commit: 668268f74ea147f3343b9f8136df3e6fcc61f4cf
INFO:nvmeof:Starting ceph nvmeof discovery service
INFO:nvmeof:Connected to Ceph with version "18.2.1-629-g329eaff9 (329eaff91af228c8469648b22c12e1f4608e7b45) reef (stable)"
INFO:nvmeof:Requested huge pages count is 2048