Failed to create large iobuf pool

[sunilkumarn417]

nvmeof GW Deployment failed with below error messages.

Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Starting gateway client.nvmeof.rbd.ceph-1sunilkumar-qimeq6-node5.nintap
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: DEBUG:control.server:Starting serve
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: DEBUG:control.server:Configuring server client.nvmeof.rbd.ceph-1sunilkumar-qimeq6-node5.nintap
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:SPDK Target Path: /usr/local/bin/nvmf_tgt
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:SPDK Socket: /var/tmp/spdk.sock
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Starting /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO: Setting log level to WARN
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:JSONRPCClient(/var/tmp/spdk.sock):Setting log level to WARN
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.436592] Starting SPDK v23.01.1 / DPDK 22.11.1 initialization...
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.436787] [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3 ]
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: TELEMETRY: No legacy callbacks, legacy socket not created
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.550555] app.c: 712:spdk_app_start: *NOTICE*: Total cores available: 1
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.602641] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.652634] thread.c:2956:spdk_iobuf_initialize: *ERROR*: Failed to create large iobuf pool
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670091] iobuf.c:  54:iobuf_subsystem_initialize: *ERROR*: Failed to initialize iobuf
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670188] subsystem.c: 143:spdk_subsystem_init_next: *ERROR*: Init subsystem iobuf failed
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670201] app.c: 819:spdk_app_stop: *WARNING*: spdk_app_stop'd on non-zero
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670221] thread.c:2184:spdk_io_device_unregister: *ERROR*: io_device 0x708b20 not found

journal1.log

[caroav]

@sunilkumarn417 can you share more data about the resources of this machine? How much RAM and how many Cores does it have? How many huge pages were allocated?

[caroav]

@sunilkumarn417 can you update?