gaowayne commented 3 years ago

Bug report

I want to use CMakeLists and cmake to build my simple hello_bdev example SPDK app application. But I suffer the link error related with SPDK app framework event core code even I am pretty sure I add spdk_event into CmakeLists.txt file.

Expected Behavior

I can build my simple SPDK example app with Cmakelists.txt.

Current Behavior

the link error is below

[root@cyp03 build]# make
[ 50%] Linking C executable hello_boostasio
/usr/bin/ld: /usr/local/lib/libspdk_event.a(app.o): in function `app_start_rpc':
/root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:285: undefined reference to `spdk_rpc_initialize'
/usr/bin/ld: /usr/local/lib/libspdk_event.a(app.o): in function `bootstrap_fn':
/root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:409: undefined reference to `spdk_subsystem_init_from_json_config'
/usr/bin/ld: /root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:414: undefined reference to `spdk_subsystem_init'
/usr/bin/ld: /root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:416: undefined reference to `spdk_rpc_initialize'
/usr/bin/ld: /usr/local/lib/libspdk_event.a(app.o): in function `_start_subsystem_fini':
/root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:606: undefined reference to `spdk_subsystem_fini'
/usr/bin/ld: /usr/local/lib/libspdk_event.a(app.o): in function `app_stop':
/root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:621: undefined reference to `spdk_rpc_finish'
/usr/bin/ld: /usr/local/lib/libspdk_event.a(app.o): in function `rpc_framework_start_init':
/root/nvme_disk/boostasiospdk/spdk/lib/event/app.c:1005: undefined reference to `spdk_subsystem_init'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/hello_boostasio.dir/build.make:103: hello_boostasio] Error 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/hello_boostasio.dir/all] Error 2
make: *** [Makefile:103: all] Error 2

Possible Solution

Steps to Reproduce

create one Cmakelists.txt to build hello_bdev


cmake_minimum_required(VERSION 2.8.9)
#set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -O0 -ggdb3 -Wall")
set (CMAKE_CXX_FLAGS "-std=c++14 -O0 -ggdb3")

include_directories(${CMAKE_SOURCE_DIR}/../../../dpdk/include ${CMAKE_SOURCE_DIR}/../../../include)

link_directories(${CMAKE_SOURCE_DIR}/../../../dpdk/lib ${CMAKE_SOURCE_DIR}/../../../lib)

dpdk spdk_trace

set(COMMLIBS uuid numa pthread dl rt curl ssl pthread librte_eal.so librte_cmdline.so librte_mbuf.so librte_mempool.so librte_mempool_ring.so librte_ring.so librte_pci.so librte_bus_pci.so librte_bus_vdev.so librte_compressdev.so librte_cryptodev.so librte_hash.so librte_kvargs.so spdk_env_dpdk spdk_env_dpdk_rpc spdk_nvme spdk_bdev spdk_notify spdk_util spdk_log spdk_sock spdk_event spdk_event_bdev spdk_thread spdk_conf spdk_rpc spdk_json spdk_util spdk_jsonrpc spdk_trace)

set (HELLOBOOSTASIO_SRC hello_boostasio.c )

add_executable(hello_boostasio ${HELLOBOOSTASIO_SRC})

target_link_libraries(hello_boostasio ${COMMLIBS})

2. create one hello_bdev.c file rename it to hello_boostasio.c file put the Cmakelists.txt and hello_boostasio.c file into the spdk/example/bdev/hello_bdev_test/
3. goes into hello_bdev_test, mkdir build, cd build/
4. run cmake ..
5. run make
6. will see above error. I am already install SPDK into local machine and copy DPDK lib modules into /usr/local/lib/
7. I run nm and confirmed that the libspdk_event.a contains the API that cmake complained about

[root@cyp03 lib]# nm /usr/local/lib/libspdk_event.a | grep spdk_rpc U spdk_rpc_finish U spdk_rpc_get_state U spdk_rpc_initialize U spdk_rpc_register_alias_deprecated U spdk_rpc_register_method U spdk_rpc_set_state U spdk_rpc_register_alias_deprecated U spdk_rpc_register_method U spdk_rpc_register_alias_deprecated U spdk_rpc_register_method

[root@cyp03 lib]# nm /usr/local/lib/libspdk_eventbdev.a | grep spdk 0000000000000000 d g_spdk_subsystem_bdev 000000000000007b t g_spdk_subsystem_bdev_register U spdk_add_subsystem U spdk_add_subsystem_depend U spdk_bdev_finish U spdk_bdev_initialize U spdk_bdev_subsystem_config_json U spdk_subsystem_fini_next U spdk_subsystem_init_next



## Context (Environment including OS version, SPDK version, etc.)
SPDK master code after v21.07. Fedora OS 33

<!--- Providing context helps us come up with a solution that is most useful in the real world -->

gaowayne commented 3 years ago

hello guys, I have fixed the build issue by modify CMakeLists.txt file sequence and static link and dynamic link. when running. I found another error, SPDK register macro is not built into my application, that cause it failed to init scheduler list.

SPDK_SCHEDULER_REGISTER(scheduler);

/*

Macro used to register new reactor balancer. */
define SPDK_SCHEDULER_REGISTER(scheduler) \

static void attribute((constructor)) _spdk_schedulerregister ## scheduler (void) \ { \ _spdk_scheduler_list_add(&scheduler); \ } \

gaowayne commented 3 years ago

I adjust CMakelists.txt. now schedule is registered well now. but bdev malloc RPC method cannot be registered.

SPDK_RPC_REGISTER("bdev_malloc_create", rpc_bdev_malloc_create, SPDK_RPC_RUNTIME)

[root@cyp03 build]# nm hello_boostasio | grep rpc_bdev_malloc_create [root@cyp03 build]#

[root@cyp03 examples]# nm hello_bdev | grep rpc_bdev_malloc_create 0000000000408255 t rpc_bdev_malloc_create 000000000040843d t rpc_register_rpc_bdev_malloc_create [root@cyp03 examples]#

below is my latest CMakeLists.txt

cmake_minimum_required(VERSION 2.8.9)
#set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -O0 -ggdb3 -Wall")
set (CMAKE_CXX_FLAGS "-std=c++14 -O0 -ggdb3 -DBOOST_LOG_DYN_LINK -DBOOST_LOG_USE_NATIVE_SYSLOG")
project (hello_boostasio)
#add_definitions(-Wno-builtin-macro-redefined --verbose )
add_definitions(--verbose )
include_directories(${CMAKE_SOURCE_DIR}/../../../dpdk/build/include
                    ${CMAKE_SOURCE_DIR}/../../../include
                    ${CMAKE_SOURCE_DIR}/../../../build/include)

#link_directories(${CMAKE_SOURCE_DIR}/../../../dpdk/lib
#                ${CMAKE_SOURCE_DIR}/../../../lib)

link_directories(/root/nvme_disk/boostasiospdk/spdk/dpdk/build/lib
                 /root/nvme_disk/boostasiospdk/spdk/build/lib)

#dpdk spdk_trace 
#set(COMMLIBS uuid numa pthread dl rt curl ssl pthread boost_program_options boost_log_setup boost_log boost_thread boost_system boost_filesystem librte_eal.so librte_cmdline.so  librte_mbuf.so librte_mempool.so librte_mempool_ring.so librte_ring.so librte_pci.so librte_bus_pci.so librte_bus_vdev.so librte_compressdev.so librte_cryptodev.so librte_hash.so librte_kvargs.so spdk_env_dpdk spdk_env_dpdk_rpc spdk_nvme spdk_bdev spdk_notify spdk_util spdk_log spdk_sock spdk_event spdk_event_bdev spdk_thread spdk_conf spdk_rpc spdk_json spdk_util spdk_jsonrpc spdk_trace spdk_rpc)

set(DYNA_COMMLIBS uuid numa pthread dl rt curl ssl pthread boost_program_options boost_log_setup boost_log boost_thread boost_system boost_filesystem librte_eal.so librte_cmdline.so  librte_mbuf.so librte_mempool.so librte_mempool_ring.so librte_ring.so librte_pci.so librte_bus_pci.so librte_bus_vdev.so librte_compressdev.so librte_cryptodev.so librte_hash.so librte_kvargs.so)
set(STAT_COMMLIBS 
    libspdk_rpc.a  
libspdk_accel.a       libspdk_bdev_ftl.a     libspdk_bdev_passthru.a    libspdk_blob_bdev.a    libspdk_env_dpdk_rpc.a  libspdk_event_nvmf.a   libspdk_init.a     libspdk_lvol.a    libspdk_scsi.a        libspdk_ut_mock.a
libspdk_accel_ioat.a  libspdk_bdev_gpt.a     libspdk_bdev_raid.a        libspdk_blobfs.a       libspdk_event.a         libspdk_event_scsi.a   libspdk_ioat.a     libspdk_nbd.a     libspdk_sock.a        libspdk_vhost.a
libspdk_bdev.a        libspdk_bdev_lvol.a    libspdk_bdev_split.a       libspdk_blobfs_bdev.a  libspdk_event_accel.a   libspdk_event_sock.a   libspdk_iscsi.a    libspdk_notify.a  libspdk_sock_posix.a  libspdk_virtio.a
libspdk_bdev_aio.a    libspdk_bdev_malloc.a  libspdk_bdev_virtio.a      libspdk_conf.a         libspdk_event_bdev.a    libspdk_event_vhost.a  libspdk_json.a     libspdk_nvme.a    libspdk_thread.a      libspdk_vmd.a
libspdk_bdev_delay.a  libspdk_bdev_null.a    libspdk_bdev_zone_block.a  libspdk_dma.a          libspdk_event_iscsi.a   libspdk_event_vmd.a    libspdk_jsonrpc.a  libspdk_nvmf.a    libspdk_trace.a       
libspdk_bdev_error.a  libspdk_bdev_nvme.a    libspdk_blob.a             libspdk_env_dpdk.a     libspdk_event_nbd.a     libspdk_ftl.a          libspdk_log.a      libspdk_rpc.a     libspdk_util.a )

set (HELLOBOOSTASIO_SRC
     hello_boostasio.cc
     )
add_library(spdkapplib OBJECT ${HELLOBOOSTASIO_SRC})

link_libraries(hello_boostasio ${STAT_COMMLIBS})

add_executable(hello_boostasio $<TARGET_OBJECTS:spdkapplib> main.cc)
#add_executable(hello_boostasio ${HELLOBOOSTASIO_SRC})

target_link_libraries(hello_boostasio ${DYNA_COMMLIBS})

jimharris commented 3 years ago

Please consider using pkg-config to generate your list of libraries. https://spdk.io/doc/pkgconfig.html

You may also try pretending Q= to the SPDK make line to compare your cmake build with what SPDK build system is linking.

AlekseyMarchuk commented 3 years ago

@gaowayne The missing symbol rpc_bdev_malloc_create is used in constructor function and probably it was ignored by the linker. SPDK docs say that when static libraries are linked, it is needed to use -Wl,--whole-archive linker option. Please refer to https://spdk.io/doc/libraries.html

spdk / spdk

failed to create one simple SPDK app following hello_bdev with CMakeLists.txt to build it #2145

Bug report

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce

dpdk spdk_trace

define SPDK_SCHEDULER_REGISTER(scheduler) \