Remote access error

[lyxxn0414]

I'm starting linefs with 3 nodes, ip addresses of them are as below:

// 1st machine - Bluefield NIC
    { .ip = "10.10.3.101", .role = HOT_REPLICA, .type = KERNFS_NIC_PEER},
        // 1st machine - Host
    { .ip = "10.10.3.1", .role = HOT_REPLICA, .type = KERNFS_PEER},
        // 2nd machine - Bluefield NIC
        { .ip = "10.10.3.102", .role = HOT_REPLICA, .type = KERNFS_NIC_PEER},
        // 2nd machine - Host
        { .ip = "10.10.3.2", .role = HOT_REPLICA, .type = KERNFS_PEER},
        // 3rd machine - Bluefield NIC
    { .ip = "10.10.3.103", .role = HOT_REPLICA, .type = KERNFS_NIC_PEER},
        // 3rd machine - Host
    { .ip = "10.10.3.3", .role = HOT_REPLICA, .type = KERNFS_PEER},

and SmartNIC of node2 has been successfully connected with its host. The terminal of node2-nic is like this:

[New Thread 0xfffd0b1e6850 (LWP 253800)]
[New Thread 0xfffd0a9e5850 (LWP 253801)]
Connecting to KernFS instance 5 [ip: 10.10.3.3]
[New Thread 0xfffd095fe850 (LWP 253802)]
Wait for connections established. 0/2
Wait for connections established. 0/2
Wait for connections established. 0/2
Wait for connections established. 0/2
Wait for connections established. 0/2

But node1-host gets the wrong ip of node1-nic. The terminal of node1-host is like this:

ip address on interface 'enp129s0f0' is 10.10.3.2
It is Ready. Write 1 to file: /opt/LineFS/LineFS_x86/signals/kernfs/node1.jinwei-121792.bfkvs-pg0.clemson.cloudlab.us
Server: cmd_hdr is s
Server: cmd_hdr is s
Reading root inode with inum: 1
[New Thread 0x7ff6721ff700 (LWP 12727)]
Connecting to KernFS instance 4 [ip: 10.10.3.103]
12767 connection.c:1357 rc_die(): unknown event
Thread 34 "kernfs" received signal SIGTRAP, Trace/breakpoint trap

Could you help me to solve this problem? Thanks a lot!

[lyxxn0414]

After debugging, I update rdma-core lib in all hosts and solve this problem. But node2-host failed when executing ibv_reg_mr, and the errno is 12(cannot allocate memory). I have checked max memory size and max locked memory are both unlimited. Could you help me solve this problem? Thanks!

[lyxxn0414]

By decreasing the size of memory registered by ibv_reg_mr, it can correctly run.

[lyxxn0414]

I edit the size in fs.c, line1692

for(int i=0; i<n_regions; i++) {
        switch(i) {
            case MR_NVM_LOG: {

                //FIXME: share all log entries; current impl shares only a single log mr
                mrs[i].type = MR_NVM_LOG;
                mrs[i].addr = (uint64_t)g_bdev[g_log_dev]->map_base_addr;
                mrs[i].length = ((disk_sb[g_log_dev].nlog) << g_block_size_shift);
                break;
            }
            case MR_NVM_SHARED: {
                mrs[i].type = MR_NVM_SHARED;
                mrs[i].addr = (uint64_t)g_bdev[g_root_dev]->map_base_addr;
                //mrs[i].length = dev_size[g_root_dev];
                //mrs[i].length = (1UL << 30);
                                mrs[i].length = (sb[g_root_dev]->ondisk->size << g_block_size_shift);   // All possible address of device. Ref) mkfs.c
                //mrs[i].length = (disk_sb[g_root_dev].datablock_start - disk_sb[g_root_dev].inode_start << g_block_size_shift);  // data blocks only.
                break;
            }
            /*
            case MR_DRAM_CACHE: {
                mrs[i].type = MR_DRAM_CACHE;
                mrs[i].addr = (uint64_t) g_fcache_base;
                mrs[i].length = (g_max_read_cache_blocks << g_block_size_shift);
                break;
            }
            */

            case MR_DRAM_BUFFER: {
                // Check nic_slab_pool does not exceed 12GB. Note that the size of
                // DRAM in SmartNIC is 16GB.
                mlfs_assert((g_max_nicrpc_buf_blocks << g_block_size_shift)
                    <= (12ULL*1024*1024*1024));
                // TODO need to adjust buf size.
                nic_slab_init((g_max_nicrpc_buf_blocks << g_block_size_shift)/4);
                mrs[i].length = (g_max_nicrpc_buf_blocks << g_block_size_shift)/4;
                mrs[i].type = MR_DRAM_BUFFER;
                mrs[i].addr = (uint64_t) nic_slab_pool->addr;

                pr_dram_alloc("[DRAM_ALLOC] MR_DRAM_BUFFER size=%lu(MB)", mrs[i].length/1024/1024);
                                break;
            }
            default:
                break;
        }
                pr_setup("mrs[%d] type %d, addr 0x%lx - 0x%lx, length %lu(%lu MB)",
                       i, mrs[i].type, mrs[i].addr, mrs[i].addr + mrs[i].length,
                       mrs[i].length, mrs[i].length / 1024 / 1024);
        }

and the same position in nic_fs.c. It seems to correctly run. But when I run the iotest, I got "remote access error" when poll_cq. Seems that I cannot edit this way, could you help me solve this problem? Thanks a lot!