Closed gwnet closed 5 years ago
buddy, here is the error if I follow your idea.
wayne@ubuntu:~/jNVMf$ java -cp target/jnvmf-1.5-jar-with-dependencies.jar:target/jnvmf-1.5-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.benchmark.NvmfClientBenchmark -a 192.168.147.130 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2014-08.org.nvmexpress.discovery -qd 1 -rw read -s 4096 -qs 64 -H -I Exception in thread "main" java.lang.IllegalArgumentException: Invalid NQN at com.ibm.jnvmf.NvmeQualifiedName.validate(NvmeQualifiedName.java:47) at com.ibm.jnvmf.NvmeQualifiedName.(NvmeQualifiedName.java:36) at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.(NvmfClientBenchmark.java:177) at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.main(NvmfClientBenchmark.java:551)
Hello buddy, new questions comes here. I feel the inline RDMA and incapsule is the same thing. it will put the data on the buffer and send out. SGL need memory transaction. let me know your insight. what is SPDK client support. what is your jNVMf support?
inline and incapsule are two completely separate things: "inline" is an RDMA concept independent of NVMf while "incapsule" data is part of the NVMf protocol. When you use RDMA to send data "inline" data is put in the work request on the send queue (of the RDMA connections) directly. You need to to do your own research to understand the difference of these concepts. SPDK does not support incapsule or inline to my knowledge.
we actually need one way that can help get below two points:
I assume you want multiple clients accessing the same NVMe devices remotely hence the need for the cache at the target side to allow some form of atomicity?
Do you need to be able to mount the device such that it appears in /dev/nvmeXX or similar? This is not possible with my code (and never will be). That means every application that wants to use jNVMf needs to be modified. Regarding unaligned accesses: The NVMf specification does not allow this however the NVMe specification (although to my knowledge there are no devices which support it) does, cf. https://nvmexpress.org/wp-content/uploads/NVM_Express_Revision_1.3.pdf page 21 - BitBucket, for an example refer to page 61. You could extend the jNVMf code to support the BitBucket descriptor to allow such access. However, that brings us to 2)
This is the harder part. You need to implement BitBucket NVMf target support for the kernel or SPDK plus buffering. I suggest using SPDK. Cached IO is not easy since you need some eviction strategy etc.
Yes, I have multiple clients will access the same target. as you said.
hello buddy, I fixed the typo issue. but I still suffer error. it told me invalid namespace. but I copy the string from the dmesg. Could you please help me?
wayne@ubuntu:~/jNVMf$ java -cp target/jnvmf-1.5-jar-with-dependencies.jar:target/jnvmf-1.5-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.benchmark.NvmfClientBenchmark -a 192.168.147.130 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5 -qd 1 -rw read -s 4096 -qs 64 -H -I read 4096bytes with QD = 1, time[s] = 3, pattern = RANDOM, runs = 10 log4j:WARN No appenders could be found for logger (com.ibm.disni). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" com.ibm.jnvmf.UnsuccessfulComandException: Command was not successful. {StatusCodeType: 1 - Command Specific, SatusCode: 130 - One or more of the parameters (Host NQN, Subsystem NQN, Host Identifier, Controller ID, Queue ID) specified are not valid., CID: 0, Do_not_retry: true, More: false, SQHD: 0} at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128) at com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36) at com.ibm.jnvmf.QueuePair.(QueuePair.java:195) at com.ibm.jnvmf.QueuePair.(QueuePair.java:134) at com.ibm.jnvmf.AdminQueuePair.(AdminQueuePair.java:31) at com.ibm.jnvmf.Controller.(Controller.java:66) at com.ibm.jnvmf.Nvme.connect(Nvme.java:50) at com.ibm.jnvmf.Nvme.connect(Nvme.java:44) at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.connect(NvmfClientBenchmark.java:216) at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.(NvmfClientBenchmark.java:203) at com.ibm.jnvmf.benchmark.NvmfClientBenchmark.main(NvmfClientBenchmark.java:551)
==below is the dmesg that I get the nqn name to put into argument list above== [34926.170782] RPC: Registered rdma transport module. [34926.170783] RPC: Registered rdma backchannel transport module. [34993.373669] nvmet_rdma: enabling port 1 (192.168.147.130:4420) [35059.810576] nvmet: creating controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5. [35059.810765] nvme nvme1: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.147.130:4420 [35059.810932] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" [35161.418867] nvmet: connect request for invalid subsystem nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5! [35474.273405] nvmet: creating controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5. [35474.273544] nvme nvme1: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.147.130:4420 [35474.275321] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" [35523.940618] nvmet: connect request for invalid subsystem nqn.2014-08.org.nvmexpress:uuid:ea997850-3aa5-47e0-b1eb-bd11046df1e5! wayne@ubuntu:~/jNVMf$