search

RC4ML / BenchBF3

Demystifying Datapath Accelerator Enhanced Off-path SmartNIC [ICNP24]
MIT License
13 stars 2 forks source link

Unable to run DPA kernels from the host (non-BF Arm cores) #1

Closed max-Hawkins closed 2 months ago

max-Hawkins commented 2 months ago

Were you all able to trigger DPA kernels from the 'traditional' (non-BF cores)? I'm able to invoke DPA kernels from the BlueField ARM cores, but not from the x86 host. I see errors similar to those reported on the NVIDIA documentation here. Considering you heavily benchmarked the DPA, do you have any thoughts on this?

cxz66666 commented 2 months ago

As far as we know, DOCA 2.7.0(also in latest 2.8.0) only supports programming DPA with Flexio on HOST under NIC MODE, you can convert to NIC MODE and try again!

Here is a simple command for convert DPU MODE to NIC MODE

sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=1 

sudo mlxconfig -d /dev/mst/mt41692_pciconf0.1 s INTERNAL_CPU_OFFLOAD_ENGINE=1

Then power cycle your machine

convet NIC MODE to DPU MODE 

sudo mlxconfig -d /dev/mst/mt41692_pciconf0 s INTERNAL_CPU_OFFLOAD_ENGINE=0 

sudo mlxconfig -d /dev/mst/mt41692_pciconf0.1 s INTERNAL_CPU_OFFLOAD_ENGINE=0

Also need power cycle!

max-Hawkins commented 2 months ago

Thanks for that! That helps clear things up.

I appreciate bolding the content "Due to DOCA driver limitation, DPA can only access Arm memory under DPU mode, and access host memory under NIC mode. So please change to correct mode before benchmark.", but I feel that it doesn't make it immediately obvious that it also effects any DPA kernel invocation. I feel like adding something like you said in the previous message would be helpful.

cxz66666 commented 2 months ago

Thank you so much for the heads up. I will update Readme for that.