Permission denied when using IB Verbs

lwa-project / ng_digital_processor

The Next Generation Digital Processor for LWA North Arm

Apache License 2.0

0 stars 0 forks source link

Permission denied when using IB Verbs #3

Closed jaycedowell closed 1 year ago

jaycedowell commented 1 year ago

I'm using OFED 5.8 for Ubuntu 20.04 with ConnectX6 cards and get a "unsupported mode" error when I try a verbs capture.

jaycedowell commented 1 year ago

Adding in the OFED kernel DMKS/tools changes things. "unsupported mode" is now "permission denied" which is more familiar.

jaycedowell commented 1 year ago

Looks like I need CAP_NET_RAW to get the pipelines running as a non-root user. I tried spead2_net_raw but that kills my special LD_LIBRARY_PATH that I use to find Bifrost. systemd seems to have a "AmbientCapabilities" service keyword that might help.

jaycedowell commented 1 year ago

584538d only seems to work on node 2 right now. I have no idea why.

jaycedowell commented 1 year ago

At least on node 1 the failure was from some residual setcap changes on /usr/bin/python3.8 from when I was testing. The head node also seems ok now.