Mellanox / rshim-user-space

Linux based user-space RSHIM driver for the Mellanox BlueField SoC
Other
24 stars 17 forks source link

Rshim IOCTL stats #152

Open mpodles opened 5 months ago

mpodles commented 5 months ago

Dear team,

I'm working on Bluefield 2 DPU devices and have received scripts that can access low-level statistics. I don't know how much of the details are neccesary but in brief what the script does is open a file descriptor to rshim.

self.rshim_fd = os.open(filename, os.O_RDWR)

with filename /dev/rshim0 and then the script tries to

adapted_addr = (channel << 16) | addr
buf= array.array("B", 16*[0])                                                                                                               
struct.pack_into('<LQ',buf, 0, adapted_addr, 0)                                                                                             
sys.stdout.flush()                                                                                                                          
fcntl.ioctl(self.rshim_fd, 1, buf) 
val=buf[4:12]                                                                                                                               
rv=struct.unpack("<Q",bytearray(val))[0] 

which results in IOError: [Errno 25] Inappropriate ioctl for device

I've looped through all the others possible values instead of 1 and the error is the same. The version:

rshim -v
rshim 2.0.20

If you need any additional info about what the script does we can go deeper into it. I would be grateful for any help.

Best regards

lsun100 commented 4 months ago

How about some code like below (for /dev/rshim0/rshim)?

RSHIM_IOC_READ = 0xc00d5200 RSHIM_IOC_WRITE = 0xc00d5201

def dev_read(fd, addr, size): try: val = os.pread(fd, (size >> 3), addr) except (OSError, IOError) as e: if e.errno == errno.ENOSYS: buf = bytearray(16) struct.pack_into('<LQB', buf, 0, addr, 0, (size >> 3)) fcntl.ioctl(fd, RSHIM_IOC_READ, buf) val = buf[4:12] return struct.unpack("<Q", val)[0]

def dev_write(fd, addr, val, size): try: raw_val = struct.pack("<Q", val) os.pwrite(fd, raw_val, addr) except (OSError, IOError) as e: if e.errno == errno.ENOSYS: buf = bytearray(20) struct.pack_into('<LQB', buf, 0, addr, val, (size >> 3)) fcntl.ioctl(fd, RSHIM_IOC_WRITE, buf)

mpodles commented 4 months ago

Dear @lsun100 ,

Thanks for coming back to me with this. Indeed using the provided RSHIM_IOC_READ changes the error to TypeError: an integer is required where I believe the problem is in the buf since ioctl() expects an integer there. I'm also not sure what should I use as the size paramater.

The address that comes in is 1360 which gets packed into bytearray(b'P\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00') when using size of 16. I've tried other sizes between 1-16 as well but they all give the same error.

Thanks again and I'm grateful for any further help. Best regards