Accel-config test fails - libaccfg: write_attr: failed to write

After following the installation steps in the README I've tried to run the tests for accel-config. The above screenshot is what I am getting. It was compiled with the debug flag enabled.

Surprisingly it has no issues setting the number of tokens allowed for group0.0, but for some reason group 0.1 is throwing it off.

Arguments and exact line in sysfs.c where it is throwing an error:

Is there something that could explain this behavior? I am using a server with SPR core, Ubuntu 21.04 with a 5.11 kernel.

I assume you are using 3.4.1 from the stable branch?

@ramesh-thomas can you please see why the test config is failing token setup?

@davejiang

Yes, we are using the latest version.

@rrosa-cn Can you please do the following?

Add idxd.dyndbg in your kernel command line boot parameters and reboot.
Run sudo dmesg -C
Run the accel-config test
Run dmesg

Please send the idxd related output of dmesg

@rrosa-cn can you list the values of the following in sysfs?

cat /sys/bus/dsa/devices/dsa0/max_tokens cat /sys/bus/dsa/devices/dsa0/token_limit cat /sys/bus/dsa/devices/dsa0/group0.1/engines cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_reserved cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed cat /sys/bus/dsa/devices/dsa0/group0.1/use_token_limit

@rrosa-cn can you list the values of the following in sysfs?

cat /sys/bus/dsa/devices/dsa0/max_tokens cat /sys/bus/dsa/devices/dsa0/token_limit cat /sys/bus/dsa/devices/dsa0/group0.1/engines cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_reserved cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed cat /sys/bus/dsa/devices/dsa0/group0.1/use_token_limit

cat /sys/bus/dsa/devices/dsa0/max_tokens 96

cat /sys/bus/dsa/devices/dsa0/token_limit 10

cat /sys/bus/dsa/devices/dsa0/group0.1/engines engine0.1

cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_reserved 1

cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed 8

cat /sys/bus/dsa/devices/dsa0/group0.1/use_token_limit 0

@rrosa-cn Can you please do the following?

Add idxd.dyndbg in your kernel command line boot parameters and reboot.

Run sudo dmesg -C

Run the accel-config test

Run dmesg

Please send the idxd related output of dmesg

I've update kernel to 5.14 upstream since I thought it might have been causing the issue. idxd wiki says it was tested at 5.12.

4. dmesg

I have tried to do that but unfortunately the server went down after shutdown command and wouldn't reboot. Will have to try that tomorrow.

cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed 8

Interestingly the value it complained about is actually set. Did you see the same issue before listing?

Since group0.0 is working, can you also list the same values in sysfs for it?

cat /sys/bus/dsa/devices/dsa0/max_tokens cat /sys/bus/dsa/devices/dsa0/token_limit cat /sys/bus/dsa/devices/dsa0/group0.0/engines cat /sys/bus/dsa/devices/dsa0/group0.0/tokens_reserved cat /sys/bus/dsa/devices/dsa0/group0.0/tokens_allowed cat /sys/bus/dsa/devices/dsa0/group0.0/use_token_limit

I got the server back up, but dmesg does not return anything after running accel-config. Is the kernel cmd arguments supposed to look like this? This is how this line looks in my grub file: GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on,sm_on, idxd.dyndbg= +p"

cat /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed 8

Interestingly the value it complained about is actually set. Did you see the same issue before listing?

Since group0.0 is working, can you also list the same values in sysfs for it?

cat /sys/bus/dsa/devices/dsa0/max_tokens cat /sys/bus/dsa/devices/dsa0/token_limit cat /sys/bus/dsa/devices/dsa0/group0.0/engines cat /sys/bus/dsa/devices/dsa0/group0.0/tokens_reserved cat /sys/bus/dsa/devices/dsa0/group0.0/tokens_allowed cat /sys/bus/dsa/devices/dsa0/group0.0/use_token_limit

cat /sys/bus/dsa/devices/dsa0/max_tokens 96

cat /sys/bus/dsa/devices/dsa0/token_limit 10

cat /sys/bus/dsa/devices/dsa0/group0.0/engines `` Blank

cat /sys/bus/dsa/devices/dsa0/group0.0/tokens_reserved 1

cat /sys/bus/dsa/devices/dsa0/group0.0/tokens_allowed 8

cat /sys/bus/dsa/devices/dsa0/group0.0/use_token_limit 1

Just idxd.dyndbg. No extra after it.

On Sep 28, 2021, at 12:16 PM, rrosa-cn @.***> wrote:

I got the server back up, but dmesg does not return anything after running accel-config. Is the kernel cmd arguments supposed to look like this? This is how this line looks in my grub file: GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on,sm_on, idxd.dyndbg= +p"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/intel/idxd-config/issues/9#issuecomment-929550380, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAWFNMDMQL55Z5HWEFV7U2TUEIIBRANCNFSM5EWBGLJQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@rrosa-cn We need to narrow down which value is not working. Can you try writing different values as below

sudo su echo 16 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 10 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 8 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 6 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 4 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 2 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed

Can you also try the pending branch?

@rrosa-cn We need to narrow down which value is not working. Can you try writing different values as below

sudo su echo 16 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 10 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 8 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 6 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 4 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed echo 2 > /sys/bus/dsa/devices/dsa0/group0.1/tokens_allowed

Can you also try the pending branch?

I was able to run every echo command without errors. More interestingly, after reinstalling accel-config with pending branch it passes all the tests, even though it throws some write errors.

accel-config test
run test_libaccfg
libkmod: DEBUG ../libkmod/libkmod.c:403 kmod_pool_get_module: get module name='idxd' found=(nil)
libkmod: DEBUG ../libkmod/libkmod.c:411 kmod_pool_add_module: add 0x55cf1eb22790 key='idxd'
libkmod: DEBUG ../libkmod/libkmod.c:403 kmod_pool_get_module: get module name='idxd_mdev' found=(nil)
libkmod: DEBUG ../libkmod/libkmod.c:411 kmod_pool_add_module: add 0x55cf1eb228a0 key='idxd_mdev'
libkmod: DEBUG ../libkmod/libkmod.c:511 lookup_builtin_file: file=/lib/modules/5.14.8-051408-generic/modules.builtin.bin modname=idxd_mdev
libkmod: DEBUG ../libkmod/libkmod-module.c:1760 kmod_module_get_initstate: could not open '/sys/module/idxd_mdev/initstate': No such file or directory
libkmod: DEBUG ../libkmod/libkmod-module.c:1770 kmod_module_get_initstate: could not open '/sys/module/idxd_mdev': No such file or directory
libkmod: DEBUG ../libkmod/libkmod-module.c:468 kmod_module_unref: kmod_module 0x55cf1eb228a0 released
libkmod: DEBUG ../libkmod/libkmod.c:419 kmod_pool_del_module: del 0x55cf1eb228a0 key='idxd_mdev'
libkmod: DEBUG ../libkmod/libkmod.c:511 lookup_builtin_file: file=/lib/modules/5.14.8-051408-generic/modules.builtin.bin modname=idxd
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: devices_init: device is initialized already
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: devices_init: device is initialized already
libaccfg: groups_init: group is initialized already
libaccfg: groups_init: group is initialized already
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: engines_init: engine is initialized already
libaccfg: engines_init: engine is initialized already
libaccfg: engines_init: engine is initialized already
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: __sysfs_device_parse: 0: processed
libaccfg: wqs_init: wq is initialized already
libaccfg: wqs_init: wq is initialized already
libaccfg: wqs_init: wq is initialized already

Running accfg-test0: set and get configurations for shared wqs
libaccfg: wqs_init: wq is initialized already
configuring device dsa0
configuring group group0.0
configuring group group0.1
configuring wq wq0.0
libaccfg: write_attr: failed to write shared to /sys/devices/pci0000:6a/0000:6a:01.0/dsa0/wq0.0/mode: Invalid argument
libaccfg: accfg_wq_set_str_mode: wq0.0: write failed: Invalid argument
config wq failed
shared wq support not available
libaccfg: wqs_init: wq is initialized already
accfg-test0 *skipped*: required feature not found

Running accfg-test1: set and get configurations for dedicated wqs
libaccfg: wqs_init: wq is initialized already
configuring device dsa0
configuring group group0.0
configuring group group0.1
configuring wq wq0.1
configuring wq wq0.3
configuring engine engine0.0
configuring engine engine0.1
configuring engine engine0.2
configuring engine engine0.3
check device dsa0
check group group0.0
check group group0.1
check wq wq0.1
check wq wq0.3
check engine engine0.0
check engine engine0.1
check engine engine0.2
check engine engine0.3
libaccfg: wqs_init: wq is initialized already
accfg-test1 passed!

Running accfg-test2: max wq size
libaccfg: wqs_init: wq is initialized already
configuring group group0.0
configuring wq wq0.1
configuring wq wq0.3
trying to set wq size exceeding max wq size
libaccfg: write_attr: failed to write 128 to /sys/devices/pci0000:6a/0000:6a:01.0/dsa0/wq0.3/size: Invalid argument
libaccfg: accfg_wq_set_size: wq0.3: write failed: Invalid argument
wq size exceeding max wq size was not accepted
libaccfg: wqs_init: wq is initialized already
accfg-test2 passed!

Running accfg-test3: wq boundary conditions
libaccfg: wqs_init: wq is initialized already
configure device dsa0, group group0.0, wq wq0.1 for bounds test
trying to set wq max_batch_size = 0
libaccfg: write_attr: failed to write 0 to /sys/devices/pci0000:6a/0000:6a:01.0/dsa0/wq0.1/max_batch_size: Invalid argument
libaccfg: accfg_wq_set_max_batch_size: wq0.1: write failed: Invalid argument
trying to set wq max_transfer_size = 0
libaccfg: write_attr: failed to write 0 to /sys/devices/pci0000:6a/0000:6a:01.0/dsa0/wq0.1/max_transfer_size: Invalid argument
libaccfg: accfg_wq_set_max_transfer_size: wq0.1: write failed: Invalid argument
trying to set wq max_batch_size exceeding device max
libaccfg: write_attr: failed to write 2048 to /sys/devices/pci0000:6a/0000:6a:01.0/dsa0/wq0.1/max_batch_size: Invalid argument
libaccfg: accfg_wq_set_max_batch_size: wq0.1: write failed: Invalid argument
trying to set wq max_transfer_size exceeding device max
libaccfg: write_attr: failed to write 4294967296 to /sys/devices/pci0000:6a/0000:6a:01.0/dsa0/wq0.1/max_transfer_size: Invalid argument
libaccfg: accfg_wq_set_max_transfer_size: wq0.1: write failed: Invalid argument
0 and greater than device max values were not accepted
libaccfg: wqs_init: wq is initialized already
accfg-test3 passed!

Running accfg-test4: 1swq type mdev creation and removal
accfg-test4 *skipped*: required feature not found

Running accfg-test5: 1dwq type mdev creation and removal
accfg-test5 *skipped*: required feature not found
libaccfg: devices_init: device is initialized already
libaccfg: wqs_init: wq is initialized already
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 10: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 12: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 14: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 2: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 4: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 6: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 8: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 1: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 11: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 13: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 15: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 3: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 5: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 7: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libaccfg: __sysfs_device_parse: 9: processed
libkmod: DEBUG ../libkmod/libkmod-module.c:468 kmod_module_unref: kmod_module 0x55cf1eb22790 released
libkmod: DEBUG ../libkmod/libkmod.c:419 kmod_pool_del_module: del 0x55cf1eb22790 key='idxd'
libkmod: INFO ../libkmod/libkmod.c:332 kmod_unref: context 0x55cf1eb22390 released
test-libaccfg: PASS
SUCCESS!
libaccfg: accfg_unref: context 0x55cf1eb21f00 released

Even though the patches in the pending branch helped in another area (skipping shared memory tests if not available), that is effective only when configuring wqs. The error you reported happened earlier while configuring groups. Since you also don't see any error whiling echoing the values, I think it must have been some unstable state.

If you try the current stable branch, it will fail at enabling wq but I think the group0.1 configuring will pass.

I will be making another release shortly that will update the stable branch with the patches in the pending branch.

The stable branch now has the changes in the pending branch. Current release is 3.4.2.

even though it throws some write errors.

Those are negative tests and getting error is the expected result. The first error writing to mode is also expected. It tries setting mode = "shared" and if it fails then it means shared memory support is not available and it skips tests requiring that feature.

Even though the patches in the pending branch helped in another area (skipping shared memory tests if not available), that is effective only when configuring wqs. The error you reported happened earlier while configuring groups. Since you also don't see any error whiling echoing the values, I think it must have been some unstable state.

If you try the current stable branch, it will fail at enabling wq but I think the group0.1 configuring will pass.

I will be making another release shortly that will update the stable branch with the patches in the pending branch.

I might have celebrated too early. Accel-config passes all the tests, but fails when trying to configure a wq with similar error. I've tried different arguments and it fails all the same. So I decided to take the previous approach you mentioned, of configuring it manually by setting one value at a time. Below is the results of that I've tried multiple different values for the threshold without success, shared mode failed to be written too. Don't mind the bind error, it's a typo on my part.

Strangely, the wqs do appear afterwards and seem to be correctly bound. I even run a simple test program which enqueues as copy with movdir64b and it has succeeded in using the working queue.

Running the script with manual DSA setup: Link to the script gist dmesg output after running the script:

[ 6510.020035] dsa dsa0: dsa0 matched
[ 6510.020067] dsa dsa0: idxd_config_bus_probe called
[ 6510.020078] idxd 0000:6a:01.0: WQ 0 CFGs
[ 6510.020083] idxd 0000:6a:01.0: WQ[0][0][0x400]: 0x10
[ 6510.020090] idxd 0000:6a:01.0: WQ[0][1][0x404]: 0x0
[ 6510.020095] idxd 0000:6a:01.0: WQ[0][2][0x408]: 0x100000a3
[ 6510.020099] idxd 0000:6a:01.0: WQ[0][3][0x40c]: 0x15f
[ 6510.020103] idxd 0000:6a:01.0: WQ[0][4][0x410]: 0x0
[ 6510.020107] idxd 0000:6a:01.0: WQ[0][5][0x414]: 0x0
[ 6510.020111] idxd 0000:6a:01.0: WQ[0][6][0x418]: 0x0
[ 6510.020115] idxd 0000:6a:01.0: WQ[0][7][0x41c]: 0x0
[ 6510.020122] idxd 0000:6a:01.0: WQ 1 CFGs
[ 6510.020125] idxd 0000:6a:01.0: WQ[1][0][0x420]: 0x10
[ 6510.020129] idxd 0000:6a:01.0: WQ[1][1][0x424]: 0x0
[ 6510.020133] idxd 0000:6a:01.0: WQ[1][2][0x428]: 0x100000a3
[ 6510.020137] idxd 0000:6a:01.0: WQ[1][3][0x42c]: 0x15f
[ 6510.020141] idxd 0000:6a:01.0: WQ[1][4][0x430]: 0x0
[ 6510.020145] idxd 0000:6a:01.0: WQ[1][5][0x434]: 0x0
[ 6510.020148] idxd 0000:6a:01.0: WQ[1][6][0x438]: 0x0
[ 6510.020152] idxd 0000:6a:01.0: WQ[1][7][0x43c]: 0x0
[ 6510.020156] idxd 0000:6a:01.0: GENCFG(0x80): 0x60
[ 6510.020159] idxd 0000:6a:01.0: Writing group 0 cfg registers
[ 6510.020163] idxd 0000:6a:01.0: GRPCFG wq[0:0: 0x300]: 0x1
[ 6510.020168] idxd 0000:6a:01.0: GRPCFG wq[0:1: 0x308]: 0x0
[ 6510.020172] idxd 0000:6a:01.0: GRPCFG wq[0:2: 0x310]: 0x0
[ 6510.020176] idxd 0000:6a:01.0: GRPCFG wq[0:3: 0x318]: 0x0
[ 6510.020180] idxd 0000:6a:01.0: GRPCFG engs[0: 0x320]: 0x3
[ 6510.020184] idxd 0000:6a:01.0: GRPFLAGS flags[0: 0x328]: 0x6000008
[ 6510.020188] idxd 0000:6a:01.0: Writing group 1 cfg registers
[ 6510.020191] idxd 0000:6a:01.0: GRPCFG wq[1:0: 0x340]: 0x2
[ 6510.020195] idxd 0000:6a:01.0: GRPCFG wq[1:1: 0x348]: 0x0
[ 6510.020199] idxd 0000:6a:01.0: GRPCFG wq[1:2: 0x350]: 0x0
[ 6510.020203] idxd 0000:6a:01.0: GRPCFG wq[1:3: 0x358]: 0x0
[ 6510.020207] idxd 0000:6a:01.0: GRPCFG engs[1: 0x360]: 0xc
[ 6510.020210] idxd 0000:6a:01.0: GRPFLAGS flags[1: 0x368]: 0x6000008
[ 6510.020213] idxd 0000:6a:01.0: Writing group 2 cfg registers
[ 6510.020217] idxd 0000:6a:01.0: GRPCFG wq[2:0: 0x380]: 0x0
[ 6510.020221] idxd 0000:6a:01.0: GRPCFG wq[2:1: 0x388]: 0x0
[ 6510.020224] idxd 0000:6a:01.0: GRPCFG wq[2:2: 0x390]: 0x0
[ 6510.020228] idxd 0000:6a:01.0: GRPCFG wq[2:3: 0x398]: 0x0
[ 6510.020232] idxd 0000:6a:01.0: GRPCFG engs[2: 0x3a0]: 0x0
[ 6510.020236] idxd 0000:6a:01.0: GRPFLAGS flags[2: 0x3a8]: 0x6000008
[ 6510.020239] idxd 0000:6a:01.0: Writing group 3 cfg registers
[ 6510.020243] idxd 0000:6a:01.0: GRPCFG wq[3:0: 0x3c0]: 0x0
[ 6510.020247] idxd 0000:6a:01.0: GRPCFG wq[3:1: 0x3c8]: 0x0
[ 6510.020252] idxd 0000:6a:01.0: GRPCFG wq[3:2: 0x3d0]: 0x0
[ 6510.020256] idxd 0000:6a:01.0: GRPCFG wq[3:3: 0x3d8]: 0x0
[ 6510.020261] idxd 0000:6a:01.0: GRPCFG engs[3: 0x3e0]: 0x0
[ 6510.020265] idxd 0000:6a:01.0: GRPFLAGS flags[3: 0x3e8]: 0x6000008
[ 6510.020269] idxd 0000:6a:01.0: idxd_cmd_exec: sending cmd: 0x1 op: 0x0
[ 6510.020531] dsa dsa0: Device dsa0 enabled
[ 6510.021990] dsa wq0.0: wq0.0 matched
[ 6510.022009] dsa wq0.0: idxd_config_bus_probe called
[ 6510.022020] idxd 0000:6a:01.0: WQ 0 CFGs
[ 6510.022025] idxd 0000:6a:01.0: WQ[0][0][0x400]: 0x10
[ 6510.022031] idxd 0000:6a:01.0: WQ[0][1][0x404]: 0x0
[ 6510.022035] idxd 0000:6a:01.0: WQ[0][2][0x408]: 0x100000a3
[ 6510.022040] idxd 0000:6a:01.0: WQ[0][3][0x40c]: 0x15f
[ 6510.022044] idxd 0000:6a:01.0: WQ[0][4][0x410]: 0x0
[ 6510.022048] idxd 0000:6a:01.0: WQ[0][5][0x414]: 0x0
[ 6510.022051] idxd 0000:6a:01.0: WQ[0][6][0x418]: 0x0
[ 6510.022055] idxd 0000:6a:01.0: WQ[0][7][0x41c]: 0x0
[ 6510.022062] idxd 0000:6a:01.0: WQ 1 CFGs
[ 6510.022065] idxd 0000:6a:01.0: WQ[1][0][0x420]: 0x10
[ 6510.022069] idxd 0000:6a:01.0: WQ[1][1][0x424]: 0x0
[ 6510.022072] idxd 0000:6a:01.0: WQ[1][2][0x428]: 0x100000a3
[ 6510.022076] idxd 0000:6a:01.0: WQ[1][3][0x42c]: 0x15f
[ 6510.022080] idxd 0000:6a:01.0: WQ[1][4][0x430]: 0x0
[ 6510.022083] idxd 0000:6a:01.0: WQ[1][5][0x434]: 0x0
[ 6510.022087] idxd 0000:6a:01.0: WQ[1][6][0x438]: 0x0
[ 6510.022090] idxd 0000:6a:01.0: WQ[1][7][0x43c]: 0x0
[ 6510.022093] idxd 0000:6a:01.0: GENCFG(0x80): 0x60
[ 6510.022096] idxd 0000:6a:01.0: Writing group 0 cfg registers
[ 6510.022100] idxd 0000:6a:01.0: GRPCFG wq[0:0: 0x300]: 0x1
[ 6510.022105] idxd 0000:6a:01.0: GRPCFG wq[0:1: 0x308]: 0x0
[ 6510.022109] idxd 0000:6a:01.0: GRPCFG wq[0:2: 0x310]: 0x0
[ 6510.022112] idxd 0000:6a:01.0: GRPCFG wq[0:3: 0x318]: 0x0
[ 6510.022116] idxd 0000:6a:01.0: GRPCFG engs[0: 0x320]: 0x3
[ 6510.022120] idxd 0000:6a:01.0: GRPFLAGS flags[0: 0x328]: 0x6000008
[ 6510.022123] idxd 0000:6a:01.0: Writing group 1 cfg registers
[ 6510.022126] idxd 0000:6a:01.0: GRPCFG wq[1:0: 0x340]: 0x2
[ 6510.022130] idxd 0000:6a:01.0: GRPCFG wq[1:1: 0x348]: 0x0
[ 6510.022134] idxd 0000:6a:01.0: GRPCFG wq[1:2: 0x350]: 0x0
[ 6510.022137] idxd 0000:6a:01.0: GRPCFG wq[1:3: 0x358]: 0x0
[ 6510.022141] idxd 0000:6a:01.0: GRPCFG engs[1: 0x360]: 0xc
[ 6510.022144] idxd 0000:6a:01.0: GRPFLAGS flags[1: 0x368]: 0x6000008
[ 6510.022147] idxd 0000:6a:01.0: Writing group 2 cfg registers
[ 6510.022150] idxd 0000:6a:01.0: GRPCFG wq[2:0: 0x380]: 0x0
[ 6510.022154] idxd 0000:6a:01.0: GRPCFG wq[2:1: 0x388]: 0x0
[ 6510.022158] idxd 0000:6a:01.0: GRPCFG wq[2:2: 0x390]: 0x0
[ 6510.022163] idxd 0000:6a:01.0: GRPCFG wq[2:3: 0x398]: 0x0
[ 6510.022167] idxd 0000:6a:01.0: GRPCFG engs[2: 0x3a0]: 0x0
[ 6510.022171] idxd 0000:6a:01.0: GRPFLAGS flags[2: 0x3a8]: 0x6000008
[ 6510.022174] idxd 0000:6a:01.0: Writing group 3 cfg registers
[ 6510.022178] idxd 0000:6a:01.0: GRPCFG wq[3:0: 0x3c0]: 0x0
[ 6510.022182] idxd 0000:6a:01.0: GRPCFG wq[3:1: 0x3c8]: 0x0
[ 6510.022186] idxd 0000:6a:01.0: GRPCFG wq[3:2: 0x3d0]: 0x0
[ 6510.022190] idxd 0000:6a:01.0: GRPCFG wq[3:3: 0x3d8]: 0x0
[ 6510.022195] idxd 0000:6a:01.0: GRPCFG engs[3: 0x3e0]: 0x0
[ 6510.022198] idxd 0000:6a:01.0: GRPFLAGS flags[3: 0x3e8]: 0x6000008
[ 6510.022202] idxd 0000:6a:01.0: idxd_cmd_exec: sending cmd: 0x6 op: 0x0
[ 6510.022456] idxd 0000:6a:01.0: WQ 0 enabled
[ 6510.023003] idxd 0000:6a:01.0: wq wq0.0 enabled
[ 6510.023137] dsa wq0.1: wq0.1 matched
[ 6510.023156] dsa wq0.1: idxd_config_bus_probe called
[ 6510.023168] idxd 0000:6a:01.0: WQ 0 CFGs
[ 6510.023173] idxd 0000:6a:01.0: WQ[0][0][0x400]: 0x10
[ 6510.023179] idxd 0000:6a:01.0: WQ[0][1][0x404]: 0x0
[ 6510.023183] idxd 0000:6a:01.0: WQ[0][2][0x408]: 0x100000a3
[ 6510.023188] idxd 0000:6a:01.0: WQ[0][3][0x40c]: 0x15f
[ 6510.023192] idxd 0000:6a:01.0: WQ[0][4][0x410]: 0x0
[ 6510.023196] idxd 0000:6a:01.0: WQ[0][5][0x414]: 0x0
[ 6510.023200] idxd 0000:6a:01.0: WQ[0][6][0x418]: 0x40000000
[ 6510.023204] idxd 0000:6a:01.0: WQ[0][7][0x41c]: 0x0
[ 6510.023211] idxd 0000:6a:01.0: WQ 1 CFGs
[ 6510.023215] idxd 0000:6a:01.0: WQ[1][0][0x420]: 0x10
[ 6510.023218] idxd 0000:6a:01.0: WQ[1][1][0x424]: 0x0
[ 6510.023222] idxd 0000:6a:01.0: WQ[1][2][0x428]: 0x100000a3
[ 6510.023226] idxd 0000:6a:01.0: WQ[1][3][0x42c]: 0x15f
[ 6510.023230] idxd 0000:6a:01.0: WQ[1][4][0x430]: 0x0
[ 6510.023233] idxd 0000:6a:01.0: WQ[1][5][0x434]: 0x0
[ 6510.023236] idxd 0000:6a:01.0: WQ[1][6][0x438]: 0x0
[ 6510.023240] idxd 0000:6a:01.0: WQ[1][7][0x43c]: 0x0
[ 6510.023244] idxd 0000:6a:01.0: GENCFG(0x80): 0x60
[ 6510.023246] idxd 0000:6a:01.0: Writing group 0 cfg registers
[ 6510.023251] idxd 0000:6a:01.0: GRPCFG wq[0:0: 0x300]: 0x1
[ 6510.023255] idxd 0000:6a:01.0: GRPCFG wq[0:1: 0x308]: 0x0
[ 6510.023259] idxd 0000:6a:01.0: GRPCFG wq[0:2: 0x310]: 0x0
[ 6510.023263] idxd 0000:6a:01.0: GRPCFG wq[0:3: 0x318]: 0x0
[ 6510.023266] idxd 0000:6a:01.0: GRPCFG engs[0: 0x320]: 0x3
[ 6510.023270] idxd 0000:6a:01.0: GRPFLAGS flags[0: 0x328]: 0x6000008
[ 6510.023274] idxd 0000:6a:01.0: Writing group 1 cfg registers
[ 6510.023277] idxd 0000:6a:01.0: GRPCFG wq[1:0: 0x340]: 0x2
[ 6510.023281] idxd 0000:6a:01.0: GRPCFG wq[1:1: 0x348]: 0x0
[ 6510.023285] idxd 0000:6a:01.0: GRPCFG wq[1:2: 0x350]: 0x0
[ 6510.023288] idxd 0000:6a:01.0: GRPCFG wq[1:3: 0x358]: 0x0
[ 6510.023292] idxd 0000:6a:01.0: GRPCFG engs[1: 0x360]: 0xc
[ 6510.023296] idxd 0000:6a:01.0: GRPFLAGS flags[1: 0x368]: 0x6000008
[ 6510.023300] idxd 0000:6a:01.0: Writing group 2 cfg registers
[ 6510.023303] idxd 0000:6a:01.0: GRPCFG wq[2:0: 0x380]: 0x0
[ 6510.023307] idxd 0000:6a:01.0: GRPCFG wq[2:1: 0x388]: 0x0
[ 6510.023311] idxd 0000:6a:01.0: GRPCFG wq[2:2: 0x390]: 0x0
[ 6510.023315] idxd 0000:6a:01.0: GRPCFG wq[2:3: 0x398]: 0x0
[ 6510.023318] idxd 0000:6a:01.0: GRPCFG engs[2: 0x3a0]: 0x0
[ 6510.023322] idxd 0000:6a:01.0: GRPFLAGS flags[2: 0x3a8]: 0x6000008
[ 6510.023324] idxd 0000:6a:01.0: Writing group 3 cfg registers
[ 6510.023328] idxd 0000:6a:01.0: GRPCFG wq[3:0: 0x3c0]: 0x0
[ 6510.023332] idxd 0000:6a:01.0: GRPCFG wq[3:1: 0x3c8]: 0x0
[ 6510.023336] idxd 0000:6a:01.0: GRPCFG wq[3:2: 0x3d0]: 0x0
[ 6510.023340] idxd 0000:6a:01.0: GRPCFG wq[3:3: 0x3d8]: 0x0
[ 6510.023344] idxd 0000:6a:01.0: GRPCFG engs[3: 0x3e0]: 0x0
[ 6510.023348] idxd 0000:6a:01.0: GRPFLAGS flags[3: 0x3e8]: 0x6000008
[ 6510.023351] idxd 0000:6a:01.0: idxd_cmd_exec: sending cmd: 0x6 op: 0x1
[ 6510.023664] idxd 0000:6a:01.0: WQ 1 enabled
[ 6510.024159] idxd 0000:6a:01.0: wq wq0.1 enabled

All those new failures are expected if you are using kernel 5.13 or later. Upstream has reverted ENQCMD support and therefore shared wq support is not supported. We are trying to get it reinstated. Hopefully by 5.16. Until then, only dedicated wq support possible.

All those new failures are expected if you are using kernel 5.13 or later. Upstream has reverted ENQCMD support and therefore shared wq support is not supported. We are trying to get it reinstated. Hopefully by 5.16. Until then, only dedicated wq support possible.

Oh, that's fair enough then. Does the 5.12 support the shared wq? I have seen in the idxd wiki that it has been originally tested on that.

Yes I think so. Although there are issues with later 5.12 stable
  series. They picked up a bunch of bug fixes, but missed a patch.
  By the time I discovered it, 5.12 already closed. I think it's
  something like 5.12.10 or later. I can't remember. So basically
  early 5.12.x works, later ones missing a fix patch and breaks the
  probe.

On 9/30/2021 8:36 AM, rrosa-cn wrote:

    All those new failures are expected if you are using kernel
      5.13 or later. Upstream has reverted ENQCMD support and
      therefore shared wq support is not supported. We are trying to
      get it reinstated. Hopefully by 5.16. Until then, only
      dedicated wq support possible.

  Oh, that's fair enough then. Does the 5.12 support the shared
    wq? I have seen in the idxd wiki that it has been originally
    tested on that.
  —
    You are receiving this because you were mentioned.
    Reply to this email directly, view it on GitHub, or unsubscribe.
    Triage notifications on the go with GitHub Mobile for iOS or Android.

  [

{ @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/intel/idxd-config/issues/9#issuecomment-931435467", "url": "https://github.com/intel/idxd-config/issues/9#issuecomment-931435467", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

intel / idxd-config

Accel-config test fails - libaccfg: write_attr: failed to write #9