search

redhat-performance / tuned

Tuning Profile Delivery Mechanism for Linux
GNU General Public License v2.0
844 stars 180 forks source link

Set elevator for disk fails #101

Open kees-closed opened 6 years ago

kees-closed commented 6 years ago

I'm trying to configure a different elevator for different disks, I've setup the following Ansible template that and seems to works well.

[main]
summary=Customized low-power usage profile
include=powersave

[bootloader]
cmdline=pcie_aspm=force

[sysctl]
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

{% for item in query('dict', ansible_devices) %}
{% if not item.key.startswith('dm-') %}
{% if item.value.support_discard is defined and item.value.support_discard != "0" %}
[disk_{{ item.key }}]
type=disk
devices={{ item.key }}
elevator=none

{% else %}
[disk_{{ item.key }}]
type=disk
devices={{ item.key }}
elevator=cfq

{% endif %}
{% endif %}
{% endfor %}

It outputs to the following.

[main]
summary=Customized balanced-power usage profile
include=balanced

[bootloader]
cmdline=pcie_aspm=force

[sysctl]
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

[disk_nvme0n1]
type=disk
devices=nvme0n1
elevator=none

But then complains that tuned.plugins.base: instance disk_nvme0n1: no matching devices available.

2018-05-30 14:02:39,382 WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0' 
2018-05-30 14:02:39,407 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2018-05-30 14:02:39,500 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2018-05-30 14:02:39,504 INFO     tuned.daemon.daemon: static tuning from profile 'custom-balanced' applied
2018-05-30 14:03:09,602 INFO     tuned.daemon.controller: terminating controller
2018-05-30 14:03:09,602 INFO     tuned.daemon.daemon: stopping tuning
2018-05-30 14:03:10,582 INFO     tuned.daemon.daemon: terminating Tuned, rolling back all changes
2018-05-30 14:03:10,645 INFO     tuned.plugins.plugin_bootloader: removing grub2 tuning previously added by Tuned
2018-05-30 14:03:10,647 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu2'
2018-05-30 14:03:10,647 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu3'
2018-05-30 14:03:10,647 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu0'
2018-05-30 14:03:10,647 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu1'
2018-05-30 14:03:10,752 INFO     tuned.daemon.application: dynamic tuning is enabled (can be overridden in plugins)
2018-05-30 14:03:10,756 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2018-05-30 14:03:10,756 INFO     tuned.daemon.daemon: dynamic tuning is enabled (can be overridden by plugins)
2018-05-30 14:03:10,756 INFO     tuned.daemon.daemon: using update interval of 10 second(s) (10 times of the sleep interval)
2018-05-30 14:03:10,757 INFO     tuned.profiles.loader: loading profile: custom-balanced
2018-05-30 14:03:10,760 INFO     tuned.daemon.controller: starting controller
2018-05-30 14:03:10,760 INFO     tuned.daemon.daemon: starting tuning
2018-05-30 14:03:10,769 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1
2018-05-30 14:03:10,772 INFO     tuned.plugins.plugin_cpu: intel_pstate detected
2018-05-30 14:03:10,772 INFO     tuned.plugins.base: instance audio: assigning devices snd_hda_intel
2018-05-30 14:03:10,773 INFO     tuned.plugins.base: instance video: assigning devices card0
2018-05-30 14:03:10,773 INFO     tuned.plugins.base: instance disk: assigning devices dm-6, dm-7, dm-4, dm-5, dm-2, dm-3, dm-0, dm-1
2018-05-30 14:03:10,773 WARNING  tuned.plugins.base: instance scsi_host: no matching devices available
2018-05-30 14:03:10,773 WARNING  tuned.plugins.base: instance disk_nvme0n1: no matching devices available
2018-05-30 14:03:10,774 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported
2018-05-30 14:03:10,774 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu3', it is not supported
2018-05-30 14:03:10,774 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu0', it is not supported
2018-05-30 14:03:10,774 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu1', it is not supported
2018-05-30 14:03:10,777 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu2'
2018-05-30 14:03:10,783 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu3'
2018-05-30 14:03:10,788 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu0'
2018-05-30 14:03:10,794 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu1'
2018-05-30 14:03:10,796 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 100
2018-05-30 14:03:10,797 ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card0/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card0/device/power_method''
2018-05-30 14:03:10,798 WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0' 
2018-05-30 14:03:10,823 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2018-05-30 14:03:10,915 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2018-05-30 14:03:10,919 INFO     tuned.daemon.daemon: static tuning from profile 'custom-balanced' applied
2018-05-30 14:05:51,117 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 1000
2018-05-30 14:06:21,156 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 100
"/var/log/tuned/tuned.log" [readonly] 560L, 58949C

While that disk is really there.

# lsblk
NAME                                            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme0n1                                         259:0    0 238,5G  0 disk  
├─nvme0n1p1                                     259:1    0   100M  0 part  /boot/efi
├─nvme0n1p2                                     259:2    0     1G  0 part  /boot
└─nvme0n1p3                                     259:3    0 237,4G  0 part

The current elevator is set to the following, when I change it to 'mq-deadline', tuned-adm won't set it back to 'none'.

$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline

According to the documentation, my config should be okay. Could this be a bug or am I doing something wrong here?

olysonek commented 6 years ago

Does /dev/nvme0n1 exist?

olysonek commented 6 years ago

Try adding the following to the profile: [disk] devices=!nvme0n1

olysonek commented 6 years ago

Sorry, on separate lines:

[disk]
devices=!nvme0n1
kees-closed commented 6 years ago 
# ls -l /dev/nvme0n1*
brw-rw----. 1 root disk 259, 0 30 mei 12:17 /dev/nvme0n1
brw-rw----. 1 root disk 259, 1 30 mei 12:17 /dev/nvme0n1p1
brw-rw----. 1 root disk 259, 2 30 mei 12:17 /dev/nvme0n1p2
brw-rw----. 1 root disk 259, 3 30 mei 12:17 /dev/nvme0n1p3

When I add a '!' as a prefix, the error is gone, but this means the disk is excluded, correct?

kees-closed commented 6 years ago

By the way, I'm using tuned-2.9.0-8.fc28.noarch.

olysonek commented 6 years ago

What I meant was to try having both the [disk_nvme0n1] and [disk] sections in the config:

[disk_nvme0n1]
type=disk
devices=nvme0n1
elevator=none

[disk]
devices=!nvme0n1

The sections are called "instances" (of the disk plugin in this case) in Tuned terminology. What I think the problem is, is that the profile you're including in your custom profile (powersave or balanced) already defines an instance of the disk plugin called "disk":

[disk]
# Comma separated list of devices, all devices if commented out.
# devices=sda

The problem is that this instance will take over all disk devices in your system, so there will be no devices left for your disk_nvme0n1 instance. The solution would be to override the included instance, so that it doesn't "eat" the nvme0n1 device, hence:

[disk]
devices=!nvme0n1
kees-closed commented 6 years ago

Below is the log when I comment the include of 'balanced' in my custom config.

2018-05-30 16:25:05,411 INFO     tuned.daemon.daemon: stopping tuning
2018-05-30 16:25:05,971 INFO     tuned.plugins.plugin_bootloader: removing grub2 tuning previously added by Tuned
2018-05-30 16:25:05,973 INFO     tuned.profiles.loader: loading profile: custom-balanced
2018-05-30 16:25:05,974 INFO     tuned.daemon.daemon: starting tuning
2018-05-30 16:25:05,979 WARNING  tuned.plugins.base: instance disk_nvme0n1: no matching devices available
2018-05-30 16:25:05,979 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2018-05-30 16:25:06,073 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2018-05-30 16:25:06,077 INFO     tuned.daemon.daemon: static tuning from profile 'custom-balanced' applied

Below is the log when I still include 'balanced', but include your suggestion as well.

2018-05-30 16:26:52,992 INFO     tuned.daemon.daemon: stopping tuning
2018-05-30 16:26:53,276 INFO     tuned.plugins.plugin_bootloader: removing grub2 tuning previously added by Tuned
2018-05-30 16:26:53,278 INFO     tuned.profiles.loader: loading profile: custom-balanced
2018-05-30 16:26:53,279 INFO     tuned.daemon.daemon: starting tuning
2018-05-30 16:26:53,286 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1
2018-05-30 16:26:53,289 INFO     tuned.plugins.plugin_cpu: intel_pstate detected
2018-05-30 16:26:53,289 INFO     tuned.plugins.base: instance audio: assigning devices snd_hda_intel
2018-05-30 16:26:53,290 INFO     tuned.plugins.base: instance video: assigning devices card0
2018-05-30 16:26:53,290 INFO     tuned.plugins.base: instance disk: assigning devices dm-6, dm-7, dm-4, dm-5, dm-2, dm-3, dm-0, dm-1
2018-05-30 16:26:53,290 WARNING  tuned.plugins.base: instance scsi_host: no matching devices available
2018-05-30 16:26:53,290 WARNING  tuned.plugins.base: instance disk_nvme0n1: no matching devices available
2018-05-30 16:26:53,291 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported
2018-05-30 16:26:53,291 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu3', it is not supported
2018-05-30 16:26:53,291 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu0', it is not supported
2018-05-30 16:26:53,291 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu1', it is not supported
2018-05-30 16:26:53,294 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu2'
2018-05-30 16:26:53,300 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu3'
2018-05-30 16:26:53,306 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu0'
2018-05-30 16:26:53,311 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu1'
2018-05-30 16:26:53,314 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 100
2018-05-30 16:26:53,315 ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card0/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card0/device/power_method''
2018-05-30 16:26:53,315 WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0'
2018-05-30 16:26:53,341 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2018-05-30 16:26:53,437 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2018-05-30 16:26:53,442 INFO     tuned.daemon.daemon: static tuning from profile 'custom-balanced' applied

The warning in the logs persists, even when the 'balanced' profile or disk section in the 'balanced' profile is excluded. The elevator in /sys/block/nvme0n1/queue/scheduler remains unaffected.

olysonek commented 6 years ago

Hmm, I didn't expect that. Right now this looks like a bug to me. I'll take a look at this later.

olysonek commented 6 years ago

What I missed before is that the device was not even assigned to the "disk" instance from the balanced profile:

2018-05-30 14:03:10,773 INFO     tuned.plugins.base: instance disk: assigning devices dm-6, dm-7, dm-4, dm-5, dm-2, dm-3, dm-0, dm-1

I think Tuned doesn't recognize these NVMe drives as supported. Could you provide the output of the following Python script? It gathers some basic info about the device.

from pyudev import Context, Devices

c = Context()
d = Devices.from_path(c, '/sys/block/nvme0n1')
print("Device type: %s" % d.device_type)
print("Removable: %s" % d.attributes.get("removable", None))
if d.parent is None:
    print("No subsystem")
else:
    print("Subsystem: %s" % d.parent.subsystem)
kees-closed commented 6 years ago

The output of the script is as follows:

$ python test.py
Device type: disk
Removable: 0
Subsystem: nvme
kees-closed commented 6 years ago

I also ran the Ansible template on another machine, it also seem to happen with other type of disks.

lsblk output

# lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                         8:0    0 223,6G  0 disk
├─sda1                      8:1    0   100M  0 part  /boot/efi
├─sda2                      8:2    0     1G  0 part  /boot
└─sda3                      8:3    0 222,5G  0 part
  ├─fedora_sovereign-root 253:0    0 112,5G  0 lvm   /
  ├─fedora_sovereign-swap 253:1    0    10G  0 lvm   [SWAP]
  └─fedora_sovereign-var  253:5    0   100G  0 lvm   /var
sdb                         8:16   0   1,8T  0 disk
└─sdb1                      8:17   0   1,8T  0 part
  └─md0                     9:0    0   5,5T  0 raid5
    ├─raid5-home          253:2    0   300G  0 lvm   /home
    ├─raid5-backups       253:3    0     1T  0 lvm   /srv/backups
    └─raid5-library       253:4    0   4,2T  0 lvm   /srv/libray
sdc                         8:32   0   1,8T  0 disk
└─sdc1                      8:33   0   1,8T  0 part
  └─md0                     9:0    0   5,5T  0 raid5
    ├─raid5-home          253:2    0   300G  0 lvm   /home
    ├─raid5-backups       253:3    0     1T  0 lvm   /srv/backups
    └─raid5-library       253:4    0   4,2T  0 lvm   /srv/libray
sdd                         8:48   0   1,8T  0 disk
└─sdd1                      8:49   0   1,8T  0 part
  └─md0                     9:0    0   5,5T  0 raid5
    ├─raid5-home          253:2    0   300G  0 lvm   /home
    ├─raid5-backups       253:3    0     1T  0 lvm   /srv/backups
    └─raid5-library       253:4    0   4,2T  0 lvm   /srv/libray
sde                         8:64   0   1,8T  0 disk
└─sde1                      8:65   0   1,8T  0 part
  └─md0                     9:0    0   5,5T  0 raid5
    ├─raid5-home          253:2    0   300G  0 lvm   /home
    ├─raid5-backups       253:3    0     1T  0 lvm   /srv/backups
    └─raid5-library       253:4    0   4,2T  0 lvm   /srv/libray

tuned profile

[main]
summary=Customized balanced-power usage profile
include=balanced

[bootloader]
cmdline=pcie_aspm=force

[sysctl]
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

[disk_sdi]
type=disk
devices=sdi
elevator=cfq

[disk_sdd]
type=disk
devices=sdd
elevator=cfq

[disk_sde]
type=disk
devices=sde
elevator=cfq

[disk_sdf]
type=disk
devices=sdf
elevator=cfq

[disk_sdg]
type=disk
devices=sdg
elevator=cfq

[disk_sda]
type=disk
devices=sda
elevator=none

[disk_sdb]
type=disk
devices=sdb
elevator=cfq

[disk_sdc]
type=disk
devices=sdc
elevator=cfq

[disk_sdh]
type=disk
devices=sdh
elevator=cfq

tuned.log

2018-06-04 19:13:49,361 INFO     tuned.daemon.daemon: stopping tuning
2018-06-04 19:13:50,239 INFO     tuned.plugins.plugin_bootloader: removing grub2 tuning previously added by Tuned
2018-06-04 19:13:50,243 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu6'
2018-06-04 19:13:50,244 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu7'
2018-06-04 19:13:50,244 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu4'
2018-06-04 19:13:50,244 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu5'
2018-06-04 19:13:50,245 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu2'
2018-06-04 19:13:50,245 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu3'
2018-06-04 19:13:50,245 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu0'
2018-06-04 19:13:50,246 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu1'
2018-06-04 19:13:50,247 INFO     tuned.profiles.loader: loading profile: custom-balanced
2018-06-04 19:13:50,252 INFO     tuned.daemon.daemon: starting tuning
2018-06-04 19:13:50,271 INFO     tuned.plugins.base: instance cpu: assigning devices cpu6, cpu7, cpu4, cpu5, cpu2, cpu3, cpu0, cpu1
2018-06-04 19:13:50,275 INFO     tuned.plugins.plugin_cpu: intel_pstate detected
2018-06-04 19:13:50,275 WARNING  tuned.plugins.base: instance audio: no matching devices available
2018-06-04 19:13:50,275 INFO     tuned.plugins.base: instance video: assigning devices card0
2018-06-04 19:13:50,276 INFO     tuned.plugins.base: instance disk: assigning devices dm-3, sdd, sde, sda, sdb, sdc, md0, dm-5, dm-2, dm-4, dm-0, dm-1
2018-06-04 19:13:50,276 INFO     tuned.plugins.base: instance scsi_host: assigning devices host6, host5, host4, host3, host2, host1, host0
2018-06-04 19:13:50,276 WARNING  tuned.plugins.base: instance disk_sdi: no matching devices available
2018-06-04 19:13:50,276 WARNING  tuned.plugins.base: instance disk_sdd: no matching devices available
2018-06-04 19:13:50,277 WARNING  tuned.plugins.base: instance disk_sde: no matching devices available
2018-06-04 19:13:50,277 WARNING  tuned.plugins.base: instance disk_sdf: no matching devices available
2018-06-04 19:13:50,277 WARNING  tuned.plugins.base: instance disk_sdg: no matching devices available
2018-06-04 19:13:50,277 WARNING  tuned.plugins.base: instance disk_sda: no matching devices available
2018-06-04 19:13:50,278 WARNING  tuned.plugins.base: instance disk_sdb: no matching devices available
2018-06-04 19:13:50,278 WARNING  tuned.plugins.base: instance disk_sdc: no matching devices available
2018-06-04 19:13:50,278 WARNING  tuned.plugins.base: instance disk_sdh: no matching devices available
2018-06-04 19:13:50,279 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu6', it is not supported
2018-06-04 19:13:50,279 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu7', it is not supported
2018-06-04 19:13:50,279 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu4', it is not supported
2018-06-04 19:13:50,279 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu5', it is not supported
2018-06-04 19:13:50,279 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported
2018-06-04 19:13:50,280 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu3', it is not supported
2018-06-04 19:13:50,280 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu0', it is not supported
2018-06-04 19:13:50,280 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu1', it is not supported
2018-06-04 19:13:50,284 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu6'
2018-06-04 19:13:50,287 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu6, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,289 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu7'
2018-06-04 19:13:50,292 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu7, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,296 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu4'
2018-06-04 19:13:50,299 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu4, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,304 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu5'
2018-06-04 19:13:50,308 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu5, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,312 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu2'
2018-06-04 19:13:50,315 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu2, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,318 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu3'
2018-06-04 19:13:50,321 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu3, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,324 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu0'
2018-06-04 19:13:50,327 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu0, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,330 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu1'
2018-06-04 19:13:50,333 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu1, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-04 19:13:50,333 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 1000
2018-06-04 19:13:50,334 ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card0/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card0/device/power_method''
2018-06-04 19:13:50,334 WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0'
2018-06-04 19:13:50,424 INFO     tuned.plugins.plugin_scsi_host: ALPM control file ('/sys/class/scsi_host/host6/link_power_management_policy') not found, skipping ALPM setting for 'host6'
2018-06-04 19:13:50,426 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2018-06-04 19:13:50,545 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2018-06-04 19:13:50,549 INFO     tuned.daemon.daemon: static tuning from profile 'custom-balanced' applied

Based on the config, /dev/sda should have the 'none' elevator

# cat /sys/block/sda/queue/scheduler
noop deadline [cfq]
olysonek commented 6 years ago

The output of the script is as follows:

$ python test.py Device type: disk Removable: 0 Subsystem: nvme

Sorry for the late reply. You should be able to make it work by applying this patch:

diff --git a/tuned/plugins/plugin_disk.py b/tuned/plugins/plugin_disk.py
index 1de1257..6283da0 100644
--- a/tuned/plugins/plugin_disk.py
+++ b/tuned/plugins/plugin_disk.py
@@ -41,7 +41,8 @@ class DiskPlugin(hotplug.Plugin):
        return  device.device_type == "disk" and \
            device.attributes.get("removable", None) == "0" and \
            (device.parent is None or \
-                   device.parent.subsystem in ["scsi", "virtio", "xen"])
+                   device.parent.subsystem in \
+                   ["scsi", "virtio", "xen", "nvme"])

    def _hardware_events_init(self):
        self._hardware_inventory.subscribe(self, "block", self._hardware_events_callback)

But before merging this upstream, we'll have to check that the disk plugin can work with NVMe devices.

olysonek commented 6 years ago

Based on the config, /dev/sda should have the 'none' elevator

What I said before still holds. You need to override the disk plugin instance from the balanced profile that you're including. From the log i can see that the instance from the balanced profile took over the device:

2018-06-04 19:13:50,276 INFO tuned.plugins.base: instance disk: assigning devices dm-3, sdd, sde, sda, sdb, sdc, md0, dm-5, dm-2, dm-4, dm-0, dm-1

Add e.g. the following to your profile:

[disk]
devices=!sda,!sdb,!sdc,!sdb,!sde
kees-closed commented 6 years ago

It works!

2018-06-06 12:24:51,486 INFO     tuned.daemon.daemon: stopping tuning
2018-06-06 12:24:51,819 INFO     tuned.plugins.plugin_bootloader: removing grub2 tuning previously added by Tuned
2018-06-06 12:24:51,823 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu2'
2018-06-06 12:24:51,823 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu3'
2018-06-06 12:24:51,824 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu0'
2018-06-06 12:24:51,825 INFO     tuned.plugins.plugin_cpu: setting governor 'powersave' on cpu 'cpu1'
2018-06-06 12:24:51,827 INFO     tuned.profiles.loader: loading profile: custom-balanced
2018-06-06 12:24:51,832 INFO     tuned.daemon.daemon: starting tuning
2018-06-06 12:24:51,854 INFO     tuned.plugins.base: instance cpu: assigning devices cpu2, cpu3, cpu0, cpu1
2018-06-06 12:24:51,864 INFO     tuned.plugins.plugin_cpu: intel_pstate detected
2018-06-06 12:24:51,866 INFO     tuned.plugins.base: instance audio: assigning devices snd_hda_intel
2018-06-06 12:24:51,867 INFO     tuned.plugins.base: instance video: assigning devices card0
2018-06-06 12:24:51,868 WARNING  tuned.plugins.base: instance disk: no matching devices available
2018-06-06 12:24:51,869 WARNING  tuned.plugins.base: instance scsi_host: no matching devices available
2018-06-06 12:24:51,870 INFO     tuned.plugins.base: instance disk_nvme0n1 (disk): assigning devices nvme0n1
2018-06-06 12:24:51,872 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu2', it is not supported
2018-06-06 12:24:51,873 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu3', it is not supported
2018-06-06 12:24:51,873 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu0', it is not supported
2018-06-06 12:24:51,874 INFO     tuned.plugins.plugin_cpu: ignoring governor 'conservative' on cpu 'cpu1', it is not supported
2018-06-06 12:24:51,884 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu2'
2018-06-06 12:24:51,894 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu2, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-06 12:24:51,905 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu3'
2018-06-06 12:24:51,916 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu3, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-06 12:24:51,926 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu0'
2018-06-06 12:24:51,934 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu0, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-06 12:24:51,944 INFO     tuned.plugins.plugin_cpu: setting energy_perf_bias 'normal' on cpu 'cpu1'
2018-06-06 12:24:51,954 ERROR    tuned.utils.commands: Executing x86_energy_perf_policy error: x86_energy_perf_policy: pwrite(cpu1, offset 0x1b0, 0x6) = -1: Operation not permitted
2018-06-06 12:24:51,955 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 100
2018-06-06 12:24:51,957 ERROR    tuned.utils.commands: Error when reading file '/sys/class/drm/card0/device/power_method': '[Errno 2] No such file or directory: u'/sys/class/drm/card0/device/power_method''                                                                                                                                                               
2018-06-06 12:24:51,958 WARNING  tuned.plugins.plugin_video: radeon_powersave is not supported on 'card0'
2018-06-06 12:24:51,958 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
2018-06-06 12:24:52,136 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2018-06-06 12:24:52,159 INFO     tuned.daemon.daemon: static tuning from profile 'custom-balanced' applied

And yes, I forgot about excluding those disks again. Just for those who might stumble upon this thread, the Ansible template looks like this to include the exclusion of the devices in the disk section.

[main]
summary={{ item.summary }}
include={{ item.include }}

[bootloader]
cmdline=pcie_aspm=force

[sysctl]
net.ipv4.tcp_rfc1337 = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

[disk]
devices=!{{ ansible_devices.keys() | join(',!') }}

{% for item in query('dict', ansible_devices) if not item.key.startswith('dm-') and not item.key.startswith('md') %}
{% if item.value.support_discard is defined and item.value.support_discard != "0" %}
[disk_{{ item.key }}]
type=disk
devices={{ item.key }}
elevator=none

{% else %}
[disk_{{ item.key }}]
type=disk
devices={{ item.key }}
elevator=cfq

{% endif %}
{% endfor %}
ouafnico commented 4 years ago

Hi,

I'm getting the same problem, using tuned 2.10 (or 2.13 from sources) on Debian 9.

It's using a virtual machine with virtio.

The test.py gets (for vda):

root@EVO:~/tuned-2.13.0# python test.py 
Device type: disk
Removable: 0
Subsystem: virtio

But tuned still says:

2020-02-26 10:23:18,914 WARNING tuned.plugins.base: instance disk: no matching devices available 2020-02-26 10:23:18,914 WARNING tuned.plugins.base: instance disk-all: no matching devices available

olysonek commented 4 years ago

Hi,

I think virtio should work. What profile are you using?

ouafnico commented 4 years ago

I'm using virtual-guest.

2020-02-26 11:46:22,939 INFO     tuned.profiles.loader: loading profile: virtual-guest
2020-02-26 11:46:22,940 INFO     tuned.daemon.daemon: starting tuning
2020-02-26 11:46:22,942 INFO     tuned.plugins.base: instance cpu: assigning devices cpu1, cpu0
2020-02-26 11:46:22,944 WARNING  tuned.plugins.plugin_cpu: unable to run x86_energy_perf_policy tool, ignoring CPU energy performance bias, is the tool installed?
2020-02-26 11:46:22,944 WARNING  tuned.plugins.base: instance disk: no matching devices available
2020-02-26 11:46:22,945 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 1000
2020-02-26 11:46:22,968 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2020-02-26 11:46:22,970 INFO     tuned.daemon.daemon: static tuning from profile 'virtual-guest' applied

The [disk] from "trhoughput-performance" which is including is creating the "2020-02-26 11:46:22,944 WARNING tuned.plugins.base: instance disk: no matching devices available".

olysonek commented 4 years ago

I'm using virtual-guest.

Unmodified?

That's odd. I don't immediatelly see what could be wrong.

If you're up to some debugging, I suggest you use pdb and take a look at what's happenning here: https://github.com/redhat-performance/tuned/blob/9bb73e8f407ae3ce17031e020ab112ac8c8cef45/tuned/plugins/plugin_disk.py#L31-L33

ouafnico commented 4 years ago

Hum yes unmodified.

Hum honestly I don't know how to use pdb :p

olysonek commented 4 years ago

@ouafnico Can you please provide the output of this script?

import pyudev

ctx = pyudev.Context()
for i in ctx.list_devices(subsystem="block"):
    print('%s %s' % (i, i.sys_name))
ouafnico commented 4 years ago

Sure :)

# python debug.py 
Device(u'/sys/devices/pci0000:00/0000:00:02.3/0000:04:00.0/virtio2/block/vda') vda
Device(u'/sys/devices/pci0000:00/0000:00:02.3/0000:04:00.0/virtio2/block/vda/vda1') vda1
olysonek commented 4 years ago

Are you running this with the same python version as Tuned uses on Debian?

ouafnico commented 4 years ago

It's tested with the default python 2.7.13 from debian stretch. If I run it with the same as tuned, 3.5.3, I got this:

# python3 debug.py 
Device('/sys/devices/pci0000:00/0000:00:02.3/0000:04:00.0/virtio2/block/vda') vda
Device('/sys/devices/pci0000:00/0000:00:02.3/0000:04:00.0/virtio2/block/vda/vda1') vda1
olysonek commented 4 years ago

OK, that's pretty much the same output. What about the output of the previous script? Is there a difference between running it with python 2 and 3?

If not, I'm out of ideas.

ouafnico commented 4 years ago

Hum :

# python test.py
Device type: disk
Removable: 0
Subsystem: virtio
# python3 test.py
Device type: disk
Removable: b'0'
Subsystem: virtio

I'm out of ideas too...

olysonek commented 4 years ago

I remembered that we've fixed this kind of problem in the past: https://github.com/redhat-performance/tuned/commit/2cc3d747986837d7e

That commit is not contained in tuned 2.10, so that could explain why it doesn't work for you with the Debian's Tuned version.

It doesn't explain though why it doesn't work for you with 2.13. But perhaps, due to some mistake, you're not really running the 2.13 version, but the Debian's version?

Can you try applying that patch to the Debian package? The file you need to patch could be somewhere like /usr/lib/python*/site-packages/tuned/plugins/plugin_disk.py.

ouafnico commented 4 years ago

I have this commit on 2.13. I'm running the good version on debian, but same problem.

I didn't have the original 2.10 package on my debian (only using the sources for 2.13).

ouafnico commented 4 years ago

Do you have any idea what I can try?

olysonek commented 4 years ago

Other than debugging, no. I haven't thought of anything else. Sorry.

ouafnico commented 4 years ago

I tried on Debian buster. I got the same issue :/