unable to mbind shard memory; performance may suffer: is this common on HDD?

[yuqin11]

This is Scylla's bug tracker, to be used for reporting bugs only. If you have a question about Scylla, and not a bug, please ask it in our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.

• [] I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --num-io-queues=1 --io-properties-file=/etc/scylla.d/io_properties.yaml --memory 8G --smp 4 --listen-address 0.0.0.0 --rpc-address 0.0.0.0 --seed-provider-parameters seeds=172.16.16.30 --broadcast-address 172.16.16.30 --broadcast-rpc-address 172.16.16.30 --blocked-reactor-notify-ms 999999999

Disks: HDD

this is the start log

`WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer:

WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer:

WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARN 2020-03-13 12:34:48,905 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE INFO 2020-03-13 12:34:49,846 [shard 0] iotune - /var/lib/scylla/data passed sanity checks WARN 2020-03-13 12:34:49,847 [shard 0] iotune - nomerges for /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges set to 0. It is recommend to set it to 2 before evaluation so that merges are disabled. Results can be skewed otherwise. Starting Evaluation. This may take a while... Measuring sequential write bandwidth: 131 MB/s Measuring sequential read bandwidth: 130 MB/s Measuring random write IOPS: 3372 IOPS Measuring random read IOPS: 3355 IOPS Recommended --num-io-queues: 1 `

[avikivity]

It's not related to HDD. Are you running on docker or some other container manager?

[yuqin11]

Yes on docker .

[yuqin11]

anyone can have a look at this problem?

[slivne]

@yuqin11 can you please provide the info on how you run the scylla docker (or the docker you have built to run scylla).

I expect that if you are using something like https://docs.docker.com/engine/security/seccomp/ it may block from scylla some capabilities (like mbind).

[yuqin11]

i get docker images from docker hub, and image version is 2.3.1; and i run scylla with command mentitioned above:

docker run -it -d --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address  ip

/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --num-io-queues=1 --io-properties-file=/etc/scylla.d/io_properties.yaml --memory 8G --smp 4 --listen-address 0.0.0.0 --rpc-address 0.0.0.0 --seed-provider-parameters seeds=172.16.16.30 --broadcast-address 172.16.16.30 --broadcast-rpc-address 172.16.16.30 --blocked-reactor-notify-ms 999999999

linux system version: centos:7.2 disk:HDD(XFS)

[slivne]

thanks

@ycui1984

can you please try the following

docker run -it -d --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address  ip \
--security-opt seccomp=unconfined

and report if this solved the issue - attempting to workaround

I expect that if you are using something like https://docs.docker.com/engine/security/seccomp/ it may block from scylla some capabilities (like mbind).

[yuqin11]

docker-entrypoint.py: error: unrecognized arguments: --security-opt seccomp=unconfined failed! Traceback (most recent call last): File "/docker-entrypoint.py", line 11, in arguments = commandlineparser.parse() File "/commandlineparser.py", line 21, in parse return parser.parse_args() File "/usr/lib64/python3.4/argparse.py", line 1731, in parseargs self.error(msg % ' '.join(argv)) File "/usr/lib64/python3.4/argparse.py", line 2386, in error self.exit(2, ('%(prog)s: error: %(message)s\n') % args) File "/usr/lib64/python3.4/argparse.py", line 2373, in exit _sys.exit(status) SystemExit: 2

[yuqin11]

thanks

@ycui1984

can you please try the following

docker run -it -d --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address  ip \
--security-opt seccomp=unconfined

and report if this solved the issue - attempting to workaround

I expect that if you are using something like https://docs.docker.com/engine/security/seccomp/ it may block from scylla some capabilities (like mbind).

docker-entrypoint.py: error: unrecognized arguments: --security-opt seccomp=unconfined failed! Traceback (most recent call last): File "/docker-entrypoint.py", line 11, in arguments = commandlineparser.parse() File "/commandlineparser.py", line 21, in parse return parser.parse_args() File "/usr/lib64/python3.4/argparse.py", line 1731, in parseargs self.error(msg % ' '.join(argv)) File "/usr/lib64/python3.4/argparse.py", line 2386, in error self.exit(2, ('%(prog)s: error: %(message)s\n') % args) File "/usr/lib64/python3.4/argparse.py", line 2373, in exit _sys.exit(status) SystemExit: 2

[slivne]

@yuqin11 my bad I placed the arg in the wrong order can you try

docker run -it -d --security-opt seccomp=unconfined --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address  ip 

[yuqin11]

thanks it works, but there still remains some warn:

Connecting to http://localhost:10000
Starting the JMX server
tuning /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/vdb1
tuning /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb
tuning: /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/scheduler noop
warning: unable to tune /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/scheduler to noop

tuning: /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges 2
warning: unable to tune /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges to 2

tuning /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/vdb1
WARN  2020-05-27 06:59:06,047 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
Scylla version 2.3.1-0.20181021.336c77166 starting ...

[slivne]

@syuu1228 / @penberg we need to resolve this somehow

It seems we are blocked from issuing some system calls in new docker versions - the workaround (above) was to disable it completely.

The real solution is to enable the capabilities we require "--cap-add=sys_nice" and maybe something else - I'll leave it for you to find the solution.

The other thing we should consider - is to make this dependent also on developer-mode so we should boot and not do some syst (optimization) calls if thats possible.

[slivne]

@tzach - I found a workaround for an issue docker users may be hitting - it may take time to have a proper fix - do we have an FAQ we can add this info

[slivne]

@yuqin11 thanks,

lets try and if we can get rid of one of them

docker run -it -d --security-opt seccomp=unconfined --cap-add=sys_nice --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address  ip 

[yuqin11]

add --privileged=true will solve warning: unable to tune /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges to 2 but this remains, `

WARN  2020-05-27 06:59:06,047 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE

--cap-add=sys_nice` will lead to error.

docker-entrypoint.py: error: unrecognized arguments: --cap-add=sys_nice
failed!
Traceback (most recent call last):
  File "/docker-entrypoint.py", line 11, in <module>
    arguments = commandlineparser.parse()
  File "/commandlineparser.py", line 21, in parse
    return parser.parse_args()
  File "/usr/lib64/python3.4/argparse.py", line 1731, in parse_args
    self.error(msg % ' '.join(argv))
  File "/usr/lib64/python3.4/argparse.py", line 2386, in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
  File "/usr/lib64/python3.4/argparse.py", line 2373, in exit
    _sys.exit(status)
SystemExit: 2

[yuqin11]

docker run -it -d --security-opt seccomp=unconfined --cap-add=sys_nice can not get rid of

WARN  2020-05-27 06:59:06,047 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE

[slivne]

@yuqin11 thanks we will try to have a look at that.

BTW 2.3.1 is EOL the two supported versions are 3.3, 4.0

[ThelonKarrde]

@slivne any estimations on when it might be resolved? Faced the same problem recently.

[zey1996]

I use --privileged . It's seem Ok. but I think it's not good.