Open yuqin11 opened 4 years ago
It's not related to HDD. Are you running on docker or some other container manager?
Yes on docker .
anyone can have a look at this problem?
@yuqin11 can you please provide the info on how you run the scylla docker (or the docker you have built to run scylla).
I expect that if you are using something like https://docs.docker.com/engine/security/seccomp/ it may block from scylla some capabilities (like mbind).
i get docker images from docker hub, and image version is 2.3.1; and i run scylla with command mentitioned above:
docker run -it -d --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address ip
/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --num-io-queues=1 --io-properties-file=/etc/scylla.d/io_properties.yaml --memory 8G --smp 4 --listen-address 0.0.0.0 --rpc-address 0.0.0.0 --seed-provider-parameters seeds=172.16.16.30 --broadcast-address 172.16.16.30 --broadcast-rpc-address 172.16.16.30 --blocked-reactor-notify-ms 999999999
linux system version: centos:7.2 disk:HDD(XFS)
thanks
@ycui1984
can you please try the following
docker run -it -d --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address ip \
--security-opt seccomp=unconfined
and report if this solved the issue - attempting to workaround
I expect that if you are using something like https://docs.docker.com/engine/security/seccomp/ it may block from scylla some capabilities (like mbind).
docker-entrypoint.py: error: unrecognized arguments: --security-opt seccomp=unconfined
failed!
Traceback (most recent call last):
File "/docker-entrypoint.py", line 11, in
thanks
@ycui1984
can you please try the following
docker run -it -d --name=scylla \ -v /data/home/scylla/data:/var/lib/scylla/data \ -v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \ --net=host \ scylladb/scylla:2.3.1 \ --overprovisioned 1 \ --seeds ip \ --listen-address 0.0.0.0 \ --developer-mode 1 \ --smp 4 --memory 8G \ --broadcast-address ip \ --broadcast-rpc-address ip \ --security-opt seccomp=unconfined
and report if this solved the issue - attempting to workaround
I expect that if you are using something like https://docs.docker.com/engine/security/seccomp/ it may block from scylla some capabilities (like mbind).
docker-entrypoint.py: error: unrecognized arguments: --security-opt seccomp=unconfined
failed!
Traceback (most recent call last):
File "/docker-entrypoint.py", line 11, in
@yuqin11 my bad I placed the arg in the wrong order can you try
docker run -it -d --security-opt seccomp=unconfined --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address ip
thanks it works, but there still remains some warn:
Connecting to http://localhost:10000
Starting the JMX server
tuning /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/vdb1
tuning /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb
tuning: /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/scheduler noop
warning: unable to tune /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/scheduler to noop
tuning: /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges 2
warning: unable to tune /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges to 2
tuning /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/vdb1
WARN 2020-05-27 06:59:06,047 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
Scylla version 2.3.1-0.20181021.336c77166 starting ...
@syuu1228 / @penberg we need to resolve this somehow
It seems we are blocked from issuing some system calls in new docker versions - the workaround (above) was to disable it completely.
The real solution is to enable the capabilities we require "--cap-add=sys_nice" and maybe something else - I'll leave it for you to find the solution.
The other thing we should consider - is to make this dependent also on developer-mode so we should boot and not do some syst (optimization) calls if thats possible.
@tzach - I found a workaround for an issue docker users may be hitting - it may take time to have a proper fix - do we have an FAQ we can add this info
@yuqin11 thanks,
lets try and if we can get rid of one of them
docker run -it -d --security-opt seccomp=unconfined --cap-add=sys_nice --name=scylla \
-v /data/home/scylla/data:/var/lib/scylla/data \
-v /data/home/scylla/commitlog:/var/lib/scylla/commitlog \
--net=host \
scylladb/scylla:2.3.1 \
--overprovisioned 1 \
--seeds ip \
--listen-address 0.0.0.0 \
--developer-mode 1 \
--smp 4 --memory 8G \
--broadcast-address ip \
--broadcast-rpc-address ip
add --privileged=true
will solve warning: unable to tune /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges to 2
but this remains, `
WARN 2020-05-27 06:59:06,047 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
--cap-add=sys_nice` will lead to error.
docker-entrypoint.py: error: unrecognized arguments: --cap-add=sys_nice
failed!
Traceback (most recent call last):
File "/docker-entrypoint.py", line 11, in <module>
arguments = commandlineparser.parse()
File "/commandlineparser.py", line 21, in parse
return parser.parse_args()
File "/usr/lib64/python3.4/argparse.py", line 1731, in parse_args
self.error(msg % ' '.join(argv))
File "/usr/lib64/python3.4/argparse.py", line 2386, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/usr/lib64/python3.4/argparse.py", line 2373, in exit
_sys.exit(status)
SystemExit: 2
docker run -it -d --security-opt seccomp=unconfined --cap-add=sys_nice
can not get rid of
WARN 2020-05-27 06:59:06,047 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
@yuqin11 thanks we will try to have a look at that.
BTW 2.3.1 is EOL the two supported versions are 3.3, 4.0
@slivne any estimations on when it might be resolved? Faced the same problem recently.
I use --privileged
. It's seem Ok. but I think it's not good.
This is Scylla's bug tracker, to be used for reporting bugs only. If you have a question about Scylla, and not a bug, please ask it in our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.
/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --num-io-queues=1 --io-properties-file=/etc/scylla.d/io_properties.yaml --memory 8G --smp 4 --listen-address 0.0.0.0 --rpc-address 0.0.0.0 --seed-provider-parameters seeds=172.16.16.30 --broadcast-address 172.16.16.30 --broadcast-rpc-address 172.16.16.30 --blocked-reactor-notify-ms 999999999
Disks: HDD
this is the start log
`WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer:
WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer:
WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARNING: unable to mbind shard memory; performance may suffer: WARN 2020-03-13 12:34:48,905 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE INFO 2020-03-13 12:34:49,846 [shard 0] iotune - /var/lib/scylla/data passed sanity checks WARN 2020-03-13 12:34:49,847 [shard 0] iotune - nomerges for /sys/devices/pci0000:00/0000:00:05.0/virtio2/block/vdb/queue/nomerges set to 0. It is recommend to set it to 2 before evaluation so that merges are disabled. Results can be skewed otherwise. Starting Evaluation. This may take a while... Measuring sequential write bandwidth: 131 MB/s Measuring sequential read bandwidth: 130 MB/s Measuring random write IOPS: 3372 IOPS Measuring random read IOPS: 3355 IOPS Recommended --num-io-queues: 1 `