scylladb / gemini

Test data integrity by comparing against an Oracle running in parallel
Apache License 2.0
31 stars 17 forks source link

Could not initialize seastar when starting scylla 5.2.9 #412

Open qiuqiu-007 opened 10 months ago

qiuqiu-007 commented 10 months ago

It occured when I do gemini test at the beginning of starting scylla image with the version of 5.2.9. Msg is Could not initialize seastar: std::system_error (error system:1, Operation not permitted), all detatiled msg is attached in file

qiuqiu-007 commented 10 months ago

It occured when I do gemini test at the beginning of starting scylla image with the version of 5.2.9. Msg is Could not initialize seastar: std::system_error (error system:1, Operation not permitted), all detatiled msg is attached in file

log.txt

mykaul commented 10 months ago

Can you provide more information on how you are trying to run it? OS, kernel, container/VM, versions?

qiuqiu-007 commented 10 months ago

Can you provide more information on how you are trying to run it? OS, kernel, container/VM, versions?

networks: gemini: name: gemini driver: bridge ipam: driver: default config:

services: gemini-oracle: image: scylladb/scylla:5.2.9 container_name: gemini-oracle command: --smp 2 --memory 512M --api-address 0.0.0.0 networks: gemini:

gemini-test: image: scylladb/scylla:5.2.9 container_name: gemini-test command: --smp 2 --memory 512M --api-address 0.0.0.0 networks: gemini: `

mykaul commented 10 months ago

I think we can move the discussion to Gemini repo, no? (I can move the issue, I believe)

mykaul commented 10 months ago
  • Env: CentOS 7

This is a bit of an old environment and kernel (and Docker...)

qiuqiu-007 commented 10 months ago
  • Env: CentOS 7

This is a bit of an old environment and kernel (and Docker...)

Is it the potential cause of this issue ?

qiuqiu-007 commented 10 months ago

I think we can move the discussion to Gemini repo, no? (I can move the issue, I believe)

I agree.

fruch commented 10 months ago

@qiuqiu-007

2023-12-13T01:19:33.687533571Z WARN  2023-12-13 01:19:33,684 seastar - Requested AIO slots too large, please increase request capacity in /proc/sys/fs/aio-max-nr. available:65536 requested:102052
2023-12-13T01:19:33.687550162Z WARN  2023-12-13 01:19:33,684 seastar - max-networking-io-control-blocks adjusted from 50000 to 31742, since AIO slots are unavailable
2023-12-13T01:19:33.687554111Z INFO  2023-12-13 01:19:33,684 seastar - Reactor backend: epoll

this should fix it:

echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.conf
qiuqiu-007 commented 10 months ago

@fruch I tried, but it doesn't work. log2.txt

fruch commented 10 months ago

@fruch I tried, but it doesn't work. log2.txt

forgot the actual update, that would get it updated:

sysctl -p 
qiuqiu-007 commented 10 months ago

@fruch I tried, but it doesn't work. log2.txt

forgot the actual update, that would get it updated:

sysctl -p 

The aio fixed. But others. log3.txt

mykaul commented 10 months ago

I guess the issue is:

2023-12-14T01:15:35.620341245Z [0.013s][warning][os,thread] Failed to start thread "VM Thread" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
2023-12-14T01:15:35.620368712Z Error occurred during initialization of VM

I wonder if it happens with newer kernels.