scylladb / gemini

Test data integrity by comparing against an Oracle running in parallel
Apache License 2.0
27 stars 16 forks source link

Could not initialize seastar when starting scylla 5.2.9 #412

Open qiuqiu-007 opened 6 months ago

qiuqiu-007 commented 6 months ago

It occured when I do gemini test at the beginning of starting scylla image with the version of 5.2.9. Msg is Could not initialize seastar: std::system_error (error system:1, Operation not permitted), all detatiled msg is attached in file

qiuqiu-007 commented 6 months ago

It occured when I do gemini test at the beginning of starting scylla image with the version of 5.2.9. Msg is Could not initialize seastar: std::system_error (error system:1, Operation not permitted), all detatiled msg is attached in file

log.txt

mykaul commented 6 months ago

Can you provide more information on how you are trying to run it? OS, kernel, container/VM, versions?

qiuqiu-007 commented 6 months ago

Can you provide more information on how you are trying to run it? OS, kernel, container/VM, versions?

networks: gemini: name: gemini driver: bridge ipam: driver: default config:

services: gemini-oracle: image: scylladb/scylla:5.2.9 container_name: gemini-oracle command: --smp 2 --memory 512M --api-address 0.0.0.0 networks: gemini:

gemini-test: image: scylladb/scylla:5.2.9 container_name: gemini-test command: --smp 2 --memory 512M --api-address 0.0.0.0 networks: gemini: `

mykaul commented 6 months ago

I think we can move the discussion to Gemini repo, no? (I can move the issue, I believe)

mykaul commented 6 months ago
  • Env: CentOS 7

This is a bit of an old environment and kernel (and Docker...)

qiuqiu-007 commented 6 months ago
  • Env: CentOS 7

This is a bit of an old environment and kernel (and Docker...)

Is it the potential cause of this issue ?

qiuqiu-007 commented 6 months ago

I think we can move the discussion to Gemini repo, no? (I can move the issue, I believe)

I agree.

fruch commented 6 months ago

@qiuqiu-007

2023-12-13T01:19:33.687533571Z WARN  2023-12-13 01:19:33,684 seastar - Requested AIO slots too large, please increase request capacity in /proc/sys/fs/aio-max-nr. available:65536 requested:102052
2023-12-13T01:19:33.687550162Z WARN  2023-12-13 01:19:33,684 seastar - max-networking-io-control-blocks adjusted from 50000 to 31742, since AIO slots are unavailable
2023-12-13T01:19:33.687554111Z INFO  2023-12-13 01:19:33,684 seastar - Reactor backend: epoll

this should fix it:

echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.conf
qiuqiu-007 commented 6 months ago

@fruch I tried, but it doesn't work. log2.txt

fruch commented 6 months ago

@fruch I tried, but it doesn't work. log2.txt

forgot the actual update, that would get it updated:

sysctl -p 
qiuqiu-007 commented 6 months ago

@fruch I tried, but it doesn't work. log2.txt

forgot the actual update, that would get it updated:

sysctl -p 

The aio fixed. But others. log3.txt

mykaul commented 6 months ago

I guess the issue is:

2023-12-14T01:15:35.620341245Z [0.013s][warning][os,thread] Failed to start thread "VM Thread" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
2023-12-14T01:15:35.620368712Z Error occurred during initialization of VM

I wonder if it happens with newer kernels.