wandb / server

W&B Server is the self hosted version of Weights & Biases
MIT License
263 stars 21 forks source link

"panic: dial tcp 127.0.0.1:3306: connect: connection refused" when start server #86

Open konatasick opened 2 years ago

konatasick commented 2 years ago

I met a problem similar to https://github.com/wandb/local/issues/7, but I cannot solve it by the method it mentions.

When I run

wandb server start --port 80

I get

wandb: W&B server started at http://localhost:80 🚀
wandb: You can stop the server by running `wandb server stop`

But I cannot open http://localhost:80 or http://my_machine_ip:80

So I try:

docker run --rm -v wandb:/vol -p 80:80 --user root --name wandb-local wandb/local

However I get:

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 58
*** Starting wandb servers...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc000470e30, {0xc000115460, 0x2, 0x2})
        /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:71 +0x974
main.main()
        /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:56 +0x234
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc00053ade0, {0xc0000a5060, 0x2, 0x2})
        /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:71 +0x974
main.main()
        /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:56 +0x234

I also try

docker volume rm wandb

But it doesn't work.

MBakirWB commented 2 years ago

Hi @konatasick if you have no data to worry about, delete the image and volume first then retry

docker stop wandb-local
docker images -a
docker rmi <wandb-Image-id>
docker volume rm wandb

wandb server start --port 80
konatasick commented 2 years ago

Hi @konatasick if you have no data to worry about, delete the image and volume first then retry

docker stop wandb-local
docker images -a
docker rmi <wandb-Image-id>
docker volume rm wandb

wandb server start --port 80

Thanks for replying.

I tried your method but the problem still exists. After delete the image and volume, when I run:

 wandb server start --port 80

I get

Command '['docker', 'inspect', 'wandb/local', '--format', '{{json .RepoDigests}}']' returned non-zero exit status 1.
Unable to find image 'wandb/local:latest' locally
latest: Pulling from wandb/local
7c3b88808835: Pulling fs layer
6ea0e81b1046: Pulling fs layer
2cdac9582559: Pulling fs layer
f1e440ae6cde: Pulling fs layer
58fb1f5d328a: Pulling fs layer
4f4fb700ef54: Pulling fs layer
f329bf250c1d: Pulling fs layer
f89d89613930: Pulling fs layer
f7213bf600c3: Pulling fs layer
4f2756b837b4: Pulling fs layer
95903d54e8a9: Pulling fs layer
f1e440ae6cde: Waiting
70ce4b92d47f: Pulling fs layer
f329bf250c1d: Waiting
6ab57cf9f4fe: Pulling fs layer
f7213bf600c3: Waiting
f89d89613930: Waiting
58fb1f5d328a: Waiting
c54b9f2f48ce: Pulling fs layer
4f58f2abe4ab: Pulling fs layer
4f2756b837b4: Waiting
e3f3570b7e3b: Pulling fs layer
4f4fb700ef54: Waiting
95903d54e8a9: Waiting
bab9d0a04a09: Pulling fs layer
70ce4b92d47f: Waiting
2ce5be258414: Pulling fs layer
c54b9f2f48ce: Waiting
779f1411faaa: Pulling fs layer
e3f3570b7e3b: Waiting
b5537def9dc3: Pulling fs layer
bab9d0a04a09: Waiting
2ce5be258414: Waiting
95d6108305b0: Pulling fs layer
b5537def9dc3: Waiting
95d6108305b0: Waiting
6ea0e81b1046: Verifying Checksum
6ea0e81b1046: Download complete
7c3b88808835: Download complete
f1e440ae6cde: Verifying Checksum
7c3b88808835: Pull complete
2cdac9582559: Verifying Checksum
2cdac9582559: Download complete
6ea0e81b1046: Pull complete
58fb1f5d328a: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
f89d89613930: Verifying Checksum
f89d89613930: Download complete
2cdac9582559: Pull complete
f1e440ae6cde: Pull complete
58fb1f5d328a: Pull complete
4f4fb700ef54: Pull complete
f329bf250c1d: Verifying Checksum
f329bf250c1d: Download complete
f329bf250c1d: Pull complete
f89d89613930: Pull complete
f7213bf600c3: Verifying Checksum
f7213bf600c3: Download complete
4f2756b837b4: Verifying Checksum
4f2756b837b4: Download complete
95903d54e8a9: Verifying Checksum
95903d54e8a9: Download complete
6ab57cf9f4fe: Verifying Checksum
6ab57cf9f4fe: Download complete
c54b9f2f48ce: Verifying Checksum
c54b9f2f48ce: Download complete
f7213bf600c3: Pull complete
4f2756b837b4: Pull complete
e3f3570b7e3b: Verifying Checksum
e3f3570b7e3b: Download complete
95903d54e8a9: Pull complete
bab9d0a04a09: Download complete
70ce4b92d47f: Verifying Checksum
70ce4b92d47f: Download complete
2ce5be258414: Verifying Checksum
2ce5be258414: Download complete
b5537def9dc3: Download complete
70ce4b92d47f: Pull complete
6ab57cf9f4fe: Pull complete
c54b9f2f48ce: Pull complete
95d6108305b0: Verifying Checksum
95d6108305b0: Download complete
779f1411faaa: Download complete
4f58f2abe4ab: Verifying Checksum
4f58f2abe4ab: Download complete
4f58f2abe4ab: Pull complete
e3f3570b7e3b: Pull complete
bab9d0a04a09: Pull complete
2ce5be258414: Pull complete
779f1411faaa: Pull complete
b5537def9dc3: Pull complete
95d6108305b0: Pull complete
Digest: sha256:5f9152da5be073c4bb02c9f759c6759cab06372cf5543383d4360b47d67215db
Status: Downloaded newer image for wandb/local:latest
wandb: W&B server started at http://localhost:80 🚀
wandb: You can stop the server by running `wandb server stop`

And I cannot open the website.

So I delete them again and try:

$ docker run --rm -v wandb:/vol -p 80:80 --user root --name wandb-local wandb/local

I get

Unable to find image 'wandb/local:latest' locally
latest: Pulling from wandb/local
7c3b88808835: Pull complete 
6ea0e81b1046: Pull complete 
2cdac9582559: Pull complete 
f1e440ae6cde: Pull complete 
58fb1f5d328a: Pull complete 
4f4fb700ef54: Pull complete 
f329bf250c1d: Pull complete 
f89d89613930: Pull complete 
f7213bf600c3: Pull complete 
4f2756b837b4: Pull complete 
95903d54e8a9: Pull complete 
70ce4b92d47f: Pull complete 
6ab57cf9f4fe: Pull complete 
c54b9f2f48ce: Pull complete 
4f58f2abe4ab: Pull complete 
e3f3570b7e3b: Pull complete 
bab9d0a04a09: Pull complete 
2ce5be258414: Pull complete 
779f1411faaa: Pull complete 
b5537def9dc3: Pull complete 
95d6108305b0: Pull complete 
Digest: sha256:5f9152da5be073c4bb02c9f759c6759cab06372cf5543383d4360b47d67215db
Status: Downloaded newer image for wandb/local:latest
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/08/23 02:51:38 Created default user
2022/08/23 02:51:38 Generating new session key for auth...
2022/08/23 02:51:38 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 50
*** Setting up mysql database...
*** Starting wandb servers...
*** Configuring minio...
Bucket created successfully `local/local-files`.
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc00020ce70, {0xc000114aa0, 0x2, 0x2})
        /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:71 +0x974
main.main()
        /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:56 +0x234
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

BTW, when I stop the container by

docker stop wandb-local

I always get

Error response from daemon: No such container: wandb-local

I also try to init the container and start it manually but always fail.

Seems the container cannot start because the connection refused error.

vanpelt commented 2 years ago

MySQL is unable to start. You can find logs for MySQL at /var/log/mysql.log which should tell us why it's not able to start.

GFiz commented 2 years ago

Experiencing the same issue

vanpelt commented 2 years ago

@GFiz we just released a new version of the server that should address this problem. Can you try upgrading and report back?

llan-ml commented 2 years ago

Hi @vanpelt , I'm running docker in WSL2, and the issue still persists.

llan-ml commented 2 years ago

@vanpelt The /var/log/mysql.log is as follows:

2022-11-03T09:15:17.201476Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:15:17.203208Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:15:17.203220Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:15:17.213164Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:15:17 0x7f83b7643740  InnoDB: Assertion failure in thread 140203694241600 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:15:17 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x55de67a4f71b]
mysqld(handle_fatal_signal+0x377)[0x55de672da907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f83b768a520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f83b76dea7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f83b768a476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f83b76707f3]
mysqld(+0x6b0adc)[0x55de672b0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55de67dfe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55de67cecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55de67da5b4a]
mysqld(+0x106e9ad)[0x55de67c6e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55de6732f5af]
mysqld(+0xc20ee5)[0x55de67820ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55de67824405]
mysqld(+0x6d4960)[0x55de672d4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55de672d602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f83b7671d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f83b7671e40]
mysqld(_start+0x2a)[0x55de672ca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-11-03T09:15:17.471965Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:15:17.472679Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 138 ...
2022-11-03T09:15:17.473715Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:15:17.473721Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:15:17.474500Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-11-03T09:15:17.474513Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-11-03T09:15:17.474517Z 0 [Note] InnoDB: Uses event mutexes
2022-11-03T09:15:17.474519Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-11-03T09:15:17.474522Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-11-03T09:15:17.474524Z 0 [Note] InnoDB: Using Linux native AIO
2022-11-03T09:15:17.474776Z 0 [Note] InnoDB: Number of pools: 1
2022-11-03T09:15:17.474846Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-11-03T09:15:17.475780Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1,
chunk size = 128M
2022-11-03T09:15:17.483818Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-11-03T09:15:17.484887Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:15:17 0x7f609bd74740  InnoDB: Assertion failure in thread 140052908164928 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:15:17 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x557d8864f71b]
mysqld(handle_fatal_signal+0x377)[0x557d87eda907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f609bdbb520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f609be0fa7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f609bdbb476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f609bda17f3]
mysqld(+0x6b0adc)[0x557d87eb0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x557d889fe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x557d888ecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x557d889a5b4a]
mysqld(+0x106e9ad)[0x557d8886e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x557d87f2f5af]
mysqld(+0xc20ee5)[0x557d88420ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x557d88424405]
mysqld(+0x6d4960)[0x557d87ed4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x557d87ed602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f609bda2d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f609bda2e40]
mysqld(_start+0x2a)[0x557d87eca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
./run: line 59: fg: job has terminated
2022-11-03T09:15:48.497455Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:15:48.498560Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:15:48.498568Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:15:48.508211Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:15:48 0x7efd9cf0a740  InnoDB: Assertion failure in thread 139627724842816 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:15:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x55ef9364f71b]
mysqld(handle_fatal_signal+0x377)[0x55ef92eda907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7efd9cf51520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7efd9cfa5a7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7efd9cf51476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7efd9cf377f3]
mysqld(+0x6b0adc)[0x55ef92eb0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55ef939fe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55ef938ecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55ef939a5b4a]
mysqld(+0x106e9ad)[0x55ef9386e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55ef92f2f5af]
mysqld(+0xc20ee5)[0x55ef93420ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55ef93424405]
mysqld(+0x6d4960)[0x55ef92ed4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55ef92ed602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7efd9cf38d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7efd9cf38e40]
mysqld(_start+0x2a)[0x55ef92eca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-11-03T09:15:48.821080Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:15:48.821765Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 505 ...
2022-11-03T09:15:48.822830Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:15:48.822835Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:15:48.823812Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-11-03T09:15:48.823837Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-11-03T09:15:48.823843Z 0 [Note] InnoDB: Uses event mutexes
2022-11-03T09:15:48.823846Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-11-03T09:15:48.823849Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-11-03T09:15:48.823852Z 0 [Note] InnoDB: Using Linux native AIO
2022-11-03T09:15:48.824507Z 0 [Note] InnoDB: Number of pools: 1
2022-11-03T09:15:48.824574Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-11-03T09:15:48.825679Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1,
chunk size = 128M
2022-11-03T09:15:48.832856Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-11-03T09:15:48.834273Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:15:48 0x7fcb2eb43740  InnoDB: Assertion failure in thread 140510638651200 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:15:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x55899b44f71b]
mysqld(handle_fatal_signal+0x377)[0x55899acda907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fcb2eb8a520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fcb2ebdea7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fcb2eb8a476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fcb2eb707f3]
mysqld(+0x6b0adc)[0x55899acb0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55899b7fe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55899b6ecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55899b7a5b4a]
mysqld(+0x106e9ad)[0x55899b66e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55899ad2f5af]
mysqld(+0xc20ee5)[0x55899b220ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55899b224405]
mysqld(+0x6d4960)[0x55899acd4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55899acd602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fcb2eb71d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fcb2eb71e40]
mysqld(_start+0x2a)[0x55899acca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
./run: line 59: fg: job has terminated
2022-11-03T09:16:19.885123Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:16:19.886232Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:16:19.886239Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:16:19.895394Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:16:19 0x7f67dc65f740  InnoDB: Assertion failure in thread 140084056028992 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:16:19 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x55948ce4f71b]
mysqld(handle_fatal_signal+0x377)[0x55948c6da907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f67dc6a6520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f67dc6faa7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f67dc6a6476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f67dc68c7f3]
mysqld(+0x6b0adc)[0x55948c6b0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55948d1fe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55948d0ecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55948d1a5b4a]
mysqld(+0x106e9ad)[0x55948d06e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55948c72f5af]
mysqld(+0xc20ee5)[0x55948cc20ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55948cc24405]
mysqld(+0x6d4960)[0x55948c6d4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55948c6d602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f67dc68dd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f67dc68de40]
mysqld(_start+0x2a)[0x55948c6ca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-11-03T09:16:20.152181Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:16:20.153556Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 827 ...
2022-11-03T09:16:20.154765Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:16:20.154773Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:16:20.155773Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-11-03T09:16:20.155790Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-11-03T09:16:20.155793Z 0 [Note] InnoDB: Uses event mutexes
2022-11-03T09:16:20.155796Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-11-03T09:16:20.155799Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-11-03T09:16:20.155802Z 0 [Note] InnoDB: Using Linux native AIO
2022-11-03T09:16:20.156051Z 0 [Note] InnoDB: Number of pools: 1
2022-11-03T09:16:20.156116Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-11-03T09:16:20.157275Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1,
chunk size = 128M
2022-11-03T09:16:20.164192Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-11-03T09:16:20.165265Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:16:20 0x7fd00b8ee740  InnoDB: Assertion failure in thread 140531523839808 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:16:20 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x5616c084f71b]
mysqld(handle_fatal_signal+0x377)[0x5616c00da907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fd00b935520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fd00b989a7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fd00b935476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fd00b91b7f3]
mysqld(+0x6b0adc)[0x5616c00b0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x5616c0bfe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x5616c0aecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x5616c0ba5b4a]
mysqld(+0x106e9ad)[0x5616c0a6e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x5616c012f5af]
mysqld(+0xc20ee5)[0x5616c0620ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x5616c0624405]
mysqld(+0x6d4960)[0x5616c00d4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x5616c00d602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fd00b91cd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fd00b91ce40]
mysqld(_start+0x2a)[0x5616c00ca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
./run: line 59: fg: job has terminated
2022-11-03T09:16:51.211032Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:16:51.212185Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:16:51.212192Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:16:51.221826Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:16:51 0x7f9360d5f740  InnoDB: Assertion failure in thread 140270961555264 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:16:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x564d9e24f71b]
mysqld(handle_fatal_signal+0x377)[0x564d9dada907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f9360da6520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f9360dfaa7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f9360da6476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f9360d8c7f3]
mysqld(+0x6b0adc)[0x564d9dab0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x564d9e5fe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x564d9e4ecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x564d9e5a5b4a]
mysqld(+0x106e9ad)[0x564d9e46e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x564d9db2f5af]
mysqld(+0xc20ee5)[0x564d9e020ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x564d9e024405]
mysqld(+0x6d4960)[0x564d9dad4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x564d9dad602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f9360d8dd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f9360d8de40]
mysqld(_start+0x2a)[0x564d9daca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-11-03T09:16:51.492200Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-11-03T09:16:51.493213Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 1149 ...
2022-11-03T09:16:51.494242Z 0 [ERROR] Failed to create timer notify thread (errno= 1).
2022-11-03T09:16:51.494248Z 0 [ERROR] Failed to initialize timer component (errno 1).
2022-11-03T09:16:51.495060Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-11-03T09:16:51.495070Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-11-03T09:16:51.495074Z 0 [Note] InnoDB: Uses event mutexes
2022-11-03T09:16:51.495077Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-11-03T09:16:51.495080Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2022-11-03T09:16:51.495083Z 0 [Note] InnoDB: Using Linux native AIO
2022-11-03T09:16:51.495318Z 0 [Note] InnoDB: Number of pools: 1
2022-11-03T09:16:51.495384Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-11-03T09:16:51.496539Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1,
chunk size = 128M
2022-11-03T09:16:51.503655Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-11-03T09:16:51.505079Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1
2022-11-03 09:16:51 0x7f37e969a740  InnoDB: Assertion failure in thread 139878115944256 in file ut0ut.cc
line 921
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:16:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x3b)[0x5597d604f71b]
mysqld(handle_fatal_signal+0x377)[0x5597d58da907]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f37e96e1520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f37e9735a7c]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f37e96e1476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f37e96c77f3]
mysqld(+0x6b0adc)[0x5597d58b0adc]
mysqld(_ZN2ib5fatalD1Ev+0x66)[0x5597d63fe406]
mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x5597d62ecc57]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x5597d63a5b4a]
mysqld(+0x106e9ad)[0x5597d626e9ad]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x5597d592f5af]
mysqld(+0xc20ee5)[0x5597d5e20ee5]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x5597d5e24405]
mysqld(+0x6d4960)[0x5597d58d4960]
mysqld(_Z11mysqld_mainiPPc+0x79e)[0x5597d58d602e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f37e96c8d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f37e96c8e40]
mysqld(_start+0x2a)[0x5597d58ca5ea]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
vanpelt commented 2 years ago

@llan-ml it looks like WSL2 doesn't play nicely with our internal mysql. You can run mysql directly on your machine or in a separate container. Instructions here: https://docs.wandb.ai/guides/self-hosted/setup/on-premise-baremetal#mysql-database

GFiz commented 2 years ago

@vanpelt : Still getting the "panic: dial tcp 127.0.0.1:3306: connect: connection refused".
FYI this is the /var/log/mysql.log: `2022-11-22T08:06:13.993266Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-11-22T08:06:13.994440Z 0 [ERROR] Failed to create timer notify thread (errno= 1). 2022-11-22T08:06:13.994447Z 0 [ERROR] Failed to initialize timer component (errno 1). 2022-11-22T08:06:14.002289Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1 2022-11-22 08:06:13 0x7f96c662d740 InnoDB: Assertion failure in thread 140285550188352 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:06:13 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=500 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 mysqld(my_print_stacktrace+0x3b)[0x560efe05571b] mysqld(handle_fatal_signal+0x377)[0x560efd8e0907] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f96c6674520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f96c66c8a7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f96c6674476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f96c665a7f3] mysqld(+0x6b0adc)[0x560efd8b6adc] mysqld(_ZN2ib5fatalD1Ev+0x66)[0x560efe404406] mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x560efe2f2c57] mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x560efe3abb4a] mysqld(+0x106e9ad)[0x560efe2749ad] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x560efd9355af] mysqld(+0xc20ee5)[0x560efde26ee5] mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x560efde2a405] mysqld(+0x6d4960)[0x560efd8da960] mysqld(_Z11mysqld_mainiPPc+0x79e)[0x560efd8dc02e] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f96c665bd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f96c665be40] mysqld(_start+0x2a)[0x560efd8d05ea] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2022-11-22T08:06:14.260730Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-11-22T08:06:14.261880Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 173 ... 2022-11-22T08:06:14.262975Z 0 [ERROR] Failed to create timer notify thread (errno= 1). 2022-11-22T08:06:14.262981Z 0 [ERROR] Failed to initialize timer component (errno 1). 2022-11-22T08:06:14.263899Z 0 [Note] InnoDB: PUNCH HOLE support available 2022-11-22T08:06:14.263909Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2022-11-22T08:06:14.263912Z 0 [Note] InnoDB: Uses event mutexes 2022-11-22T08:06:14.263915Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2022-11-22T08:06:14.263917Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12 2022-11-22T08:06:14.263920Z 0 [Note] InnoDB: Using Linux native AIO 2022-11-22T08:06:14.264180Z 0 [Note] InnoDB: Number of pools: 1 2022-11-22T08:06:14.264305Z 0 [Note] InnoDB: Using CPU crc32 instructions 2022-11-22T08:06:14.265583Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2022-11-22T08:06:14.270538Z 0 [Note] InnoDB: Completed initialization of buffer pool 2022-11-22T08:06:14.271478Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1 2022-11-22 08:06:14 0x7f2e21c5f740 InnoDB: Assertion failure in thread 139836111845184 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:06:14 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=500 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 mysqld(my_print_stacktrace+0x3b)[0x55bdada8c71b] mysqld(handle_fatal_signal+0x377)[0x55bdad317907] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f2e21ca6520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f2e21cfaa7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f2e21ca6476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f2e21c8c7f3] mysqld(+0x6b0adc)[0x55bdad2edadc] mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55bdade3b406] mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55bdadd29c57] mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55bdadde2b4a] mysqld(+0x106e9ad)[0x55bdadcab9ad] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55bdad36c5af] mysqld(+0xc20ee5)[0x55bdad85dee5] mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55bdad861405] mysqld(+0x6d4960)[0x55bdad311960] mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55bdad31302e] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f2e21c8dd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f2e21c8de40] mysqld(_start+0x2a)[0x55bdad3075ea] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ./run: line 59: fg: job has terminated 2022-11-22T08:06:45.295088Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-11-22T08:06:45.296264Z 0 [ERROR] Failed to create timer notify thread (errno= 1). 2022-11-22T08:06:45.296270Z 0 [ERROR] Failed to initialize timer component (errno 1). 2022-11-22T08:06:45.303701Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1 2022-11-22 08:06:45 0x7f8813f86740 InnoDB: Assertion failure in thread 140222427326272 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:06:45 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=500 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 mysqld(my_print_stacktrace+0x3b)[0x55cb1720c71b] mysqld(handle_fatal_signal+0x377)[0x55cb16a97907] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f8813fcd520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f8814021a7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f8813fcd476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f8813fb37f3] mysqld(+0x6b0adc)[0x55cb16a6dadc] mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55cb175bb406] mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55cb174a9c57] mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55cb17562b4a] mysqld(+0x106e9ad)[0x55cb1742b9ad] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55cb16aec5af] mysqld(+0xc20ee5)[0x55cb16fddee5] mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55cb16fe1405] mysqld(+0x6d4960)[0x55cb16a91960] mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55cb16a9302e] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f8813fb4d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f8813fb4e40] mysqld(_start+0x2a)[0x55cb16a875ea] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2022-11-22T08:06:45.549050Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-11-22T08:06:45.550105Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 752 ... 2022-11-22T08:06:45.551379Z 0 [ERROR] Failed to create timer notify thread (errno= 1). 2022-11-22T08:06:45.551386Z 0 [ERROR] Failed to initialize timer component (errno 1). 2022-11-22T08:06:45.552270Z 0 [Note] InnoDB: PUNCH HOLE support available 2022-11-22T08:06:45.552281Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2022-11-22T08:06:45.552284Z 0 [Note] InnoDB: Uses event mutexes 2022-11-22T08:06:45.552286Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2022-11-22T08:06:45.552288Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12 2022-11-22T08:06:45.552291Z 0 [Note] InnoDB: Using Linux native AIO 2022-11-22T08:06:45.552525Z 0 [Note] InnoDB: Number of pools: 1 2022-11-22T08:06:45.552614Z 0 [Note] InnoDB: Using CPU crc32 instructions 2022-11-22T08:06:45.553738Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2022-11-22T08:06:45.558113Z 0 [Note] InnoDB: Completed initialization of buffer pool 2022-11-22T08:06:45.559018Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1 2022-11-22 08:06:45 0x7f7acb820740 InnoDB: Assertion failure in thread 140165377034048 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:06:45 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=500 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 mysqld(my_print_stacktrace+0x3b)[0x559bc222571b] mysqld(handle_fatal_signal+0x377)[0x559bc1ab0907] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f7acb867520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f7acb8bba7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f7acb867476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f7acb84d7f3] mysqld(+0x6b0adc)[0x559bc1a86adc] mysqld(_ZN2ib5fatalD1Ev+0x66)[0x559bc25d4406] mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x559bc24c2c57] mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x559bc257bb4a] mysqld(+0x106e9ad)[0x559bc24449ad] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x559bc1b055af] mysqld(+0xc20ee5)[0x559bc1ff6ee5] mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x559bc1ffa405] mysqld(+0x6d4960)[0x559bc1aaa960] mysqld(_Z11mysqld_mainiPPc+0x79e)[0x559bc1aac02e] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f7acb84ed90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f7acb84ee40] mysqld(_start+0x2a)[0x559bc1aa05ea] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ./run: line 59: fg: job has terminated 2022-11-22T08:07:16.593572Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-11-22T08:07:16.594739Z 0 [ERROR] Failed to create timer notify thread (errno= 1). 2022-11-22T08:07:16.594747Z 0 [ERROR] Failed to initialize timer component (errno 1). 2022-11-22T08:07:16.602209Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1 2022-11-22 08:07:16 0x7f20fa275740 InnoDB: Assertion failure in thread 139779612563264 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:07:16 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=500 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 mysqld(my_print_stacktrace+0x3b)[0x564c6064b71b] mysqld(handle_fatal_signal+0x377)[0x564c5fed6907] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f20fa2bc520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f20fa310a7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f20fa2bc476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f20fa2a27f3] mysqld(+0x6b0adc)[0x564c5feacadc] mysqld(_ZN2ib5fatalD1Ev+0x66)[0x564c609fa406] mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x564c608e8c57] mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x564c609a1b4a] mysqld(+0x106e9ad)[0x564c6086a9ad] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x564c5ff2b5af] mysqld(+0xc20ee5)[0x564c6041cee5] mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x564c60420405] mysqld(+0x6d4960)[0x564c5fed0960] mysqld(_Z11mysqld_mainiPPc+0x79e)[0x564c5fed202e] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f20fa2a3d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f20fa2a3e40] mysqld(_start+0x2a)[0x564c5fec65ea] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2022-11-22T08:07:16.823731Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-11-22T08:07:16.824821Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 1033 ... 2022-11-22T08:07:16.825926Z 0 [ERROR] Failed to create timer notify thread (errno= 1). 2022-11-22T08:07:16.825932Z 0 [ERROR] Failed to initialize timer component (errno 1). 2022-11-22T08:07:16.826810Z 0 [Note] InnoDB: PUNCH HOLE support available 2022-11-22T08:07:16.826819Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2022-11-22T08:07:16.826821Z 0 [Note] InnoDB: Uses event mutexes 2022-11-22T08:07:16.826824Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2022-11-22T08:07:16.826826Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12 2022-11-22T08:07:16.826829Z 0 [Note] InnoDB: Using Linux native AIO 2022-11-22T08:07:16.827079Z 0 [Note] InnoDB: Number of pools: 1 2022-11-22T08:07:16.827174Z 0 [Note] InnoDB: Using CPU crc32 instructions 2022-11-22T08:07:16.828340Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2022-11-22T08:07:16.832916Z 0 [Note] InnoDB: Completed initialization of buffer pool 2022-11-22T08:07:16.833810Z 0 [ERROR] [FATAL] InnoDB: pthread_create returned 1 2022-11-22 08:07:16 0x7f2dfe4ae740 InnoDB: Assertion failure in thread 139835516577600 in file ut0ut.cc line 921 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:07:16 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=500 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206887 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 mysqld(my_print_stacktrace+0x3b)[0x55c38d0bf71b] mysqld(handle_fatal_signal+0x377)[0x55c38c94a907] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f2dfe4f5520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f2dfe549a7c] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f2dfe4f5476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f2dfe4db7f3] mysqld(+0x6b0adc)[0x55c38c920adc] mysqld(_ZN2ib5fatalD1Ev+0x66)[0x55c38d46e406] mysqld(_Z21os_thread_create_funcPFPvS_ES_Pm+0x257)[0x55c38d35cc57] mysqld(_Z34innobase_start_or_create_for_mysqlv+0x1c7a)[0x55c38d415b4a] mysqld(+0x106e9ad)[0x55c38d2de9ad] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55c38c99f5af] mysqld(+0xc20ee5)[0x55c38ce90ee5] mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1e5)[0x55c38ce94405] mysqld(+0x6d4960)[0x55c38c944960] mysqld(_Z11mysqld_mainiPPc+0x79e)[0x55c38c94602e] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f2dfe4dcd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f2dfe4dce40] mysqld(_start+0x2a)[0x55c38c93a5ea] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. `

vanpelt commented 2 years ago

@GFiz you'll need to run mysql outside of the container and specify it's connection string with the MYSQL environment variable.

GFiz commented 2 years ago

@vanpelt Thank you very much; could you give me step by step instructions?

vanpelt commented 2 years ago

@GFiz most of our customers run this in a cloud that provides a managed MySQL instance. We have terraform and instructions for deploying into clouds here: https://docs.wandb.ai/guides/self-hosted

If you can't run this in a cloud, you can find instructions for setting things up in a bare metal environment here: https://docs.wandb.ai/guides/self-hosted/setup/on-premise-baremetal#mysql-database Ideally you would have a kubernetes cluster available and use our official helm chart to deploy MySQL.