giovtorres / slurm-docker-cluster

A Slurm cluster using docker-compose
MIT License
319 stars 188 forks source link

Cannot open connection to slurmdbd #33

Closed abhinavDhulipala closed 2 months ago

abhinavDhulipala commented 1 year ago

Following the setup from the README, I did:

During registration, I get the following error:

sacctmgr: error: slurm_persist_conn_open_without_init: failed to open persistent connection to slurmdbd:6819: Connection timed out
sacctmgr: error: slurmdbd: Sending PersistInit msg: Connection timed out
sacctmgr: error: Problem talking to the database: Connection timed out

I have waited several minutes for all services to be initialized. Here is the tail of my docker-compose logs.

mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Number of transaction pools: 1
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
mysql      | 2023-05-28 22:13:12 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Completed initialization of buffer pool
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: 128 rollback segments are active.
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: log sequence number 46702; transaction id 14
mysql      | 2023-05-28 22:13:12 0 [Note] Plugin 'FEEDBACK' is disabled.
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
mysql      | 2023-05-28 22:13:12 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
mysql      | 2023-05-28 22:13:12 0 [Note] InnoDB: Buffer pool(s) load completed at 230528 22:13:12
mysql      | 2023-05-28 22:13:12 0 [Note] Server socket created on IP: '0.0.0.0'.
mysql      | 2023-05-28 22:13:12 0 [Note] Server socket created on IP: '::'.
mysql      | 2023-05-28 22:13:12 0 [Note] mariadbd: ready for connections.
mysql      | Version: '10.10.4-MariaDB-1:10.10.4+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
$ docker ps
21cef238b374   slurm-docker-cluster:19.05.2   "/usr/local/bin/dock…"   14 minutes ago   Up 14 minutes   6818/tcp   c1
6e9f84fcb3e9   slurm-docker-cluster:19.05.2   "/usr/local/bin/dock…"   14 minutes ago   Up 14 minutes   6818/tcp   c2
0b081fd724d6   slurm-docker-cluster:19.05.2   "/usr/local/bin/dock…"   14 minutes ago   Up 10 minutes   6817/tcp   slurmctld
9871f52a1d76   slurm-docker-cluster:19.05.2   "/usr/local/bin/dock…"   14 minutes ago   Up 10 minutes   6819/tcp   slurmdbd
bd5d5c751998   mariadb:10.10                  "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   3306/tcp   mysql

Local Machine Details:

$ docker version
Client:
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.1
 Git commit:        20.10.21-0ubuntu1~20.04.2
 Built:             Thu Apr 27 05:56:19 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.24
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       5d6db84
  Built:            Wed May 24 23:31:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker compose:

$ docker-compose version
Docker Compose version v2.17.2
giovtorres commented 2 months ago

Is this still an issue for you?