[SOLVED] QUESTION: If i wanna to make persistent the log files and not stdout?

xtreme-d / docker-slurm-cluster

Simple Slurm cluster in docker.

MIT License

9 stars 6 forks source link

[SOLVED] QUESTION: If i wanna to make persistent the log files and not stdout? #5

Closed m0rfeo closed 2 years ago

m0rfeo commented 2 years ago

I have to make changes on supervisor and conf files or just conf files are okay?

hackprime commented 2 years ago

I think you can try the following algorithm:

Remove logs related options from supervisor configs: stdout_logfile, stdout_logfile_maxbytes, stderr_logfile, stderr_logfile_maxbytes
Tell slurm processes to write logs into files
- Update slurmd.conf, set path to logs in options SlurmctldLogFile and SlurmdLogFile
- Update slurmdbd.conf, set path to logs in the option LogFile.
Recreate containers: docker-compose down && docker-compose up -d

That should work. Let me know if it doesn't.

m0rfeo commented 2 years ago

I create volume /logs to centralize all logs files there outside container and automatically backup with script created.

On conf file for slurm services just need declare path to save logs and remove options from supervisor configs.

But on munge, ssh and the other services the output of log file on /logs still empty

hackprime commented 2 years ago

@m0rfeo You might want to update the munged config for supervisor as well so its output will be placed into log files too. Something like this:

diff --git a/conf/supervisor.d/munged.ini b/conf/supervisor.d/munged.ini
index e53e027..fb2c27f 100644
--- a/conf/supervisor.d/munged.ini
+++ b/conf/supervisor.d/munged.ini
@@ -4,8 +4,8 @@ command=/usr/local/bin/prefix-log.sh /sbin/munged -F --key-file /etc/munge/munge
 environment=LD_LIBRARY_PATH=/usr/local
 user=munge
 group=munge
-stdout_logfile=/dev/stdout
+stdout_logfile=/var/log/slurm/munged.out.log
 stdout_logfile_maxbytes = 0
-stderr_logfile=/dev/stderr
+stderr_logfile=/var/log/slurm/munged.err.log
 stderr_logfile_maxbytes = 0
 priority=60

m0rfeo commented 2 years ago

@hackprime I did and for Munge works but for some reason ssh logs still empty.

Additionally, stdout_logfile_maxbytes=0 have to been commented rigth?

hackprime commented 2 years ago

I did and for Munge works but for some reason ssh logs still empty.

The current bundle doesn't contain the ssh daemon.

Additionally, stdout_logfile_maxbytes=0 have to been commented right?

Yes, correct.

m0rfeo commented 2 years ago

@hackprime I can access to the head container with ssh connection, don't undestand why logs still empty

sshd.ini: [program:sshd] autostart=true command=/usr/sbin/sshd -D stdout_logfile=/logs/sshd.log stderr_logfile=/logs/sshd.err.log priority=65

hackprime commented 2 years ago

I think you need to add option -e to route sshd logs into stdout instead of syslog. /usr/sbin/sshd -D -e

m0rfeo commented 2 years ago

@hackprime It is possible, but instead -e better -E, this is the correct one to choose log_file. I will try as fast as I can. Thanks!

m0rfeo commented 2 years ago

@hackprime SOLVED with -E log_path. In addition with -e can get output from stderr too