Log rotation using newsyslog

[emanzx]

Hi,

I was notice that when I use newsyslog rotation options eye does not know the log file is rotated and keep writing on old log file. what type of signal should I send to eye to make eye know that the log file has been rotated and update the log output.

[kostya]

how you use it? example of config.

[emanzx]

Here are the newsyslog config I used for eye

[system@dnsvgitnpm ~]$ more /etc/newsyslog.conf.d/eye.conf 
#logfilename                       [owner:group] mode count size   when  flags [/pid_file] [sig_num]
/var/log/dnsvault/eye/*.log        system:wheel  644  7     *      $D0   GN

and here are my eye configs

Eye.config do
  logger '/var/log/dnsvault/eye/eye.log'
end

Eye.application 'services' do
  check :cpu, every: 10.seconds, below: 100, times: 3 # global check for all processes

    working_dir "/root/"
    stdall "/var/log/dnsvault/eye/trash.log" # stdout,err logs for processes by default
    env 'APP_ENV' => 'development' # global env for each processes
    trigger :flapping, times: 10, within: 1.minute, retry_in: 5.minutes
    chain grace: 10.seconds # chained start-restart with 10s interval, one by one.
    process :tincd do
       pid_file "/var/run/tinc.dnsvnet.pid"
       start_command "service tincd onestart"
       stop_command "service tincd onestop"
       restart_command "service tincd onerestart"
       stdall "/var/log/dnsvault/eye/tincd.log"
       start_timeout 60.seconds
    end
    process :avahi_daemon do
       pid_file "/var/run/avahi-daemon/pid"
       start_command "service avahi-daemon onestart"
       stop_command "service avahi-daemon onestop"
       restart_command "service avahi-daemon onerestart"
       stdall "/var/log/dnsvault/eye/avahi-daemon.log"
       start_timeout 60.seconds
       depend_on [:tincd, :avahi_autoipd]
    end
    process :avahi_autoipd do
       pid_file "/var/run/avahi-autoipd.tap0.pid"
       start_command "/usr/local/sbin/avahi-autoipd -w -D tap0"
       stop_command "/usr/local/sbin/avahi-autoipd -k tap0"
       stdall "/var/log/dnsvault/eye/avahi-autoipd.log"
       start_timeout 60.seconds
       depend_on :tincd
    end
    process :netdata do
       pid_file "/var/run/netdata/netdata.pid"
       start_command "service netdata start"
       stop_command "service netdata stop"
       restart_command "service netdata restart"
       stdall "/var/log/dnsvault/eye/netdata.log"
       start_timeout 60.seconds
    end
    process :postgresql do
       pid_file "/usr/home/pgsql/data/postmaster.pid"
       start_command "service postgresql start"
       stop_command "service postgresql stop"
       restart_command "service postgresql restart"
       stdall "/var/log/dnsvault/eye/postgresql.log"
       start_timeout 60.seconds
    end
    process :pgbouncer do
       pid_file "/var/run/pgbouncer/pgbouncer.pid"
       start_command "service pgbouncer start"
       stop_command "service pgbouncer stop"
       restart_command "service pgbouncer restart"
       stdall "/var/log/dnsvault/eye/pgbouncer.log"
       start_timeout 60.seconds
    end
    process :timescale_prometheus do
       pid_file "/var/run/timescale-prometheus.pid"
       start_command "/usr/home/system/dnsvault/sbin/timescale-prometheus -db-port 7997 -db-user ***** -db-password **** -db-name *****"
       stdall "/var/log/dnsvault/eye/timescale-prometheus.log"
       start_timeout 60.seconds
       daemonize true
    end
    process :nginx do
       pid_file "/var/run/nginx.pid"
       start_command "service nginx start"
       stop_command "service nginx stop"
       restart_command "service nginx restart"
       stdall "/var/log/dnsvault/eye/nginx.log"
       start_timeout 60.seconds
       depend_on [:tincd, :avahi_autoipd, :avahi_daemon]
    end
    process :redis do
       pid_file "/var/run/redis/redis.pid"
       start_command "service redis start"
       stop_command "service redis stop"
       restart_command "service redis restart"
       stdall "/var/log/dnsvault/eye/redis.log"
       start_timeout 60.seconds
    end
    process :syncthing do
       env 'HOME' => "/usr/home/system"
       working_dir "/usr/home/system/"
       uid "system"
       gid "system"
       pid_file "/var/run/syncthing.pid"
       start_command "/usr/home/syncthing/syncthing"
       stdall "/var/log/dnsvault/eye/syncthing.log"
       start_timeout 60.seconds
       daemonize true
    end
end

Eye.application 'dnsvault' do
  env 'RBENV_ROOT' => '/usr/local/bin/rbenv', 'PATH' => "/home/system/.rbenv/shims:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/system/bin"
  check :cpu, every: 10.seconds, below: 100, times: 3 # global check for all processes

    working_dir "/usr/home/system/api"
    stdall "log/trash.log" # stdout,err logs for processes by default
    env 'APP_ENV' => 'development' # global env for each processes
    trigger :flapping, times: 10, within: 1.minute, retry_in: 10.minutes
    chain grace: 5.seconds # chained start-restart with 5s interval, one by one.
    uid "system"
    #gid "wheel"

    process :sidekiq do |p|
      daemonize true
      pid_file "tmp/pids/sidekiq.pid"
      stdall "log/sidekiq.log"
      start_command "bundle exec sidekiq -e development"
      start_timeout 30.seconds
      stop_signals [:USR1, 0, :TERM, 10.seconds, :KILL]
      check :cpu, every: 30, below: 100, times: 5
      check :memory, every: 30, below: 300.megabytes, times: 5
    end
    process :rails do
      pid_file "tmp/pids/server.pid"
      start_command "bundle exec rails s  puma -b 0.0.0.0 -d"
      stop_command "kill -9 {PID}"
      #daemonize true
      #stop_signals [:QUIT, 2.seconds, :TERM, 1.seconds, :KILL]
      start_timeout 60.seconds
      stop_timeout 60.seconds
    end
    process :zone_watcher do
       pid_file "/tmp/zone_watcher.pid"
       start_command "/usr/home/system/api/bin/zone_watcher.sh"
       stdall "/var/log/dnsvault/eye/zone_watcher.log"
       stop_signals [:INT, 5.seconds, :QUIT, 2.second, :KILL]
       start_timeout 60.seconds
       daemonize true
    end
    process :settings_watcher do
       pid_file "/tmp/settings_watcher.pid"
       start_command "/usr/home/system/api/bin/settings_watcher.sh"
       stdall "/var/log/dnsvault/eye/settings_watcher.log"
       stop_signals [:INT, 5.seconds, :QUIT, 2.second, :KILL]
       start_timeout 60.seconds
       daemonize true
    end
    process :log_publisher do
       pid_file "/tmp/log_publisher.pid"
       start_command "ruby /usr/home/system/api/bin/log_publisher.rb"
       stdall "/var/log/dnsvault/eye/log_publisher.log"
       stop_signals [:INT, 5.seconds, :QUIT, 2.second, :KILL]
       start_timeout 60.seconds
       daemonize true
    end
end

Looking at the settings I dump all of the log to /var/log/dnsvault/eye for all eye related process and I used the newsyslog rotate to wildcard all file with .log and rotate the file every night. normally newsyslog will signal the process that the log file it process and the process will update reopen and close the log file to update to new rotated log file. But I dont know how to handle it using eye apart from just sending "eye restart all" command which will cause unnecessarily short downtime and maybe some data lost for critical process.

[emanzx]

or eye have its own logging facility? which can do rotation too?

[kostya]

i not familiar with newsyslog, but rotating utils should have option to trunc log file, instead of move it, so no need to do anything for eye with logs. i think you should use it.

[emanzx]

Do you mean logrotate ?

[emanzx]

noted. It seem logrotate can do copytruncate which is better than newsyslog. closing this issue.