Question: How to see command logs in docker logs?

[bubbajames-docker]

I have jobber configured and running in a docker container. My jobs log 100's of JSON lines to stderr/stdout. If I configure the job to redirect output to a file, everything works nicely except I must attach a shell to inspect logs.

sitemap:
    cmd: /usr/local/bin/generateSitemap.sh >> /var/log/generateSitemap.log
    time: '*/30 0 3'
    onError: Continue

I would like the output of my scripts to be written to container's stdout/stderr (similar to how Jobber's run log is done) so that logs may be inspected using docker logs -f <container_name> or forwarded to ELK stack using Docker syslog log driver.

Result Sinks Using the resultSinks: with type: stdout is not an option as the output is buffered then wrapped with Jobber status. If 100's of log entries occur during script execution, the resulting output is too long to send to remote syslog listener (aka logstash). I would like EACH log entry to be written individually and forwarded to a syslog listener for processing.

Sample Log Entry as received by ELK (SHORTEST OF ALL COMMANDS) stderr contains four syslog entries prefixed with 2020-10-02T02:29:03+00:00 INFO (6):

{
  "fate": 0,
  "job": {
    "command": "/usr/local/bin/generateSitemap.sh",
    "name": "eventsAll",
    "status": "Good",
    "time": "*/30 0 3"
  },
  "startTime": 1601605740,
  "stderr": "2020-10-02T02:29:03+00:00 INFO (6): {\"function\":\"generateSitemap\",\"processor\":\"generateSitemap\",\"script\":\"/var/lib/blahblah/generateSitemap.php\",\"eventIds\":\"\",\"excludeEventIds\":\"19,44,74,75,222,1001,1011,1021,2000\",\"msg\":\"START\"}\n2020-10-02T02:29:03+00:00 INFO (6): {\"class\":\"Event_Dispatcher\",\"function\":\"processQueue\",\"processor\":\"processEvents-all\",\"script\":\"/var/lib/blahblah/generateSitemap.php\",\"eventIds\":\"\",\"excludeEventIds\":\"19,44,74,75,222,1001,1011,1021,2000\",\"eliminateDups\":0,\"batchSize\":1000,\"processAll\":0}\n2020-10-02T02:29:03+00:00 INFO (6): {\"class\":\"Event_Dispatcher\",\"function\":\"processQueue\",\"processor\":\"processEvents-all\",\"script\":\"/var/lib/blahblah/generateSitemap.php\",\"eventIds\":\"\",\"excludeEventIds\":\"19,44,74,75,222,1001,1011,1021,2000\",\"eliminateDups\":0,\"batchSize\":1000,\"processAll\":0,\"countEvents\":0,\"countProcessed\":0,\"countSkipped\":0,\"countError\":0,\"msg\":\"PROCESSED ALL EVENTS\"}\n2020-10-02T02:29:03+00:00 INFO (6): {\"function\":\"generateSitemap\",\"processor\":\"processEvents-all\",\"script\":\"/var/lib/blahblah/generateSitemap.php\",\"eventIds\":\"\",\"excludeEventIds\":\"19,44,74,75,222,1001,1011,1021,2000\",\"tt\":7,\"msg\":\"END\"}\n",
  "succeeded": true,
  "user": "jobberuser",
  "version": "1.4"
}

Other options tried:

sitemap:
    cmd: /usr/local/bin/generateSitemap.sh 2>&1
    time: '*/30 0 3'
    onError: Continue

Symlink log file to /dev/stderr

$ ls -l /var/log
lrwxrwxrwx 1 jobberuser jobberuser  11 Oct  2 04:14  generateSitemap.log -> /dev/stderr

Any assistance in how to accomplish the above would be greatly appreciated.

[bubbajames-docker]

Would it be possible to pass Jobber's logPath file handle to the Command's stdout and stderr instead of creating temporary files which buffer command's output for return in resultSinks?

https://github.com/dshearer/jobber/blob/658675e6fb73986cfae790142f76b8c7202e78c6/common/exec.go#L107

    // make temp files for stdout/stderr
    stdout, err := ioutil.TempFile(TempDirPath(), "")
    if err != nil {
        return nil, err
    }
    stderr, err := ioutil.TempFile(TempDirPath(), "")
    if err != nil {
        cleanUpTempfile(stdout)
        return nil, err
    }

    // give them to cmd
    cmd.Stdout = stdout
    cmd.Stderr = stderr

Then in job file, define Command's log entry such that stdout and/or stderr are directed to same file as prefs.logPath.

prefs:
  logPath: /var/log/jobber/jobber.log
  runLog:
    type: file
    path: /var/log/jobber/jobber-runs.log
    maxFileLen: 100m
    maxHistories: 2

jobs:
  sitemap:
    cmd: /usr/local/bin/generateSitemap.sh
    time: '*/30 0 3'
    onError: Continue
    log:
     - stdout
     - stderr

The above approach still allows use of resultSinks with stdout and/or stderr by using the io.MultiWriter() call passing common.Logger.Writer() or common.ErrLogger.Writer() and temporary file handle.

[jpabbuehl]

I would need this feature as well