ENV variables have no effect

[joshco]

Per the docs, I have an ENV file like this:

SYSLOG_HOSTNAME=xxxxx
SYSLOG_TAG=yyyyy

But neither the host or the tag is changing in papertrail. The logs are delivered, but they look like this:

Mar 09 02:52:12 107.21.222.222 logger:  [2a2f3a:foo] method=GET path=/...

Doing dokku logspout:info doesn't show them either:

# dokku logspout:info
=====> logspout status
       running
=====> dokku-logspout config
       DOKKU_LOGSPOUT_PORT=18000
       DOKKU_LOGSPOUT_IMAGE_VERSION=v3.1
       DOKKU_LOGSPOUT_SYSLOG_SERVER=tls://logs3.papertrailapp.com:NNNNN

What gives?

[michaelshobbs]

Admittedly, I don't use those features of logspout. It's quite possible this is a bug.

[michaelshobbs]

I have a patch for the info output. However the env is getting applied to the logspout container properly in the current version. Can you share the output of the following command? Are you using golang templates for either ENV vars (i.e. {{.Container....}})?

$ docker inspect -f '{{ .Config.Env }}' logspout
[SYSLOG_HOSTNAME=www.my-dokku-host.com SYSLOG_TAG={{.Container.Config.Hostname}} PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]

Additionally, can you please provide the version of docker you're running this on?

[joshco]

The output of the inspect is

[SYSLOG_HOSTNAME=myhero SYSLOG_TAG=myhero PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin]

In PaperTrail, I see this:

Mar 12 01:23:30 107.21.235.129 logger:  [9dad4d:node] method=GET path=/admin/...

[michaelshobbs]

Yeah I see the same thing in my test env. This is either an issue with something changing in logspout or Papertrail as this worked at one point in time. Not sure how the former is possible unless the image tag was overwritten. I don't handle logspout releases so I cannot speak to this. The second seems unlikely but who knows.

I can say that in production I'm using the dokku-logspout plugin and am seeing a custom host name come in with the messages. However I believe that is being controlled by the rsyslog configuration and not the dokku plugin.

Additionally I'm using the container name (logspout default I believe) as my syslog tag since dokku now names containers based on the app and proc names. IMO that's more useful than an arbitrary static tag but I suppose that may not work for your use case. I bring it up, however, just in case you weren't aware of the named containers feature.

[joshco]

I'm now in catchup mode since I was last active with the team here. Since the CHECKS work and other discussion, my system has been running a very old dokku (version 0.3?) and I'm now upgrading to the new dokku. The team has done a lot of excellent work and there's lots of new features!

In my previous dokku setup, I was using logging supervisord to manage both scaling and log persistence. I was using http://github.com/papertrail/remote_syslog2 to forward the generated log files content to papertrail.

With the new scaling and logging model in dokku, and no longer needing supervisord, I wasn't sure how to achieve the same result. Your logspout plugin caught my attention and I charged down that path. I'm now wondering if that isn't the best approach. These complications aside, I'm also ending up with logging data from all containers (redis, postgres) making my papertrail logs quite cluttered.

If I was able to resolve the tag name and host name issue, I could probably use the templating to segregate each container's log data in papertrail. But I don't want to do that because logs for redis, postgres et al are rarely looked at and they can be gotten on-demand. Furthermore, I don't want that noise counting towards my papertrail quota nor do i want to futz with the papertrail filtering system to suppress those messages to avoid the quota.

I'd rather just specifically identify which logs I want sent to papertrail and how they are is tagged. My initial and uninformed approach was to just copy my remote_syslog2 log_files.yml file to my new server until I realized I didn't know the dokku way to make the log files land in a persistent location like before (/var/log/dokku/appname/web.0.log)

What's the new best practice for that?

[michaelshobbs]

I believe there is an env var you can set to filter out particular containers by label from the logspout stream. Check the logspout docs (maybe?) or more likely search the GH issues or the codebase for how to do it.

[michaelshobbs]

Looks like it's EXCLUDE_LABEL https://github.com/gliderlabs/logspout/blob/105c59b70667dcb26ff6792923b4b5c71434a4b6/router/pump.go#L72

[michaelshobbs]

@joshco were you able to try out EXCLUDE_LABEL?

[joshco]

So I finally got to the point where I'm upgrading a production dokku to the current version and reexamining this.
Upgrading the logspout version to 3.2.6 made EXCLUDE_LABEL and ?filter start working. Excluding label dokku value service cut alot of the noise. Other noisy containers can be silence with ENV LOGSPOUT=ignore

My remaining issue is that DOCKER_OPTIONS_RUN

--link dokku.redis.osdiproxy:dokku-redis-osdiproxy
-l papertrail=true

does not add the label to the newly instantiated container as the docker docs say it should. This is probably a dokku issue

[joshco]

doh. That label needs to be in DOCKER_OPTIONS_DEPLOY. I'm all set.

[michaelshobbs]

Glad you got it sorted out