Extractors need update for working with nginx in docker container

[philicious]

tl;dr: nginx in a container + log-driver syslog produces messages like

docker/3242b31f11bc[795]: 87.12.12.250 - - [16/Aug/2016:13:32:58 +0000] "GET /df-swagger-ui/dist/images/logo-navbar-194x42.png HTTP/2.0" 200 21659 "https://acme.de/df-swagger-ui/dist/index.html" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0" "-" <msec=1471354378.222|connection=1|connection_requests=97|millis=0.017>

However the extractors expect the message to start with nginx: but we now have docker/3242b31f11bc[795]: which is docker/CONTAINERID[idontknow]:

So instead of nginx:\s+(\S+) we need to match ^.*:\s+(\S+)

I'm fixing up the extractors and hopefully will have a PR soon

PS: also the condition regexp need to be updated

[philicious]

I got all the access-log-input regex updated and working more generically, however when exporting, the JSON is all in one line and when beautifying, its still not suitable for a clean diff. So I made a gist https://gist.github.com/philicious/39a2b2317a79f4c01f6747d651fae0d6 (error-log-regex are fine)

lemme know if you want a PR or wanna pick the new regex from the gist

[philicious]

stupid me.. I noticed I can aswell only export the extractors. So I updated the gist to only contain the extractors.

error log extractors are working for dockerized nginx. however I added an extractor for the nginx error level and also a message extractor for cleaning the message field, just as the access.log-input does. lemme know if you are interested in that

[CrazyBS]

I could use this in a PR. I am trying to get my docker nginx logging to the central graylog services. However, I am slowly working through manually patching each of the extractors. It's just tedious.

[philicious]

@CrazyBS as I mentioned, I cant get a clean diff by exporting and would need to manually change all extractors to make a PR. so the best I can offer is that gist. However updating the extractors by copy/paste can be done in like 10mins

[Kiina]

So I'm kinda trying to understand this. I send my access logs to STDOUT and my error logs to STDERR. But with gelf-address I can only specify one port so I can't send to the access_log and error_log input because they are seperate? How is it supposed to work?

[philicious]

@Kiina that is true: docker log-drivers send both the STDOUT and STDERR to the same host+port as you can only specify one set.

What I did to solve this is running a separate tiny log "companion" container that has log dir shared with nginx container. nginx container only outputs access log and the other the error.log

  nginx-log:
    image: busybox
    container_name: nginx-log
    restart: always
    volumes_from:
      - nginx
    command: "tail -F /var/log/nginx/error.log"
    depends_on:
      - "nginx"
    logging:
      driver: gelf
      options:
       gelf-address: "udp://127.0.0.1:12201"

you can have a look at my issue covering this topic over at https://github.com/docker/docker/issues/25683

[Kiina]

Well I wanted to prevent using another docker container for this (because I already have a lot), but if that's the only option. Hopefully someone provides plugin support after 17.05 release. Thank you for your help

[philicious]

@Kiina dont worry about the extra container. its super small thanks to image busybox. it only consumes ~300KB of RAM and like no CPU.

[philicious]

Maybe the Graylog guys who did the GELF logging driver can make an optional logging plugin ontop that will split the merged stdout/stderr stream again based on the message field that holds this info and send both to different (supplied) host+ports. cc @lennartkoopmann @mariussturm related to https://github.com/graylog-labs/graylog-contentpack-nginx/issues/4

[mariussturm]

@philicious I haven't looked at the plugin api yet but for the core driver there was simply no option to distinguish between stderr and stdout.

[philicious]

@mariussturm ye the plugin API is quite new and only available in the upcoming 17.05. in the GELF driver, the field "level" is populated with the source stream (stderr/out) , see https://github.com/docker/docker/blob/master/daemon/logger/gelf/gelf.go#L127 the actual problem is that you cannot specify more than one host-port in log_opts. after my discussion with cpuguy here https://github.com/docker/docker/issues/25683 , I have some hope this can be solved without to much effort via a logging plugin. but I havent dived into that yet myself.

[mariussturm]

Ah right the level field is set. But then you can route the messages based on the value into different streams. Why do you need another input then?

[philicious]

@mariussturm err the problem was/is https://github.com/graylog-labs/graylog-contentpack-nginx/issues/4 so if its doable (meanwhile) , maybe an update to the nginx contentpack also would do the trick? (but I can think of more generic scenarios where it might be helpful to have the streams separated)

[mariussturm]

Nja ok, this content pack as it is now makes the assumption of having two inputs. So a user can simply copy&paste the nginx.conf snippet and all is setup. But from a technical standpoint there is no difference in separating messages by input or by a field value. So my suggestion would be to change the contentpack in a way that it works with a single input if you are using the Docker logging driver. That's way easier than changing the log driver. And it's more in the direction of doing it in the 'Graylog' way where streams and extractors/pipeline rules are the base entities.

[philicious]

hm ok. well then maybe a graylog-contentpack-nginx-docker would make sense :)

[joschi]

@philicious Feel free to fork this repository, make the necessary changes, and add it to the Graylog Marketplace.

[philicious]

@joschi unfortunately not working on that project anymore where I build the logging stack like that :/

[ronlut]

@joschi @philicious @mariussturm Took the time to do it as I needed it anyway, welcome to use and make a PR if you have any ideas for improvement :)

Links: github, marketplace

[joschi]

@ronlut Awesome, thanks a lot!

[philicious]

@ronlut awesome !

[philicious]

closing this issue while I salute to @ronlut 🥇