error in date format parsing in logs

[rafipiccolo]

At first i did a Clean install and everything works :)

Then I install loki's docker log plugin to redirect all logs to loki : it keeps the default json log of docker. https://github.com/grafana/loki/tree/master/cmd/docker-driver Every logs gets redirected to loki as expected and everything continue working.

• docker logs : OK (because loki's plugin is also writing json logs file as usual)
• portainer : OK (can show this logs too)
• Dozzle : KO :(

I think its related to the date format : in your event stream (https://dozzle.flatbay.fr/api/logs/stream?id=c958bfe53d2a) i can see the following

Data: 2020-04-27T10:52:09.307471665Z xxxxxx

with loki's pluggin enabled i can see

Data: 2020-04-27T12:35:43.272974324+02:00 xxxxx

While both formats are correct, and means the same, you seem to only parse the first succesfully. Which results in an empty black screen for me.

Could you detect this format, or at least show the line without formating date when you can't parse the date ?

thanks.

[rafipiccolo]

Is this solution acceptable ? seems to work for me.

https://github.com/amir20/dozzle/blob/master/assets/components/LogEventSource.vue line 12

function parseMessage(data) {
  // the date is "all but space" then space, then message
  const matches = data.match(/([^ ]+) (.*)/);
  return {
    key: matches[1],
    date: new Date(matches[1]),
    message: matches[2].trim(),
  };
}

[amir20]

Hello, @rafipiccolo. Thank you for your amazing investigation skills. As someone without context, it really helps to understand what you are trying to do.

I haven't installed loki yet. I will try to do that this week.

I think you have the right idea. I looked at it and seems like new Date(...) will work with new format.

Instead of using match can you try indexOf(" ").. RegEx can be slow.

That should yield faster results. Initially when Dozzle loads, it fetches 100s of logs and performance really matters because it will block the UI thread.

Send me a PR once you have it. Also try to add a single new test case if you can.

[amir20]

1.22.4 should be up now. Give it a try.

[rafipiccolo]

it's working thx.

ps: my config for loki if you would like to give a try.

The docker log driver : https://github.com/grafana/loki/tree/master/cmd/docker-driver

The docker-compose

version: "3.3"
services:

  dozzle:
    image: amir20/dozzle
    container_name: dozzle
    restart: always
    ports:
        - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  datelogger:
    image: busybox
    container_name: datelogger
    command: sh -c "while true; do $$(echo date); sleep 1; done"
    restart: always

  # get a sample config file : 
  # docker run --rm --entrypoint cat grafana/loki /etc/loki/local-config.yaml
  loki:
    image: grafana/loki:latest
    restart: always
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki:/etc/loki
    command: -config.file=/etc/loki/config.yaml

# monitoring
# curl -X GET -H 'Content-type:application/json' 127.0.0.1:3100/ready
# curl -X GET -H 'Content-type:application/json' 127.0.0.1:3100/metrics

# discovery
# curl -s "http://localhost:3100/api/prom/label"
# curl -s "http://localhost:3100/api/prom/label/filename/values"

# querying
curl -G -s  "http://127.0.0.1:3100/loki/api/v1/query" --data-urlencode 'query={container_name="datelogger"}' | jq

# streaming
# curl -G -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Extensions: permessage-deflate' -H 'Sec-WebSocket-Key: v4vMUSLqpDDrrvhrCqfE+Q==' -H 'Connection: keep-alive, Upgrade' -H 'Upgrade: websocket' 'http://127.0.0.1:3100/loki/api/v1/tail' --data-urlencode 'query={job="varlogs"}'

# inserting
# curl -v -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}'

[amir20]

Thanks, Loki sounds interesting. So I'll give it a try when I get a chance.

[amir20]

There was a bug but I think I fixed it in https://github.com/amir20/dozzle/commit/d277b4e87801c4e3bf271fd2d620875f0db6dd52