Log viewer seems to show logs in scrambled order

[john8329]

Bug description The log viewer window seems to display log lines in a chaotic order when the service is crashing at every startup, making inspections more difficult.

Expected behavior The logs should not mix lines but display them in the same order as they were emitted by the container(s)

Steps to reproduce the issue:

1. Create a service that crashes on startup
2. Run it inside a global service with an restart policy set to on-failure
3. Check the log viewer

Technical details:

• Portainer version: 2.6.1
• Docker version (managed by Portainer): 20
• Platform (windows/linux): linux
• Use Case (delete as appropriate): Small startup, evaluating the tool for bigger deployments
• Have you reviewed our technical documentation and knowledge base? Yes

You can see in the screenshot that the golang trace is mixed with other lines (some details are obscured for security).

More details:

• The "server.yaml" message shouldn't happen in that moment in my app, it surely was from before the issue was fixed.
• This doesn't happen if I inspect the single container, only when I check the service's logs (which merges its containers' logs)
• Swarm tries to recreate the container and leaves the failed ones (rightly so), which are the ones that originate the crash logs

[srebala]

Hi please share the service yaml used to reproduce this issue

[john8329]

version: '3.7'
services:
  max:
    image: '###############'
    ports:
      - '#####:#####'
    volumes:
      - max_data:/opt/max/data
    environment:
      DB_HOST: 'database'
      DB_PORT: '#####'
      DB_USER: '#####'
      DB_PASS: '#####'
      DB_NAME: '#####'
      SERVER_NAME: ${CLIENT_NAME} Server
      PRINT_SERVICE_WEB_HOST: print
    networks:
      - backend
    depends_on:
      - print
      - database
    labels:
      #####.client-code: ${CLIENT_CODE}
      #####.client-name: ${CLIENT_NAME}
    deploy:
      mode: global
      restart_policy:
        condition: on-failure
...

[john8329]

After doing some more tests today, I can confirm it's quite reproducible. New logs aren't shown, and old logs get mixed up. The timestamps should be the evidence. Happens only when seeing them from the service, not the container.

[huib-portainer]

This has come up before as well https://github.com/portainer/portainer/issues/1942.

[SvenDowideit]

I'm going to suggest that this is really a Docker issue - https://github.com/moby/moby/issues/33673 (and many others) and I think you can improve it by using one of the more complicated log drivers - but there is buffering in the system, preferring not-losing info instead of preferring order