search

mattermost / mattermost-docker

Deprecated
Apache License 2.0
963 stars 576 forks source link

Hub processing might be deadlock with events in the buffer #473

Open geckiss opened 4 years ago

geckiss commented 4 years ago

Hello,

so our mattermost-docker EE

Mattermost Version: 5.20.0
Database Schema Version: 5.20.0
Database: postgres

has been running smoothly for 6 weeks. Around a hour ago, messages started being sent with delay. Some messages, AFTER being sent, appeared as NOT sent(the wheel was spinning and message was grayed out) and after that a little window on right side of the message appeared with 'Retry - Cancel' options. When i clicked on retry, message wasn't sent, and after switching to different channel and back, the message was sent(EDIT: only some messages were sent after channel switch)

Logs are showing this(the same error keeps popping up):

{"level":"error","ts":1588248976.2550566,"caller":"app/web_hub.go:98","msg":"Hub processing might be deadlock with events in the buffer","hub":2,"goroutine":4470,"events":4096}
{"level":"error","ts":1588248991.2550468,"caller":"app/web_hub.go:98","msg":"Hub processing might be deadlock with events in the buffer","hub":2,"goroutine":4470,"events":4096}
{"level":"error","ts":1588249006.2550635,"caller":"app/web_hub.go:98","msg":"Hub processing might be deadlock with events in the buffer","hub":2,"goroutine":4470,"events":4096}
{"level":"error","ts":1588249021.2550383,"caller":"app/web_hub.go:98","msg":"Hub processing might be deadlock with events in the buffer","hub":2,"goroutine":4470,"events":4096}
{"level":"error","ts":1588249036.255079,"caller":"app/web_hub.go:98","msg":"Hub processing might be deadlock with events in the buffer","hub":2,"goroutine":4470,"events":4096}
{"level":"error","ts":1588249051.2550786,"caller":"app/web_hub.go:98","msg":"Hub processing might be deadlock with events in the buffer","hub":2,"goroutine":4470,"events":4096}

Any idea how to fix/predict this? What could cause this? I'm going to restart container to see if it helps.

geckiss commented 4 years ago

Ok, so after 'restart', the container refused to get up:

a67d3a409dbb47c1f6b98f28f4f9fac728438cbad2aade9782098952fdac7eec: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/home/user/mattermost-docker/nfs-volumes/mattermost/config\\\" to rootfs \\\"/mnt/data/docker/overlay2/e3a32ed3e82bf852a998e7dd1794a7f6af070f0da1c5f6310ccc6a3b3a140fd2/merged\\\" at \\\"/mnt/data/docker/overlay2/e3a32ed3e82bf852a998e7dd1794a7f6af070f0da1c5f6310ccc6a3b3a140fd2/merged/mattermost/config\\\" caused \\\"no such file or directory\\\"\"": unknown

We have docker in /mnt/data/docker, not in /var/lib/docker, if you wondered.

So i've run sudo docker-compose down followed by sudo docker-compose build --no-cache. Built successfully. But after sudo docker-compose up -d, same error occured. This only happened for mattermost container, not for database.

The problem was that we had host volume directories on CIFS FS. Precisely, volume directory was on the server running container, and this directory was used as CIFS mountpoint. After 6 months of smooth run of around 8 containers, this happened for the first time. Partially solved by not using volume directory on CIFS but on host, which is using ext4 FS. I don't know if it's not going to happen again, but by searching I found that some mac users encountered the same problem again and again. Don't know it it is mattermost-docker problem or Docker Engine problem since it happened for the first time and with mattermost container. I would keep this open.

Example:

volumes:
  # Ext4 - works, for now...
  - /mnt/data/mattermost-docker/config:/mattermost/config:rw
  - /mnt/data/mattermost-docker/data:/mattermost/data:rw
  - /mnt/data/mattermost-docker/logs:/mattermost/logs:rw
  - /mnt/data/mattermost-docker/plugins:/mattermost/plugins:rw
  - /mnt/data/mattermost-docker/client-plugins:/mattermost/client/plugins:rw
  # CIFS - caused the issue...
  #- ./nfs-volumes/mattermost/config:/mattermost/config:rw
  #- ./nfs-volumes/mattermost/data:/mattermost/data:rw
  #- ./nfs-volumes/mattermost/logs:/mattermost/logs:rw
  #- ./nfs-volumes/mattermost/plugins:/mattermost/plugins:rw
  #- ./nfs-volumes/mattermost/client-plugins:/mattermost/client/plugins:rw