KafkaException: Failed to acquire lock on file .lock in /usr/share/graylog/data/journal

[riverar]

I tried deploying Graylog 3.1 as-is, vanilla, with its own exclusive volumes, and it fails to start up.

Exception snippet:

graylog_1 | 2020-01-10 05:09:23,944 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
graylog_1 | 2020-01-10 05:09:23,979 INFO : org.graylog2.plugin.system.NodeId - Node ID: 374dc272-6ef5-4bc6-9e2f-e605f70ddab9
graylog_1 | 2020-01-10 05:09:24,265 INFO : kafka.log.LogManager - Loading logs.
graylog_1 | 2020-01-10 05:09:24,277 INFO : kafka.log.LogManager - Logs loading complete.
graylog_1 | 2020-01-10 05:09:24,349 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
graylog_1 | 2020-01-10 05:09:24,372 ERROR: org.graylog2.shared.journal.KafkaJournal - Unable to start logmanager.
graylog_1 | kafka.common.KafkaException: Failed to acquire lock on file .lock in /usr/share/graylog/data/journal. A Kafka instance in another process or thread is using this directory.
graylog_1 |      at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
graylog_1 |      at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
graylog_1 |      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]

Repro steps:

1. Set up Docker-EE or Docker-CE on a Windows host with Linux container support
2. Create a graylog folder to work within
3. Inside this folder, create docker-compose.yml with the contents of the file below
4. Inside this folder, create a config folder
5. Inside the config folder, download the following files (from docs):
   • https://raw.githubusercontent.com/Graylog2/graylog-docker/3.1/config/graylog.conf
   • https://raw.githubusercontent.com/Graylog2/graylog-docker/3.1/config/log4j2.xml
6. Inside the graylog folder, issue command: docker-compose up
7. Observe .lock errors

docker-compose.yml:

version: '2.4' # Bumped to 2.4 for "platform: linux" support
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    platform: linux
    image: mongo:3
    volumes:
      - mongo_data:/data/db
    networks:
      graylog:
        aliases:
          - mongo

  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/docker.html
  elasticsearch:
    platform: linux
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      graylog:
        aliases:
          - elasticsearch
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    mem_limit: 1g

  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    platform: linux
    image: graylog/graylog:3.1
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
      - ./config:/usr/share/graylog/data/config
    networks:
      graylog:
        aliases:
          - graylog
    environment:
      - GRAYLOG_PASSWORD_SECRET=aaaabbbbccccdddd
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp

networks:
  graylog:
    driver: nat

volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local

docker version

Client: Docker Engine - Enterprise
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        2ee0c57608
 Built:             11/13/2019 08:00:16
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Enterprise
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.8
  Git commit:       c92ab06ed9
  Built:            09/03/2019 16:35:47
  OS/Arch:          windows/amd64
  Experimental:     true

[jalogisch]

dear @riverar

the solution might be simple - please check this FAQ entry: https://docs.graylog.org/en/3.1/pages/faq.html#dedicated-partition-for-the-journal

This is the result from my research in: https://github.com/Graylog2/graylog2-server/issues/2348

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

[riverar]

Hi @jalogisch, this is a new volume, the equivalent of a brand new dedicated partition. Am I missing something? Thanks!

[jalogisch]

check if the user is able to write to this directory and if that directory does contain other files.

[riverar]

@jalogisch It's writable. And I'm using a new volume each time.

[jalogisch]

can the userid 1100 what is used by the user graylog write to that folder?

[riverar]

@jalogisch It's writable. And I'm using a new volume each time. The graylog user is able to write to the folder during startup.

[jalogisch]

what filesystem is that volumen on @riverar ?

Did you have the same system if you create a docker-volumen and use that?