Mongos doesn't log inside container

[vinay-khatri]

I am trying to run mongos in docker with the below config. But docker container hangs at the start and sometimes output message with error code 38, failed to open /var/log/mongodb/mongos.log file. When i turn off the systemLog, everything seems working fine. Tried all different versions of mongo for docker but no solution. This bug is only inside docker containers. In real-world mongo application, this bug is not present and mongos logging fine on my ubuntu.

systemLog:`
    destination: file
    logAppend: true
    path: /var/log/mongodb/mongos.log

net:
    port: 27017
    bindIp: localhost

processManagement:
    timeZoneInfo: /usr/share/zoneinfo

sharding:
    configDB: config_set/192.168.101.11:27021,192.168.101.51:27021,192.168.101.52:27021,192.168.101.53:27021

While my Dockerfile looks like something this

RUN mkdir -p /home/app

COPY ./ ./home/app/

RUN chown -R mongodb:mongodb /home/app/*
RUN chmod 777 /var/log/mongodb

EXPOSE 27017
EXPOSE 27021
EXPOSE 27031
EXPOSE 27032

CMD ["mongos", "-f", "/home/app/mongos.yaml"]

[wglambert]

The --logpath is set to stdout, this is because docker logs reads from the container's stdout/stderr https://github.com/docker-library/mongo/blob/5761b490a391393ba4804dd171e8f4f2608b6d2d/4.4/docker-entrypoint.sh#L295

[vinay-khatri]

@wglambert i didn't understand what are you saying. Docker container is logging to my terminal fine in std output but its failing to create a log file inside container file system. Can you please elaborate in noobie terms?

[wglambert]

Well it's not doing both is the thing. That --logpath "/proc/$$/fd/1" means logpath is set to stdout instead of a file in the container.

If you want you can override that by setting --logpath /var/log/mongodb/mongos.log

$ docker run -d --name mongo mongo --logpath /var/log/mongodb/mongos.log
2b5dc0e41ae8660d4253d71e8e647cbd9cd0c454d06b3ffcd37f4787a7a5917a

$ docker logs mongo

$ docker exec mongo cat /var/log/mongodb/mongos.log
{"t":{"$date":"2021-07-06T15:40:59.783+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2021-07-06T15:40:59.787+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2021-07-06T15:40:59.788+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2021-07-06T15:40:59.789+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"2b5dc0e41ae8"}}
{"t":{"$date":"2021-07-06T15:40:59.789+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.6","gitVersion":"72e66213c2c3eab37d9358d5e78ad7f5c1d0d0d7","openSSLVersion":"OpenSSL 1.1.1  11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-07-06T15:40:59.789+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2021-07-06T15:40:59.789+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"systemLog":{"destination":"file","path":"/var/log/mongodb/mongos.log"}}}}
{"t":{"$date":"2021-07-06T15:40:59.791+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15575M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2021-07-06T15:41:00.945+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1625586060:945285][1:0x7f47ce306ac0], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
{"t":{"$date":"2021-07-06T15:41:00.945+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1625586060:945359][1:0x7f47ce306ac0], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)"}}
{"t":{"$date":"2021-07-06T15:41:00.951+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":1159}}
{"t":{"$date":"2021-07-06T15:41:00.951+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2021-07-06T15:41:00.959+00:00"},"s":"I",  "c":"STORAGE",  "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modifications are required for existing WiredTiger tables","attr":{"loggingEnabled":true}}
{"t":{"$date":"2021-07-06T15:41:00.959+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2021-07-06T15:41:00.961+00:00"},"s":"W",  "c":"CONTROL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
{"t":{"$date":"2021-07-06T15:41:00.962+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"admin.system.version","uuidDisposition":"provided","uuid":{"uuid":{"$uuid":"311cc23a-cf4b-418f-a587-4eaf78f98938"}},"options":{"uuid":{"$uuid":"311cc23a-cf4b-418f-a587-4eaf78f98938"}}}}
{"t":{"$date":"2021-07-06T15:41:00.979+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"admin.system.version","index":"_id_","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2021-07-06T15:41:00.979+00:00"},"s":"I",  "c":"COMMAND",  "id":20459,   "ctx":"initandlisten","msg":"Setting featureCompatibilityVersion","attr":{"newVersion":"4.4"}}
{"t":{"$date":"2021-07-06T15:41:00.979+00:00"},"s":"I",  "c":"STORAGE",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2021-07-06T15:41:00.980+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.startup_log","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"e5b039ce-7b07-4333-9bac-02ad216f99da"}},"options":{"capped":true,"size":10485760}}}
{"t":{"$date":"2021-07-06T15:41:00.996+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.startup_log","index":"_id_","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2021-07-06T15:41:01.005+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
{"t":{"$date":"2021-07-06T15:41:01.010+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"LogicalSessionCacheRefresh","msg":"createCollection","attr":{"namespace":"config.system.sessions","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"0a7fddd6-1c0c-45ed-8d76-c27903ddf665"}},"options":{}}}
{"t":{"$date":"2021-07-06T15:41:01.011+00:00"},"s":"I",  "c":"CONTROL",  "id":20712,   "ctx":"LogicalSessionCacheReap","msg":"Sessions collection is not set up; waiting until next sessions reap interval","attr":{"error":"NamespaceNotFound: config.system.sessions does not exist"}}
{"t":{"$date":"2021-07-06T15:41:01.011+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2021-07-06T15:41:01.011+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
{"t":{"$date":"2021-07-06T15:41:01.011+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
{"t":{"$date":"2021-07-06T15:41:01.040+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.system.sessions","index":"_id_","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2021-07-06T15:41:01.040+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.system.sessions","index":"lsidTTLIndex","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}

[vinay-khatri]

@wglambert you are right its actually saving logs fine even without --logpath command. But i am not able to connect my mongo client to this container : docker run -d -p 27017:27017 --name mongos mongos

Though i successfully connected using --netowrk host but i want to connect without host netowrk. Can you suggest what i am doing wrong?

[wglambert]

Here's a quick example

$ docker run -d -p 27017:27017 --name mongos -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=example mongo
590d7349bcfc0b266a2e578a3ca6fa691ad8027ac173d2b6a6d5d54827276f8a

$ docker inspect mongos | grep -i gateway
            "Gateway": "172.17.0.1",
            "IPv6Gateway": "",
                    "Gateway": "172.17.0.1",
                    "IPv6Gateway": "",

Getting the gateway since we're going to use a second mongo container as a client, pointed toward that address (which would be the host). Since the initialized mongo database is listening on the host's port 27017

$ docker run -it --rm mongo mongo --host 172.17.0.1 -u root -p example
MongoDB shell version v4.4.6
connecting to: mongodb://172.17.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b733ec47-7246-4ebd-80fc-42d91a8974cc") }
MongoDB server version: 4.4.6
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
        https://community.mongodb.com
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB

So the second container connects to the host's port 27017 which redirects to the first mongo container.

For inter-container connections though they'd want to be on the same Docker network, you wouldn't have to publish any ports since it'd be just internal between containers in that situation. Here's another example

$ docker network create test
86a0f6c2938797f18d1fd3d7c47ddf3c2417b4533eaad55b5dd4d6f5315a7498

$ docker run -d --network test --name mongos -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=example mongo
8e6fee4a2bc50f002d673dc62b0691cdc7057830a4f5a4625baefb946027ead9

$ docker run -it --rm --network test mongo mongo --host mongos -u root -p example
MongoDB shell version v4.4.6
connecting to: mongodb://mongos:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("c663de42-cd4e-4437-bea8-0f384d768806") }
MongoDB server version: 4.4.6
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
        https://community.mongodb.com
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB

In the future questions like these should be asked over at the Docker Community Forums, Docker Community Slack, or Stack Overflow. Since these repos aren't really a user-help forum

[vinay-khatri]

@wglambert i would rather use docker-compose to connect the client instead of creating networks manually, thanks for helping out anyway

[yosifkit]

Remove the bindIp from the net section of your Mongo yaml to make it accessible outside of container localhost (ie to other containers and to your host).

net:
    port: 27017
    bindIp: localhost