Issues with mongo with mounted volume for data

[skyksandr]

I've created container with MongoDB. Dockerfile:

FROM centos:centos6.7
MAINTAINER Aleksandr Kunin <akunin@c.ru>

RUN yum -y update \
    && yum -y install epel-release \
    && yum clean all

RUN yum -y install mongodb-server

RUN mkdir -p /data/db

EXPOSE 27017
ENTRYPOINT ["/usr/bin/mongod", "--smallfiles"]

docker-compose.yml

db:
  image: qstnnr/mongodb
  volumes:
    - /data/db:/data/db

I have two problems with it:

1. If I put in docker-compose.yml OS X folder

    - ./db:/data/db

Container do not start with error: Assertion: 13651:Couldn't fsync directory '/data/db': errno:22 Invalid argument

What I found on this topic is http://stackoverflow.com/questions/29570989/docker-mongodb-share-volume-with-mac-os-x:

It seems to be specific to the mongo software and the use of virtualbox , see the README

WARNING: because MongoDB uses memory mapped files it is not possible to use it through vboxsf to your host vbox bug. It doesn't behave like that if you keep the datadir out of the mounted volume.

2. If I put in docker-compose.yml folder from my docker-machine VM

 - /data/db:/data/db

Container starts but do not finishes. In result I have to reboot docker-machine VM because nothing can kill mongod process.

[legal90]

I should to clarify some important details about Docker volumes and shared folders:

• While using Docker Machine, it is assumed that the "Docker Host" is a VM, not a Mac host. It means that when we run docker (or docker-compose) on Mac, all volume paths are located on the VM. In your example, /data/db (on the left side of the colon) is the VM's path which will be mapped to the container. Docker will create this path if it does not exist.
• For the convenience, local drivers for Docker Machine (as well as virtualbox and parallels) mount the whole /Users folder from your Mac host to the /Users path in each Docker Machine VM. So, your /Users content became available from the VM by the same path and could be mapped to containers.

So, if you set the volume path which is a child of /Users (for example, relative path ./db), then the Mac's folder will be mapped to container. In this case the filesystem is prl_fs or vboxsf, depending on the driver. If you set some other volume path (/data/db), then this dir will be created on the VM only and it will not be accessible from your Mac directly. In this case there are no shared folders involved.

[legal90]

Let's abstract away from the Compose use case, there is an easier way to reproduce the subject issue. And yes, seems like Parallels Shared Folders could not be used for Mongo DB storage at this moment:

# We assume that Parallels Desktop VM is created and env is configured:
# eval "$(docker-machine env your-vm-name)"

$ docker run -v ~/test-docker-volume/data/db:/data/db -p 27017:27017 mongo --smallfiles

2015-12-13T08:50:55.589+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=6d735a8a9892
2015-12-13T08:50:55.590+0000 I CONTROL  [initandlisten] db version v3.2.0
2015-12-13T08:50:55.590+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] modules: none
2015-12-13T08:50:55.591+0000 I CONTROL  [initandlisten] build environment:
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten]     distmod: debian71
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten]     distarch: x86_64
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2015-12-13T08:50:55.592+0000 I CONTROL  [initandlisten] options: { storage: { mmapv1: { smallFiles: true } } }
2015-12-13T08:50:55.598+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-12-13T08:50:55.608+0000 E STORAGE  [initandlisten] WiredTiger (1) [1449996655:608691][1:0x7f33ade64c80], connection: /data/db/WiredTiger.wt: Operation not permitted
2015-12-13T08:50:55.610+0000 I -        [initandlisten] Assertion: 28595:1: Operation not permitted
2015-12-13T08:50:55.622+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 1: Operation not permitted, terminating
2015-12-13T08:50:55.623+0000 I CONTROL  [initandlisten] dbexit:  rc: 100

Seems like our prl_fs filesystem returns "Operation not permitted" for wiredtiger_open call. @romankulikov, is it similar to Virtualbox ticket about mmapped access? (https://www.virtualbox.org/ticket/819).

It works fine on the same VM, outside of Parallels Shared Folders:

$ docker run -v /data/db:/data/db -p 27017:27017 mongo --smallfiles

2015-12-13T09:20:54.234+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=de303b29f580
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] db version v3.2.0
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2015-12-13T09:20:54.235+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten] modules: none
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten] build environment:
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten]     distmod: debian71
2015-12-13T09:20:54.236+0000 I CONTROL  [initandlisten]     distarch: x86_64
2015-12-13T09:20:54.237+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2015-12-13T09:20:54.237+0000 I CONTROL  [initandlisten] options: { storage: { mmapv1: { smallFiles: true } } }
2015-12-13T09:20:54.240+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-12-13T09:20:54.280+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2015-12-13T09:20:54.280+0000 I CONTROL  [initandlisten]
2015-12-13T09:20:54.280+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten]
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-12-13T09:20:54.281+0000 I CONTROL  [initandlisten]
2015-12-13T09:20:54.287+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2015-12-13T09:20:54.289+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2015-12-13T09:20:54.289+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker

[romankulikov]

@legal90, I do think this is the same problem. prl_fs supports mmap-ed files, but with significant restrictions. For example, opening mmap-ed files for read-write is prohibited. Seems we should consider this to be improved from our side.

[alevlasu]

It needs to be checked against Parallels Desktop 14.

[legal90]

The issue still persist on Parallels Desktop 15.1.3.

1. With image mongo:3.2.0, the same what we tried above 4 years ago:

   
   $ docker run -v $(pwd)/:/data/db -p 27017:27017 mongo:3.2.0

2020-04-26T18:42:48.998+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=86c88cc84166 2020-04-26T18:42:48.998+0000 I CONTROL [initandlisten] db version v3.2.0 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] allocator: tcmalloc 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] modules: none 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] build environment: 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] distmod: debian71 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] distarch: x86_64 2020-04-26T18:42:48.999+0000 I CONTROL [initandlisten] target_arch: x86_64 2020-04-26T18:42:49.000+0000 I CONTROL [initandlisten] options: {} 2020-04-26T18:42:49.003+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2020-04-26T18:42:49.010+0000 E STORAGE [initandlisten] WiredTiger (17) [1587926569:10438][1:0x7ff458c55c80], connection: /data/db/WiredTiger.wt: File exists 2020-04-26T18:42:49.010+0000 I STORAGE [initandlisten] WiredTiger unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1 2020-04-26T18:42:49.011+0000 E STORAGE [initandlisten] WiredTiger (1) [1587926569:11385][1:0x7ff458c55c80], connection: /data/db/WiredTiger.wt: Operation not permitted 2020-04-26T18:42:49.012+0000 I - [initandlisten] Assertion: 28595:1: Operation not permitted 2020-04-26T18:42:49.012+0000 I STORAGE [initandlisten] exception in initAndListen: 28595 1: Operation not permitted, terminating 2020-04-26T18:42:49.012+0000 I CONTROL [initandlisten] dbexit: rc: 100

2. With image `mongo:4.2.6`, the latest at this moment:

$ docker run -v $(pwd)/:/data/db -p 27017:27017 mongo:4.2.6

2020-04-26T18:47:39.495+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' 2020-04-26T18:47:39.497+0000 W ASIO [main] No TransportLayer configured during NetworkInterface startup 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=e6fffb9170e6 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] db version v4.2.6 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] git version: 20364840b8f1af16917e4c23c1b5f5efd8b352f8 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] allocator: tcmalloc 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] modules: none 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] build environment: 2020-04-26T18:47:39.498+0000 I CONTROL [initandlisten] distmod: ubuntu1804 2020-04-26T18:47:39.499+0000 I CONTROL [initandlisten] distarch: x86_64 2020-04-26T18:47:39.499+0000 I CONTROL [initandlisten] target_arch: x86_64 2020-04-26T18:47:39.499+0000 I CONTROL [initandlisten] options: { net: { bindIp: "*" } } 2020-04-26T18:47:39.500+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress], 2020-04-26T18:47:39.893+0000 E STORAGE [initandlisten] WiredTiger error (17) [1587926859:893216][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:893216][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists 2020-04-26T18:47:39.894+0000 I STORAGE [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.7 2020-04-26T18:47:39.895+0000 E STORAGE [initandlisten] WiredTiger error (1) [1587926859:895416][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:895416][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted 2020-04-26T18:47:39.901+0000 E STORAGE [initandlisten] WiredTiger error (17) [1587926859:901779][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:901779][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists 2020-04-26T18:47:39.903+0000 I STORAGE [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.8 2020-04-26T18:47:39.903+0000 E STORAGE [initandlisten] WiredTiger error (1) [1587926859:903528][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:903528][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted 2020-04-26T18:47:39.909+0000 E STORAGE [initandlisten] WiredTiger error (17) [1587926859:909346][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:909346][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists 2020-04-26T18:47:39.910+0000 I STORAGE [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.9 2020-04-26T18:47:39.910+0000 E STORAGE [initandlisten] WiredTiger error (1) [1587926859:910614][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:910614][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted 2020-04-26T18:47:39.917+0000 E STORAGE [initandlisten] WiredTiger error (17) [1587926859:917648][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:917648][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists 2020-04-26T18:47:39.918+0000 I STORAGE [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.10 2020-04-26T18:47:39.918+0000 E STORAGE [initandlisten] WiredTiger error (1) [1587926859:918737][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:918737][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted 2020-04-26T18:47:39.924+0000 E STORAGE [initandlisten] WiredTiger error (17) [1587926859:924934][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1587926859:924934][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: File exists 2020-04-26T18:47:39.925+0000 I STORAGE [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.11 2020-04-26T18:47:39.926+0000 E STORAGE [initandlisten] WiredTiger error (1) [1587926859:926055][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted Raw: [1587926859:926055][1:0x7f236ff77b00], connection: posix_open_file, 667: /data/db/WiredTiger.wt: handle-open: open: Operation not permitted 2020-04-26T18:47:39.927+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 2020-04-26T18:47:39.928+0000 F STORAGE [initandlisten] Reason: 1: Operation not permitted 2020-04-26T18:47:39.928+0000 F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 915 2020-04-26T18:47:39.928+0000 F - [initandlisten]

***aborting after fassert() failure

Although, it's pretty rear use case. I'm not sure how often people want to mount the db dir to the container from their Mac host 🤔 

[romankulikov]

The issue still persist on Parallels Desktop 15.1.3.

Investigating this. The thing is that WiredTiger starts successfully over SharedFolders on Ubuntu distro, but does fail in boot2docker.

[teran]

My +1 for this issue. I've noticed kinda similar behaviour with PostgreSQL database on external volume mounted from the host.

Any updates on that?

[romankulikov]

I've failed to debug the problem the previous year and postponed an attempt. If there's an interest in the problem I may try again. Any additional info and steps to reproduce are welcome.

[teran]

Hello @romankulikov,

Thanks for the update!

I've made a really trivial thing: launched docker run -v /Volumes/externalDisk/directory:/data postgres and tried to run initdb on that directory.

Things I've noted:

• chown -R postgres:postgres /data made no changes on source filesystem, i.e. have no changed owner UID:GID on /Volumes/externalDisk/directory so it looks like there's some mapping should be within the driver or it's not supported at all
• initdb failed on creating directory for wal so it looks like mkdir was failed in my case.

I could gather more information including strace output if needed but it looks like it should be reproduced in really simple way.

BTW. RO-volume is working perfectly so it looks like it's only about writes.

Please let let me know if you need any further details.