"Fatal Assertion" with "fsync: Invalid Argument"

[franz-josef-kaiser]

After a quick try to pull this container in with Docker Compose up, the following error (about the WiredTiger package) can be found in the logs:

$ docker-compose logs

db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=df7349b3db2d
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] db version v3.2.1
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] allocator: system
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] modules: none
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] build environment:
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten]     distarch: x86_64
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten]     target_arch: x86_64
db_1  | 2016-02-11T12:30:54.530+0000 I CONTROL  [initandlisten] options: {}
db_1  | 2016-02-11T12:30:54.535+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
db_1  | 2016-02-11T12:30:54.539+0000 E STORAGE  [initandlisten] WiredTiger (22) [1455193854:539175][1:0x7f2b9d5c0808], connection: /data/db/WiredTiger.wt: fsync: Invalid argument
db_1  | 2016-02-11T12:30:54.541+0000 I -        [initandlisten] Fatal Assertion 28561
db_1  | 2016-02-11T12:30:54.541+0000 I -        [initandlisten] 
db_1  | 
db_1  | ***aborting after fassert() failure

The docker-compose.yml file part:

db:
    image: mvertes/alpine-mongo
    volumes:
        - ./db:/data/db
    ports:
        - 27017

The Volume mapping successfully adds to the local FS the following structure:

\~
 |- /journal
 |- mongodb.lock
 |- WiredTiger
 \- WiredTiger.lock

… with the following contents – the mongodb.lock and the /journal folder stay empty:

# WiredTiger
WiredTiger
WiredTiger 2.7.0: (December  8, 2015)

# WiredTiger.lock
WiredTiger lock file

The error is repeatable. System: Docker on OS X via default Docker-Machine:

$ docker-machine -v
docker-machine version 0.5.1 (7e8e38e)

$ docker -v
Docker version 1.9.1, build a34a1d5

$ docker-compose -v
docker-compose version: 1.5.1

[mvertes]

Thanks for your report. I never used docker-compose until now. I'm investigating.

[mvertes]

Ok, I think it is a limitation of shared volumes (the ./db volume set by compose) and virtualbox driver of docker-machine. From the mongo db doc https://docs.mongodb.org/manual/administration/production-notes/#prod-notes-platform-considerations, I see the following:

MongoDB requires a filesystem that supports fsync() on directories. For example, HGFS and Virtual Box’s shared folders do not support this operation.

If the volume is not shared, but is local to the docker-machine, it works.

[mvertes]

I will add a caveat in the doc.

[franz-josef-kaiser]

If the volume is not shared, but is local to the docker-machine, it works.

What do you mean by "local to the docker-machine"? The volume currently is local inside the Docker-Machine.

[mvertes]

I mean a volume not shared between the virtualbox guest (the docker-machine running the container) and the virtualbox host (your MacOSX directory, where ./db comes from).

[mvertes]

In docker-compose.yml, comment out the volume section and test again:

db:
    image: mvertes/alpine-mongo
#    volumes:
#        - ./db:/data/db
    ports:
        - 27017

[franz-josef-kaiser]

comment out the volume section

Done that and it works. Still I need to find a way to tackle this. Holding the data inside the container is the opposite of good. Adding a second/fallback/backup instance is not possible this way.

Thanks for your pointers tough!

[mvertes]

The weakness is really in the virtualbox shared storage driver, it makes the MacOSX a less than ideal platform for docker, at least for mongodb. A possible solution is maybe to use a so-called data-only container for storage as described for example in https://medium.com/@ramangupta/why-docker-data-containers-are-good-589b3c6c749e#.2h1jhh557

[franz-josef-kaiser]

First off: Thanks for sticking to that topic. Ad OS X – coming from Win as dev environment, believe me, it could be worse.

A possible solution is maybe to use a so-called data-only container for storage

That's the first thing I tried. Problem then is, that the NFS filesystem is not natively supported by docker-machine. There're plugins to extend to that (yes, we're already at that point and yes, we already know it from Vagrant VMs), but it still does not solve the real issues. I guess the only thing we can decide now is to

1. leave that issue close and move further
2. try really, really hard to solve an unrelated issue as the official sources simply don't care

Again, thanks for sticking to it.

[mvertes]

I leave it open to keep a trace, but will not fix.

[indicalabs]

So does that mean it works in other operating systems other than MAC? does anyone have a work around for docker on MAC to host volume on host's os? The data container with volume mounting to host's directory is still a problem.

[mvertes]

It may be fixed now with new docker beta 1.12 on MacOS which doesn't rely on virtualbox. Not tested yet. Interested to get feedback on it.

[indicalabs]

@mvertes Thanks lot for the info. This problem doesn't exist with new docker beta 1.12 on MacOS. I think you can close this ticket.

[SivolcC]

Hi guys I have the same issue on Docker 1.12.3 running on Windows. I do not use compose, but specify the shared folder when I run my container docker run [...] -v $PWD/data:/data When checking the mongodb log, I find a message "aborting after fassert() failure" Is it related?

[mvertes]

It should be confirmed with the full logged details, but it has probably something to do with underlying host storage, i.e window filesystem. I do not run MS-Windows myself, and can not help much more here.

[neilgirardi-zz]

I am experiencing a similar issue with Docker on Windows 10 using MongoDB with the Wired Tiger storage engine. In my case, it works on OS X. To the best of my knowledge, Docker for Windows does not use Virtual Box, it uses a hypervisor.

The offending lines in the mongo service section of my docker-compose.yml were:

volumes:
  - ./data/mongodb:/data/db

It works fine on Windows 10 if I create a named volume for Mongo but do not mount it on the file system.

[stu-co]

I faced a similar issue using windows 10 and Vagrant.

So I have fixed this by switching to rsync instead of the Virtual Box provider. I then excluded the data folder from the rsync.

[grillazz]

Hi, Can confirm that issue exist on WInows 10 Ent and Docker version 17.09.0-ce, build afdb6d4 and docker-compose version 1.16.1, build 6d1ac219

Attaching to prototype_db_1
db_1   | going to wait for Mongo
db_1   | => Waiting for confirmation of MongoDB service startup...
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/data/db master=1 64-bit host=a178001671a4
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] db version v3.4.10
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] allocator: tcmalloc
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] modules: none
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten] build environment:
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten]     distmod: rhel62
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten]     distarch: x86_64
db_1   | 2017-11-08T07:42:05.416+0000 I CONTROL  [initandlisten]     target_arch: x86_64
db_1   | 2017-11-08T07:42:05.417+0000 I CONTROL  [initandlisten] options: { master: true, net: { http: { RESTInterfaceEnabled: true, enabled: true } }, security: { authorization: "enabled" }, storage: { engine: "wiredTiger" } }
db_1   | 2017-11-08T07:42:05.427+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=s
nappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
db_1   | 2017-11-08T07:42:05.455+0000 E STORAGE  [initandlisten] WiredTiger error (17) [1510126925:455834][7:0x7f4ad7805dc0], connection: /data/db/WiredTiger.wt: handle-open: open: File exists
db_1   | 2017-11-08T07:42:05.473+0000 I STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1
db_1   | 2017-11-08T07:42:05.476+0000 E STORAGE  [initandlisten] WiredTiger error (22) [1510126925:476364][7:0x7f4ad7805dc0], connection: /data/db/: directory-sync: fdatasync: Invalid argument
db_1   | 2017-11-08T07:42:05.481+0000 I -        [initandlisten] Fatal Assertion 28561 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 273
db_1   | 2017-11-08T07:42:05.481+0000 I -        [initandlisten]

[cpapidas]

The issue also exists on ubuntu

providers_service_mongo    | 2017-12-03T18:47:04.068+0000 E STORAGE  [initandlisten] WiredTiger (17) [1512326824:68460][6:0x7fb2f2a81c80], connection: /data/db/WiredTiger.wt: handle-open: open: File exists
providers_service_mongo    | 2017-12-03T18:47:04.069+0000 I STORAGE  [initandlisten] WiredTiger unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1
providers_service_mongo    | 2017-12-03T18:47:04.070+0000 E STORAGE  [initandlisten] WiredTiger (22) [1512326824:70371][6:0x7fb2f2a81c80], connection: /data/db/: directory-sync: fdatasync: Invalid argument
providers_service_mongo    | 2017-12-03T18:47:04.071+0000 I -        [initandlisten] Fatal Assertion 28561```