search

docker-library / mongo

Docker Official Image packaging for MongoDB
https://www.mongodb.org/
Apache License 2.0
1.03k stars 620 forks source link

Initialization of a user defined database, username, and password using environment variables #329

Open johnwyles opened 5 years ago

johnwyles commented 5 years ago

This issue (I would call it a bug but perhaps it is a feature request) is that users would like to a la docker-compose.yml and/or environment variables be able to set a database with a username and password they specify upon launch of the image.

Background:

This issue was filed https://github.com/docker-library/mongo/issues/174 and closed because the behavior of a PR https://github.com/docker-library/mongo/pull/145 was mentioned as the solution. What https://github.com/docker-library/mongo/pull/145 actually does and what users expect are entirely different. What PR https://github.com/docker-library/mongo/pull/145 does is set a user with elevated permissions (i.e. "root" user) that has superuser access to the entire MongoDB instance (as mentioned in https://github.com/docker-library/mongo/issues/174#issuecomment-460448991. However what most users expect from these environment variables is that a database they specify is initialized with the username and password they have set. It is confusing that these environment variables (MONGO_INITDB_DATABASE, MONGO_INITDB_ROOT_PASSWORD and MONGO_INITDB_ROOT_USERNAME) pertain to only setting a user with the role root on the database admin and initializing an user specified database for .js and .sh scripts in /docker-entrypoint-initdb.d/ to be run against.

Proposal:

We should make the environment variables very explicitly named in what they do in addition to adding others for the behavior I think most users come to expect when reading the variable names. Since it is the case most users would like their instance initialized with a database of their specification we should add this feature to meet that expectation.

Reasons for change:

References:

Involved Persons:

@mmi-rperez @tianon @vutran1710 @yosifkit @lonix1 @johnwyles

lonix1 commented 5 years ago

Agreed! There should be an easy way to spin up a container, with a new database, username and password - all specified in docker-compose / environment variables.

yosifkit commented 5 years ago

I understand your confusion. We only create a user with the root role (ie superuser as is done in the Postgres image) and users expect that the user is created in the database that they specified by MONGO_INITDB_DATABASE. This is not possible for the root role. The root role only exists in the admin database (probably for security reasons). Any other role not created in the admin database would not have full access to control MongoDB.

See also,

As for the database not existing, that is just how MongoDB works as I have explained before. If nothing is inserted the database does not exist (or conversely every database exists, you just have to use it).

I'm mostly against adding more environment variables for creating a non-admin user, especially if it breaks backwards compatibility. We try to keep the images as close to upstream releases as possible with minimal maintenance and the entrypoint script is already very complex.

A simple Dockerfile:

FROM mongo:4.0
COPY custom-user.sh /docker-entrypoint-initdb.d/

and a short custom-user.sh file:

#!/bin/bash
set -e;

# a default non-root role
MONGO_NON_ROOT_ROLE="${MONGO_NON_ROOT_ROLE:-readWrite}"

if [ -n "${MONGO_NON_ROOT_USERNAME:-}" ] && [ -n "${MONGO_NON_ROOT_PASSWORD:-}" ]; then
    "${mongo[@]}" "$MONGO_INITDB_DATABASE" <<-EOJS
        db.createUser({
            user: $(_js_escape "$MONGO_NON_ROOT_USERNAME"),
            pwd: $(_js_escape "$MONGO_NON_ROOT_PASSWORD"),
            roles: [ { role: $(_js_escape "$MONGO_NON_ROOT_ROLE"), db: $(_js_escape "$MONGO_INITDB_DATABASE") } ]
            })
    EOJS
else
    # print warning or kill temporary mongo and exit non-zero
fi

Combine that with automated builds (https://docs.docker.com/docker-hub/builds/) and repository links (https://docs.docker.com/docker-hub/builds/#repository-links) and it's reasonably easy to have an up-to-date image built FROM mongo with the custom non-root user creation modifications.

$ docker run -d -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=supersecret -e MONGO_INITDB_DATABASE=notadmin -e MONGO_NON_ROOT_USERNAME=normal -e MONGO_NON_ROOT_PASSWORD=secret --name some-mongo custom-mongo

$ docker run -it --rm --link some-mongo mongo:4.0 mongo -u normal -p secret some-mongo/notadmin
MongoDB shell version v4.0.6
connecting to: mongodb://some-mongo:27017/notadmin?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("0941eaa4-9180-4012-a24c-60fe8369b06a") }
MongoDB server version: 4.0.6
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
    http://docs.mongodb.org/
Questions? Try the support group
    http://groups.google.com/group/mongodb-user
> use admin;
switched to db admin
> db.users.findOne();
2019-02-05T23:57:10.838+0000 E QUERY    [js] Error: error: {
    "ok" : 0,
    "errmsg" : "not authorized on admin to execute command { find: \"users\", filter: {}, limit: 1.0, singleBatch: true, lsid: { id: UUID(\"c86d23b2-9ed4-4799-9d3e-74f327366550\") }, $db: \"admin\" }",
    "code" : 13,
    "codeName" : "Unauthorized"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
DBCommandCursor@src/mongo/shell/query.js:708:1
DBQuery.prototype._exec@src/mongo/shell/query.js:113:28
DBQuery.prototype.hasNext@src/mongo/shell/query.js:288:5
DBCollection.prototype.findOne@src/mongo/shell/collection.js:260:10
@(shell):1:1
> use notadmin;
switched to db notadmin
> db.users.findOne();
null

This unclear unless you look at docker-entrypoint.sh in this repository and no where clearly stated in the documentation as such

This seems clear to me (emphasis added):

This variable allows you to specify the name of a database to be used for creation scripts in /docker-entrypoint-initdb.d/*.js (see Initializing a fresh instance below). MongoDB is fundamentally designed for "create on first use", so if you do not insert data with your JavaScript files, then no database is created.

- source of docs from Docker Hub

@tianon has a PR for improving the user section: https://github.com/docker-library/docs/pull/1418

johnwyles commented 5 years ago

@yosifkit Thank you for that response. Given the majority of developers are writing their code and initialization and configuration such that they assume (whether rightly or wrongly) that their database is there and a username and password they know is able to talk to and operate on it I think it would be a great addition to the image.

Given your excellently laid out example above I think this should be baked in and not putting onus on the users of this image to essentially all have copies of the script you mentioned running around in it's various permutations. What do you say to including it in the image:

I think this is excellent if you were driving at building this into the image and see no reason not to since 99% of cases users will want a pre-initialized database with a username and password - putting this in documentation will help but having the extra step of them essentially copy/pasting your script so they can get what would be expected behavior of an image with an upstream project like this where the vast majority of developers and users are not having to write the database / username/password initialization into their code after connecting with a root role user. It also centralizes the location credentials can be kept and modified. Were it not for your example script other folks might have, understandably, done it like I did where I keep credentials in a .js file for the user database and MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD in docker-compose.yml (or k8s or whatever).

And of course simply adding your custom-user.sh code instead of in the /docker-entrypoint-initdb.d/ directory it would, of course, instead live alongside the rest in docker-entrypoint.sh.

lonix1 commented 5 years ago

I understand the reluctance to add functionality to working software. I also understand @yosifkit's response, and I'm using his advice in my own code.

But this functionality is obvious and necessary - as evidenced by all those who complained about it in #174 and StackOverflow. We assumed it to work the way we've described, because that is the obvious need.

One of the reasons we use docker is the promise of "just spinning up" a new container that is ready for action. Extra steps and config (that exists in multiple places, and liable to fall out of sync) are an extra burden, which are easily avoided by the (backwards compatible) proposed additions.

johnwyles commented 5 years ago

I have submitted PR https://github.com/docker-library/mongo/pull/331 as outlined by @yosifkit comments and code example as well taking into account his comments:

I'm mostly against adding more environment variables for creating a non-admin user, especially if it breaks backwards compatibility. We try to keep the images as close to upstream releases as possible with minimal maintenance and the entrypoint script is already very complex.

This does not break backwards compatibility and is a fully optional feature added for expected user behavior as itterated above by @lonix1 @johnwyles in this issue and numerous others in issue https://github.com/docker-library/mongo/issues/174#issuecomment-460566429.

Combine that with automated builds (https://docs.docker.com/docker-hub/builds/) and repository links (https://docs.docker.com/docker-hub/builds/#repository-links) and it's reasonably easy to have an up-to-date image built FROM mongo with the custom non-root user creation modifications.

While more mature software projects will certainly require /docker-entrypoint-initdb.d/* files the image as it is out of the box is unusable without an extra step of gaining root access and performing the user and password creation on the DB that /docker-entrypoint-initdb.d/ operates against. In fact without a stanza in these files that creates a username and password then users would be left with the default behavior to use the root role to perform all operations. To me this is bad security prescribed by the approach of not having a non-root username and password set. Most users at the start of their projects and when they are in less maturity and without security conscious concerns would simply use this root user which is bad practice and an anti-pattern to be using this level of access.

And then I will just echo all the points in: https://github.com/docker-library/mongo/issues/329#issuecomment-460902952

I will also put together a PR for the documentation update if we can proceed on this 👍

johnwyles commented 5 years ago

I have also added the documentation for this in PR https://github.com/docker-library/docs/pull/1422 in the documentation as well.

johnwyles commented 5 years ago

BUMP What did we think about PR https://github.com/docker-library/mongo/pull/331 and https://github.com/docker-library/docs/pull/1422 to provide users with expected behavior, backwards compatibility, and optional parameters but that when present provide the expectation the user requires for database initialization? Let me know if the PR needs a bit of modification but the discussions here seem to have all been addressed in the PR and documentation PR that provide comfortably merging this in.

prokhorovn commented 5 years ago

Actually, I've posted a comment in a different issue, but it seems to be very mongo initialization process related, so that will leave it here also #323

https://github.com/docker-library/mongo/issues/323#issuecomment-474243238

To summarize: there are problems with MONGO_NON_ROOT_USERNAME and MONGO_NON_ROOT_PASSWORD when applying this variables to mongod instances in configsvr mode. Detailed description could be found by link above

yosifkit commented 5 years ago

@npiskunov, on your issue it looks like we just need to account for the default dbpath change for configsvr (#341).

On a related note, this also means that the PR for this issue(#331) has another edge case. The non-root user values cannot work when running as a config server since it cannot create a user outside of admin (or config).

When running with this option, clients (i.e. other cluster components) cannot write data to any database other than config and admin.

- https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-configsvr

Finding more edge cases like this is why I am so hesitant to add more environment variables.

Admittedly this means that, apart from #331, we might want to note that limitation for /docker-entrypoint-initdb.d/ scripts as well as MONGO_INITDB_DATABASE, but I haven't seen an issue about this, so it seems like a rare case.

prokhorovn commented 5 years ago

@yosifkit Did I get right, that changes in #341 fix behavior of MONGO_NON_ROOT_USERNAME in configsvr by pointing user create on correct DB (admin)? In this case MONGO_INITDB_DATABASE becomes useless for configsvr mode... I believe it would be useful to mention this limitation in docs and maybe even in container logic, e.g. log message "MONGO_INITDB_DATABASE is not supported in config server mode"?

Regarding #341. In my opinion changes described (mongo initialization variables) become clearer, but still not fully clear :) It would be perfect if there was e.g. compose.yml illustrating minimalistic sharded cluster composition with security (3-members RS for shard, 3-members CSRS, single mongos. Root user / Security Key file / sample DB initial seed);

carloschneider commented 5 years ago

Why this example is not in the documentation?

#!/bin/bash
set -e;

# a default non-root role
MONGO_NON_ROOT_ROLE="${MONGO_NON_ROOT_ROLE:-readWrite}"

if [ -n "${MONGO_NON_ROOT_USERNAME:-}" ] && [ -n "${MONGO_NON_ROOT_PASSWORD:-}" ]; then
  "${mongo[@]}" "$MONGO_INITDB_DATABASE" <<-EOJS
      db.createUser({
          user: $(_js_escape "$MONGO_NON_ROOT_USERNAME"),
          pwd: $(_js_escape "$MONGO_NON_ROOT_PASSWORD"),
          roles: [ { role: $(_js_escape "$MONGO_NON_ROOT_ROLE"), db: $(_js_escape "$MONGO_INITDB_DATABASE") } ]
          })
  EOJS
else
  # print warning or kill temporary mongo and exit non-zero
fi
johnwyles commented 5 years ago

to echo @carloschneider and @lonix1 - @yosifkit I think there is an "expectation" that the legwork not be on the implementer or user of this to have some expectation to add a lot of custom work and have to wade through these github PR's and Issues to find out how to implement what should otherwise be straightforward behavior one would expect - I think pushing PR https://github.com/docker-library/docs/pull/1421/files (docs) (which was closed) but also something similar or the PR I originally filed on this: https://github.com/docker-library/mongo/pull/331/files will resolve a lot of headache you clearly can see causing people to pull their hair out only to find they need to write some custom code and implementation with what should otherwise be expected or "obvious" behavior - sorry for getting back to this so late as I have been caught up in a lot of other work which let this linger for longer than it should have

fullammo commented 4 years ago

This is exactly what my development environment needs!

I want to test my application's connection to a "production like" mongo instance with authentication and so on.

I'm able to do this with my local docker-compose utilizing the custom made .js or .sh files. But I'm unable to do so in our CI/CD pipeline, because I can't control this behavior through environment variables.

This feature could provide a centralized way to look up your mongo credentials and your application credentials in a clear and consistent manner in your docker-compose.yml or other configuration files.

Cheers for the efforts!

unstephenk commented 4 years ago

Can someone post an example of how to run the docker commands without --link and using --network? I can't seem to get my nodeJS app to connect to the mongo container.

NikolaiT commented 4 years ago

Why is a custom non root user still not a part of the default Dockerfile?

People need it and people end up implementing it themselves with custom-user.sh!

It could save so many hours of work if the maintainer would do it once and folks can just add some environment variables instead of creating an additional dockerfile + script...

wglambert commented 4 years ago

You can use the standard --user way of running as an arbitrary user

https://github.com/docker-library/mongo/issues/315#issuecomment-439981173

$ mkdir db && sudo chown 777 db

$ docker run --rm -dit -v $PWD/db:/data/db -v /etc/passwd:/etc/passwd:ro --user 1000:1001 --name mongo mongo
6c4a4e74b314a2a01893bc0b7405106db3100db37e9ad466772f56f2f88ec2b2

$ echo $UID
1000

$ docker exec -it mongo bash
groups: cannot find name for group ID 1001
rei@6c4a4e74b314:/$ echo $UID
1000
KenEucker commented 3 years ago

Bumping this because I find it a little frustrating that I'm blocked because I can't initialize a mongo container with a usable database + username/password pair.

user @lonix1 stated it better than I could:

One of the reasons we use docker is the promise of "just spinning up" a new container that is ready for action. Extra steps and config (that exists in multiple places, and liable to fall out of sync) are an extra burden, which are easily avoided by the (backwards compatible) proposed additions.

My use case is simple: A MERN application where a single docker-compose.yml sets up the requisite stack ahead of my node application starting. *note: my stack doesn't include bash, it is not named BMERN or MERNB -- it's MERN.

Can I connect to my mongodb container with a username and password on application start-up and begin using my target database without issue from a single docker-compose.yml file?

No, because environment variables don't cover this extremely basic configuration.

Has this been reconsidered since 2019, @yosifkit?

UPDATE: Went with bintami/node and never looked back. This feels like a failure to deliver on a community agreement but it's not a hill worth dying on.

CodePint commented 3 years ago

Can we please get some documentation added and add the init script to run as part of the entrypoint? I've just spent the last 45 minutes digging around trying to understand why my container was not creating my databases on startup.

johnwyles commented 3 years ago

@CodePint That is the point of the ticket so not sure if you were just echoing it, but you can see for help: Docs: https://github.com/docker-library/docs/pull/1422 Initialization: https://github.com/docker-library/mongo/pull/331

Other than getting the maintainers to merge those tickets you are going to be stuck where a lot of folks are on these two open PRs.

aeweda commented 3 years ago

I added the following to my init.sh script that basically runs the whole app

# ! update mongodb user for authentication
sleep 10
export $(sed 's/[[:blank:]]//g; /^#/d' .env | xargs)
docker exec -it mongodb bash -c "mongo $MONGO_DB \
         -u $MONGO_USERNAME \
         -p $MONGO_PASSWORD \
         --authenticationDatabase admin \
         --eval 'db.createUser({user: \"$MONGO_USERNAME\", pwd: \"$MONGO_PASSWORD\", roles:[{role:\"userAdminAnyDatabase\", db: \"admin\"}]});
                db.grantRolesToUser(\"$MONGO_USERNAME\",[{ role: \"root\", db: \"admin\" }]);'"

I'd like to leave it here in case some poor soul is in need of it

it was my first time using mongodb the other day and I went for docker straight away(big mistake). I spent the better part of 2 days trying to figure out a way around all this that didn't involve volume mounting or overriding the entry_point.sh file

it's not the cleanest solution but it does the job for now

openmindculture commented 3 years ago

Thanks everyone for your effort and documentation. I agree that it would be helpful to have an easy way to use docker to set up a local connection with default values without adding bash script files.

draeder commented 3 years ago

After spending all evening last night and this morning trying to figure out why these environment variables "were not working", I finally found this discussion. I would like to add my support for this feature. I'm considering moving away from this image and using Bitnami's image instead for this very reason.

CodePint commented 3 years ago

That is the point of the ticket so not sure if you were just echoing it, but you can see for help: Docs: docker-library/docs#1422 Initialization: #331

Other than getting the maintainers to merge those tickets you are going to be stuck where a lot of folks are on these two open PRs.

@johnwyles I was just just echoing and voicing my shared frustration.
Thanks for your work with these PR's, its a shame there is no movement towards getting them merged.

After spending all evening last night and this morning trying to figure out why these environment variables "were not working", I finally found this discussion. I would like to add my support for this feature.

@draeder Thanks for mentioning Bitnami's image where this issue has been solved and the solution clearly documented. This looks like a viable solution for those that are frustrated with the lack of movement and rigidity from the maintainers here.

I will be using their image in the future. As an aid to frustrated developers who will visit this page over the coming years, I've included the relevant info for the Bitnami MongoDB image below:

GitHub Repository: https://github.com/bitnami/bitnami-docker-mongodb

Running the docker image with "a user defined database, username, and password using environment variables":

# Creating a user and database on first run
# You can create a user with restricted access to a database while starting the container for the first time. 
# To do this, provide the MONGODB_USERNAME, MONGODB_PASSWORD and MONGODB_DATABASE environment variables.

$ docker run --name mongodb \
  -e MONGODB_USERNAME=my_user \
  -e MONGODB_PASSWORD=password123 \
  -e MONGODB_DATABASE=my_database bitnami/mongodb:latest
# or by modifying the docker-compose.yml file present in this repository:

services:
  mongodb:
  ...
    environment:
      - MONGODB_USERNAME=my_user
      - MONGODB_PASSWORD=password123
      - MONGODB_DATABASE=my_database
  ...
nettnikl commented 2 years ago

For everyone running into issues using the script by @carloschneider: The script is not copy/paste, as a noop is needed in the else (at least on my machine):

#!/bin/bash
set -e;

# a default non-root role
MONGO_NON_ROOT_ROLE="${MONGO_NON_ROOT_ROLE:-readWrite}"

if [ -n "${MONGO_NON_ROOT_USERNAME:-}" ] && [ -n "${MONGO_NON_ROOT_PASSWORD:-}" ]; then
    "${mongo[@]}" "$MONGO_INITDB_DATABASE" <<-EOJS
        db.createUser({
            user: $(_js_escape "$MONGO_NON_ROOT_USERNAME"),
            pwd: $(_js_escape "$MONGO_NON_ROOT_PASSWORD"),
            roles: [ { role: $(_js_escape "$MONGO_NON_ROOT_ROLE"), db: $(_js_escape "$MONGO_INITDB_DATABASE") } ]
            })
    EOJS
else
    true # print warning or kill temporary mongo and exit non-zero
fi
jdelforno commented 2 years ago

As someone who's spent the last day or more, trying to make this container work in Azure Container Instances / App Services / Container Apps, I'm laughing at how ignorant I am and how bad the documentation is.

I've inserted the required MONGO_INITDB_ROOT_USERNAME / MONGO_INITDB_ROOT_PASSWORD and MONGO_INITDB_DATABASE, got the container working with an Azure File Share and then spent hours debugging why the user and database hasn't been creating.

All of the files are present in the Azure File Share though, indicating that persistent storage will work.

Thank you to everyone for providing the missing steps. It seems there's no real way to have this container build out of terraform with the required .js files to actually create the user/db. It's a shame the scripts aren't included by default.

AfrazHussain commented 2 years ago

It's been more than 3 years and it's absolutely funny how such a simple thing is so misleading for many developers wanting to quickly spin up a db instance. The frustration is more about a simple set of environment variables not working as expected, and developers wasting a ton of time in diagnosing why the connection isn't going through.

marfer commented 1 year ago

It's been more than 3 years and it's absolutely funny how such a simple thing is so misleading for many developers wanting to quickly spin up a db instance. The frustration is more about a simple set of environment variables not working as expected, and developers wasting a ton of time in diagnosing why the connection isn't going through.

correct, just killed 3 hours until figure it out, then went here to write a comment to support this feature (bug), so that in next 3 year someone fix it, and my children would not be as angry as I am right now.

Khaaz commented 1 year ago

Please add this, it would be so much easier and logical to have a way to create the image with a default DB, User, Password. It's just adding unecessary work, that almost everyone will do, or else they will just have a poorly secured DB or use something else.

Thanks for mentionning bitnami image in the answers above!!

flpms commented 1 year ago

How much time and money could be saved with this issue and script are included in the docs or variables name be cleaner and logical.