search

testcontainers / testcontainers-java

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
https://testcontainers.org
MIT License
7.91k stars 1.62k forks source link

`MongoDbContainer` is not working with custom username, password and db #4695

Open abdheshkumar opened 2 years ago

abdheshkumar commented 2 years ago 
import org.testcontainers.containers.MongoDBContainer
import org.testcontainers.utility.DockerImageName
val mongoDBContainer = new MongoDBContainer(DockerImageName.parse(“mongo:4.0.10”))
.withEnv(“MONGO_INITDB_DATABASE”, “mytestdb”)
.withEnv(“MONGO_INITDB_ROOT_USERNAME”, “any-user”)
.withEnv(“MONGO_INITDB_ROOT_PASSWORD”, “whatever”)

mongoDBContainer.start()

mongoDBContainer.getLogs

Getting below error:

ERROR 24/11/2021 16:16:44.815 org.testcontainers.containers.MongoDBContainer A single node replica set was not initialized in a set timeout: 60 attempts
ERROR 24/11/2021 16:16:44.819 :whale: [mongo:4.0.10] Could not start container
org.testcontainers.containers.MongoDBContainer$ReplicaSetInitializationException: A single node replica set was not initialized in a set timeout: 60 attempts

Below are containers logs:

2021-11-24T10:46:44.009+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2021-11-24T10:46:44.012+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] MongoDB starting : pid=28 port=27017 dbpath=/data/db 64-bit host=9d088e8e9add
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] db version v4.0.10
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] git version: c389e7f69f637f7a1ac3cc9fae843b635f20b766
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] allocator: tcmalloc
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] modules: none
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] build environment:
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] distmod: ubuntu1604
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] distarch: x86_64
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] target_arch: x86_64
2021-11-24T10:46:44.017+0000 I CONTROL [initandlisten] options: { net: { bindIp: “127.0.0.1”, port: 27017, ssl: { mode: “disabled” } }, processManagement: { fork: true, pidFilePath: “/tmp/docker-entrypoint-temp-mongod.pid” }, systemLog: { destination: “file”, logAppend: true, path: “/proc/1/fd/1” } }
2021-11-24T10:46:44.018+0000 I STORAGE [initandlisten]
2021-11-24T10:46:44.018+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2021-11-24T10:46:44.018+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2021-11-24T10:46:44.018+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4470M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-11-24T10:46:44.482+0000 I STORAGE [initandlisten] WiredTiger message [1637750804:482274][28:0x7f2b33840a80], txn-recover: Set global recovery timestamp: 0
2021-11-24T10:46:44.490+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2021-11-24T10:46:44.504+0000 I CONTROL [initandlisten]
2021-11-24T10:46:44.504+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2021-11-24T10:46:44.504+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2021-11-24T10:46:44.504+0000 I CONTROL [initandlisten]
2021-11-24T10:46:44.505+0000 I STORAGE [initandlisten] createCollection: admin.system.version with provided UUID: 147d194f-30f2-47ec-8121-87068d509b1e
2021-11-24T10:46:44.514+0000 I COMMAND [initandlisten] setting featureCompatibilityVersion to 4.0
2021-11-24T10:46:44.518+0000 I STORAGE [initandlisten] createCollection: local.startup_log with generated UUID: 8dc76bdb-13dc-44c7-857c-365fb90a93de
2021-11-24T10:46:44.529+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘/data/db/diagnostic.data’
2021-11-24T10:46:44.531+0000 I NETWORK [initandlisten] waiting for connections on port 27017
2021-11-24T10:46:44.531+0000 I STORAGE [LogicalSessionCacheRefresh] createCollection: config.system.sessions with generated UUID: 385865bb-893f-407a-a5bc-26c3b8628135
child process started successfully, parent exiting
2021-11-24T10:46:44.547+0000 I INDEX [LogicalSessionCacheRefresh] build index on: config.system.sessions properties: { v: 2, key: { lastUse: 1 }, name: “lsidTTLIndex”, ns: “config.system.sessions”, expireAfterSeconds: 1800 }
2021-11-24T10:46:44.547+0000 I INDEX [LogicalSessionCacheRefresh] building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2021-11-24T10:46:44.549+0000 I INDEX [LogicalSessionCacheRefresh] build index done. scanned 0 total records. 0 secs
2021-11-24T10:46:44.583+0000 I NETWORK [listener] connection accepted from 127.0.0.1:46572 #1 (1 connection now open)
2021-11-24T10:46:44.587+0000 I NETWORK [conn1] received client metadata from 127.0.0.1:46572 conn1: { application: { name: “MongoDB Shell” }, driver: { name: “MongoDB Internal Client”, version: “4.0.10” }, os: { type: “Linux”, name: “Ubuntu”, architecture: “x86_64”, version: “16.04” } }
2021-11-24T10:46:44.591+0000 I NETWORK [conn1] end connection 127.0.0.1:46572 (0 connections now open)
2021-11-24T10:46:44.646+0000 I NETWORK [listener] connection accepted from 127.0.0.1:46574 #2 (1 connection now open)
2021-11-24T10:46:44.647+0000 I NETWORK [conn2] received client metadata from 127.0.0.1:46574 conn2: { application: { name: “MongoDB Shell” }, driver: { name: “MongoDB Internal Client”, version: “4.0.10” }, os: { type: “Linux”, name: “Ubuntu”, architecture: “x86_64”, version: “16.04” } }
2021-11-24T10:46:44.652+0000 I NETWORK [listener] connection accepted from 127.0.0.1:46576 #3 (2 connections now open)
2021-11-24T10:46:44.653+0000 I NETWORK [conn3] received client metadata from 127.0.0.1:46576 conn3: { application: { name: “MongoDB Shell” }, driver: { name: “MongoDB Internal Client”, version: “4.0.10” }, os: { type: “Linux”, name: “Ubuntu”, architecture: “x86_64”, version: “16.04” } }
2021-11-24T10:46:44.658+0000 I NETWORK [conn3] end connection 127.0.0.1:46576 (1 connection now open)
2021-11-24T10:46:44.691+0000 I STORAGE [conn2] createCollection: admin.system.users with generated UUID: 5e685272-8668-4ca7-bab6-c0f2e39c8767
Successfully added user: {
“user” : “any-user”,
“roles” : [
{
“role” : “root”,
“db” : “admin”
}
]
}
2021-11-24T10:46:44.706+0000 E - [main] Error saving history file: FileOpenFailed: Unable to open() file /home/mongodb/.dbshell: Unknown error
2021-11-24T10:46:44.708+0000 I NETWORK [conn2] end connection 127.0.0.1:46574 (0 connections now open)

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2021-11-24T10:46:44.738+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2021-11-24T10:46:44.740+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’
killing process with pid: 28
2021-11-24T10:46:44.742+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2021-11-24T10:46:44.742+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2021-11-24T10:46:44.742+0000 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2021-11-24T10:46:44.743+0000 I CONTROL [signalProcessingThread] Shutting down free monitoring
2021-11-24T10:46:44.743+0000 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2021-11-24T10:46:44.743+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2021-11-24T10:46:44.744+0000 I STORAGE [signalProcessingThread] Shutting down session sweeper thread
2021-11-24T10:46:44.745+0000 I STORAGE [signalProcessingThread] Finished shutting down session sweeper thread
2021-11-24T10:46:44.826+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2021-11-24T10:46:44.826+0000 I CONTROL [signalProcessingThread] now exiting
2021-11-24T10:46:44.826+0000 I CONTROL [signalProcessingThread] shutting down with code:0
MitchelNijdam-Rockstars commented 2 years ago

Facing the same problem. Any solution found @abdheshkumar ?

krisgerhard commented 2 years ago

At the end of mongo container log you'd see `MongoDB init process complete; ready for start up.

BadValue: security.keyFile is required when authorization is enabled with replica sets try 'mongod --help' for more information`

MongoDBContainer is using replica sets. Use GenericContainer instead.

MitchelNijdam-Rockstars commented 2 years ago

Thanks, I indeed solved it used GenericContainer;

public class CustomMongoDbContainer extends GenericContainer<CustomMongoDbContainer> {
  public CustomMongoDbContainer() {
    super(DockerImageName.parse("mongo:latest"));
    withEnv("MONGO_INITDB_ROOT_USERNAME", "someUser");
    withEnv("MONGO_INITDB_ROOT_PASSWORD", "somePassword");
    withEnv("MONGO_INITDB_DATABASE", "mongodb");
    withExposedPorts(27017);
  }
}
kiview commented 2 years ago

Thanks for sharing the solution @MitchelNijdam-Rockstars and @krisgerhard. So setting those ENV values just breaks the assumptions MongoDBContainer uses for interacting with the container.

What is the use case to set those values in the first place?

abdheshkumar commented 2 years ago

In my use case, I have an application that uses MongoDB connection with username/passwords internally so when I run integration tests then I have to pass the username/password with database name to run the application successfully.

kiview commented 2 years ago

Ok, thanks for the clarification. While I think the proposed workaround is decent for many users, I'd also say this a missing feature of our included MongoDbContainer (that should allow setting those values directly via methods).

Not sure how replica sets play into this story though.

ManuelTS commented 2 years ago

Upvote, this is highly critical for using and testing

kiview commented 2 years ago

Can you please elaborate on why this is highly critical if there exists a workaround that seems to be fine for many users? What are constraints that would not allow you to use a different username/password in your tests or inject it dynamically into your system during test execution or alternatively use GenericContainer?

ManuelTS commented 2 years ago

Sorry my wording was far off, I meant to say important.

  1. A workaround suggests brittle code
  2. Why shipping dysfunctional code? More users will run into this issue using MongoDBContainer in the first place
  3. Testing is most effective when close to production, auth is a complex issue and also doing it makes the own code more robust
  4. The code is more readable and clear without injection and the workaround
abdheshkumar commented 2 years ago

It's just a request, It would be nice if we could fix this issue with MongoDbContainer otherwise whoever wants to connect to mongo testcontrainer using credentials then there is no use of MongoDbContainer

ykrautsou commented 2 years ago

@abdheshkumar , I faced the same problem and found 2 solutions:

  1. Use GenericContainer as mentioned before: GenericContainer mongoDBContainer = new GenericContainer(MONGODB_IMAGE) .withExposedPorts(MONGODB_PORT) .withEnv("MONGO_INITDB_ROOT_USERNAME", "testuser") .withEnv ("MONGO_INITDB_ROOT_PASSWORD", "testpassword") .withEnv("MONGO_INITDB_DATABASE", "testdb")

OR

  1. Use MongoDbContainer with 'auth' param (you have to authenticate each time when run container): MongoDBContainer mongoDBContainer = new MongoDBContainer(MONGODB_IMAGE) .withExposedPorts(MONGODB_PORT) .withEnv("MONGO_INITDB_ROOT_USERNAME", "testuser") .withEnv ("MONGO_INITDB_ROOT_PASSWORD", "testpassword") .withEnv("MONGO_INITDB_DATABASE", "testdb") .withCommand("--auth");
abdheshkumar commented 2 years ago

@abdheshkumar , I faced the same problem and found 2 solutions:

  1. Use GenericContainer as mentioned before: GenericContainer mongoDBContainer = new GenericContainer(MONGODB_IMAGE) .withExposedPorts(MONGODB_PORT) .withEnv("MONGO_INITDB_ROOT_USERNAME", "testuser") .withEnv ("MONGO_INITDB_ROOT_PASSWORD", "testpassword") .withEnv("MONGO_INITDB_DATABASE", "testdb")

OR 2. Use MongoDbContainer with 'auth' param (you have to authenticate each time when run container): MongoDBContainer mongoDBContainer = new MongoDBContainer(MONGODB_IMAGE) .withExposedPorts(MONGODB_PORT) .withEnv("MONGO_INITDB_ROOT_USERNAME", "testuser") .withEnv ("MONGO_INITDB_ROOT_PASSWORD", "testpassword") .withEnv("MONGO_INITDB_DATABASE", "testdb") .withCommand("--auth");

@ykrautsou Lovely solutions, thank you

abdheshkumar commented 2 years ago

@ykrautsou I think option 2 is not working.

Exception in thread "main" org.testcontainers.containers.ContainerLaunchException: Container startup failed
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:330)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)

Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311) 
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:497)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    ... 46 more
Caused by: org.testcontainers.containers.MongoDBContainer$ReplicaSetInitializationException: A single node replica set was not initialized in a set timeout: 60 attempts
    at org.testcontainers.containers.MongoDBContainer.checkMongoNodeExitCodeAfterWaiting(MongoDBContainer.java:119)
    at org.testcontainers.containers.MongoDBContainer.initReplicaSet(MongoDBContainer.java:141)
    at org.testcontainers.containers.MongoDBContainer.containerIsStarted(MongoDBContainer.java:80)
    at org.testcontainers.containers.GenericContainer.containerIsStarted(GenericContainer.java:659)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:476)
    ... 48 more
kiview commented 2 years ago

@abdheshkumar We need more logs (especially the container logs) to understand what is happening here.

@ManuelTS @abdheshkumar We are open for PRs adding this feature. Since all interactions with MongoDB in MongoDBContainer are done via command execution inside the container, this is probably only a question of calling the MongoDB commands with the correct authentication parameters.

abdheshkumar commented 2 years ago

@kiview Here is complete logs

2022-02-20T10:07:03.074+0000 I  CONTROL  [main] ***** SERVER RESTARTED *****
2022-02-20T10:07:03.076+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2022-02-20T10:07:03.083+0000 W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=29 port=27017 dbpath=/data/db 64-bit host=c89e304328d2
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] db version v4.2.12
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] git version: 5593fd8e33b60c75802edab304e23998fa0ce8a5
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] modules: none
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] build environment:
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2022-02-20T10:07:03.084+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1", port: 27017, tls: { mode: "disabled" } }, processManagement: { fork: true, pidFilePath: "/tmp/docker-entrypoint-temp-mongod.pid" }, systemLog: { destination: "file", logAppend: true, path: "/proc/1/fd/1" } }
2022-02-20T10:07:03.085+0000 I  STORAGE  [initandlisten] 
2022-02-20T10:07:03.085+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2022-02-20T10:07:03.085+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2022-02-20T10:07:03.086+0000 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4470M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2022-02-20T10:07:03.548+0000 I  STORAGE  [initandlisten] WiredTiger message [1645351623:548820][29:0x7ff31c23fb00], txn-recover: Set global recovery timestamp: (0, 0)
2022-02-20T10:07:03.559+0000 I  RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2022-02-20T10:07:03.569+0000 I  STORAGE  [initandlisten] No table logging settings modifications are required for existing WiredTiger tables. Logging enabled? 1
2022-02-20T10:07:03.570+0000 I  STORAGE  [initandlisten] Timestamp monitor starting
2022-02-20T10:07:03.575+0000 I  CONTROL  [initandlisten] 
2022-02-20T10:07:03.575+0000 I  CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2022-02-20T10:07:03.575+0000 I  CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2022-02-20T10:07:03.575+0000 I  CONTROL  [initandlisten] 
2022-02-20T10:07:03.576+0000 I  STORAGE  [initandlisten] createCollection: admin.system.version with provided UUID: 2499922c-6bf0-4ffd-8c83-88d585e70a78 and options: { uuid: UUID("2499922c-6bf0-4ffd-8c83-88d585e70a78") }
2022-02-20T10:07:03.586+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns admin.system.version
2022-02-20T10:07:03.587+0000 I  SHARDING [initandlisten] Marking collection admin.system.version as collection version: <unsharded>
2022-02-20T10:07:03.587+0000 I  COMMAND  [initandlisten] setting featureCompatibilityVersion to 4.2
2022-02-20T10:07:03.587+0000 I  SHARDING [initandlisten] Marking collection local.system.replset as collection version: <unsharded>
2022-02-20T10:07:03.587+0000 I  STORAGE  [initandlisten] Flow Control is enabled on this deployment.
2022-02-20T10:07:03.587+0000 I  SHARDING [initandlisten] Marking collection admin.system.roles as collection version: <unsharded>
2022-02-20T10:07:03.588+0000 I  STORAGE  [initandlisten] createCollection: local.startup_log with generated UUID: ef8534ab-deaf-4f90-847f-24c35668e224 and options: { capped: true, size: 10485760 }
2022-02-20T10:07:03.598+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns local.startup_log
2022-02-20T10:07:03.598+0000 I  SHARDING [initandlisten] Marking collection local.startup_log as collection version: <unsharded>
2022-02-20T10:07:03.601+0000 I  FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2022-02-20T10:07:03.604+0000 I  SHARDING [LogicalSessionCacheRefresh] Marking collection config.system.sessions as collection version: <unsharded>
2022-02-20T10:07:03.604+0000 I  CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist
2022-02-20T10:07:03.604+0000 I  NETWORK  [listener] Listening on /tmp/mongodb-27017.sock
2022-02-20T10:07:03.604+0000 I  NETWORK  [listener] Listening on 127.0.0.1
2022-02-20T10:07:03.604+0000 I  NETWORK  [listener] waiting for connections on port 27017
2022-02-20T10:07:03.605+0000 I  STORAGE  [LogicalSessionCacheRefresh] createCollection: config.system.sessions with provided UUID: 0e28b4ec-dd4e-4786-b6a5-609f08955e7a and options: { uuid: UUID("0e28b4ec-dd4e-4786-b6a5-609f08955e7a") }
child process started successfully, parent exiting
2022-02-20T10:07:03.619+0000 I  INDEX    [LogicalSessionCacheRefresh] index build: done building index _id_ on ns config.system.sessions
2022-02-20T10:07:03.632+0000 I  INDEX    [LogicalSessionCacheRefresh] index build: starting on config.system.sessions properties: { v: 2, key: { lastUse: 1 }, name: "lsidTTLIndex", ns: "config.system.sessions", expireAfterSeconds: 1800 } using method: Hybrid
2022-02-20T10:07:03.632+0000 I  INDEX    [LogicalSessionCacheRefresh] build may temporarily use up to 200 megabytes of RAM
2022-02-20T10:07:03.633+0000 I  INDEX    [LogicalSessionCacheRefresh] index build: collection scan done. scanned 0 total records in 0 seconds
2022-02-20T10:07:03.637+0000 I  INDEX    [LogicalSessionCacheRefresh] index build: inserted 0 keys from external sorter into index in 0 seconds
2022-02-20T10:07:03.640+0000 I  INDEX    [LogicalSessionCacheRefresh] index build: done building index lsidTTLIndex on ns config.system.sessions
2022-02-20T10:07:03.674+0000 I  NETWORK  [listener] connection accepted from 127.0.0.1:35058 #1 (1 connection now open)
2022-02-20T10:07:03.682+0000 I  NETWORK  [conn1] received client metadata from 127.0.0.1:35058 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.12" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2022-02-20T10:07:03.689+0000 I  NETWORK  [conn1] end connection 127.0.0.1:35058 (0 connections now open)
2022-02-20T10:07:03.753+0000 I  NETWORK  [listener] connection accepted from 127.0.0.1:35060 #2 (1 connection now open)
2022-02-20T10:07:03.753+0000 I  NETWORK  [conn2] received client metadata from 127.0.0.1:35060 conn2: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.12" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2022-02-20T10:07:03.761+0000 I  NETWORK  [listener] connection accepted from 127.0.0.1:35062 #3 (2 connections now open)
2022-02-20T10:07:03.761+0000 I  NETWORK  [conn3] received client metadata from 127.0.0.1:35062 conn3: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.12" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2022-02-20T10:07:03.771+0000 I  NETWORK  [conn3] end connection 127.0.0.1:35062 (1 connection now open)
2022-02-20T10:07:03.803+0000 I  SHARDING [conn2] Marking collection admin.system.users as collection version: <unsharded>
2022-02-20T10:07:03.803+0000 I  STORAGE  [conn2] createCollection: admin.system.users with generated UUID: 5b316663-b0b8-4269-8d1f-db607dd092d9 and options: {}
2022-02-20T10:07:03.815+0000 I  INDEX    [conn2] index build: done building index _id_ on ns admin.system.users
2022-02-20T10:07:03.821+0000 I  INDEX    [conn2] index build: done building index user_1_db_1 on ns admin.system.users
Successfully added user: {
    "user" : "testuser",
    "roles" : [
        {
            "role" : "root",
            "db" : "admin"
        }
    ]
}
2022-02-20T10:07:03.828+0000 E  -        [main] Error saving history file: FileOpenFailed: Unable to open() file /home/mongodb/.dbshell: No such file or directory
2022-02-20T10:07:03.834+0000 I  NETWORK  [conn2] end connection 127.0.0.1:35060 (0 connections now open)

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2022-02-20T10:07:03.863+0000 I  CONTROL  [main] ***** SERVER RESTARTED *****
2022-02-20T10:07:03.866+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2022-02-20T10:07:03.870+0000 W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
killing process with pid: 29
2022-02-20T10:07:03.871+0000 I  CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2022-02-20T10:07:03.871+0000 I  REPL     [signalProcessingThread] Stepping down the ReplicationCoordinator for shutdown, waitTime: 10000ms
2022-02-20T10:07:03.871+0000 I  SHARDING [signalProcessingThread] Shutting down the WaitForMajorityService
2022-02-20T10:07:03.872+0000 I  CONTROL  [signalProcessingThread] Shutting down the LogicalSessionCache
2022-02-20T10:07:03.873+0000 I  NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2022-02-20T10:07:03.874+0000 I  NETWORK  [listener] removing socket file: /tmp/mongodb-27017.sock
2022-02-20T10:07:03.874+0000 I  NETWORK  [signalProcessingThread] Shutting down the global connection pool
2022-02-20T10:07:03.874+0000 I  STORAGE  [signalProcessingThread] Shutting down the FlowControlTicketholder
2022-02-20T10:07:03.874+0000 I  -        [signalProcessingThread] Stopping further Flow Control ticket acquisitions.
2022-02-20T10:07:03.874+0000 I  STORAGE  [signalProcessingThread] Shutting down the PeriodicThreadToAbortExpiredTransactions
2022-02-20T10:07:03.874+0000 I  STORAGE  [signalProcessingThread] Shutting down the PeriodicThreadToDecreaseSnapshotHistoryIfNotNeeded
2022-02-20T10:07:03.875+0000 I  REPL     [signalProcessingThread] Shutting down the ReplicationCoordinator
2022-02-20T10:07:03.875+0000 I  SHARDING [signalProcessingThread] Shutting down the ShardingInitializationMongoD
2022-02-20T10:07:03.875+0000 I  REPL     [signalProcessingThread] Enqueuing the ReplicationStateTransitionLock for shutdown
2022-02-20T10:07:03.875+0000 I  -        [signalProcessingThread] Killing all operations for shutdown
2022-02-20T10:07:03.875+0000 I  COMMAND  [signalProcessingThread] Shutting down all open transactions
2022-02-20T10:07:03.875+0000 I  REPL     [signalProcessingThread] Acquiring the ReplicationStateTransitionLock for shutdown
2022-02-20T10:07:03.875+0000 I  INDEX    [signalProcessingThread] Shutting down the IndexBuildsCoordinator
2022-02-20T10:07:03.875+0000 I  NETWORK  [signalProcessingThread] Shutting down the ReplicaSetMonitor
2022-02-20T10:07:03.875+0000 I  CONTROL  [signalProcessingThread] Shutting down free monitoring
2022-02-20T10:07:03.875+0000 I  CONTROL  [signalProcessingThread] Shutting down free monitoring
2022-02-20T10:07:03.876+0000 I  FTDC     [signalProcessingThread] Shutting down full-time data capture
2022-02-20T10:07:03.876+0000 I  FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2022-02-20T10:07:03.876+0000 I  STORAGE  [signalProcessingThread] Shutting down the HealthLog
2022-02-20T10:07:03.877+0000 I  STORAGE  [signalProcessingThread] Shutting down the storage engine
2022-02-20T10:07:03.877+0000 I  STORAGE  [signalProcessingThread] Deregistering all the collections
2022-02-20T10:07:03.877+0000 I  STORAGE  [signalProcessingThread] Timestamp monitor shutting down
2022-02-20T10:07:03.877+0000 I  STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2022-02-20T10:07:03.878+0000 I  STORAGE  [signalProcessingThread] Shutting down session sweeper thread
2022-02-20T10:07:03.878+0000 I  STORAGE  [signalProcessingThread] Finished shutting down session sweeper thread
2022-02-20T10:07:03.878+0000 I  STORAGE  [signalProcessingThread] Shutting down journal flusher thread
2022-02-20T10:07:03.961+0000 I  STORAGE  [signalProcessingThread] Finished shutting down journal flusher thread
2022-02-20T10:07:03.961+0000 I  STORAGE  [signalProcessingThread] Shutting down checkpoint thread
2022-02-20T10:07:03.961+0000 I  STORAGE  [signalProcessingThread] Finished shutting down checkpoint thread
2022-02-20T10:07:03.995+0000 I  STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2022-02-20T10:07:03.995+0000 I  -        [signalProcessingThread] Dropping the scope cache for shutdown
2022-02-20T10:07:03.995+0000 I  CONTROL  [signalProcessingThread] now exiting
2022-02-20T10:07:03.995+0000 I  CONTROL  [signalProcessingThread] shutting down with code:0
ykrautsou commented 2 years ago

@abdheshkumar , I catched 'ReplicaSetInitializationException' too and for now do not completly understand the root cause. I included retry mechanism during mongo test container start:

new RetryTemplateBuilder()
          .maxAttempts(3)
          .retryOn(ContainerLaunchException.class)
          .build()
          .execute(retryContext -> {
            log.debug("Starting MongoDB container attempt {}", retryContext.getRetryCount() + 1);
            mongoDBContainer.start();
            return null;
          });
kiview commented 2 years ago

This mongo command is likely failing once a password is set. I can only assume the script needs to authenticate before running any commands (see https://docs.mongodb.com/manual/reference/method/db.auth/), but since I have no experience with MongoDB, it is just a guess.

However, I did not get what was the problem with using GenericContainer @abdheshkumar as provided by @ykrautsou?

abdheshkumar commented 2 years ago

@kiview I don't have any problem using GenericContainer I am just wondering that would be the use of MongoDBContainer. Is it something like If someone wants to connect with mongo without credentials then they should use MongoDBContainer if someone wants to connect with mongo with credentials then they should use GenericContainer?

abdheshkumar commented 2 years ago

@kiview Another thing is, we have written a small wrapper on top of MongoDBContainer which is widely used in my company. If we want to use GenericContainer then I have to remove the wrapper on testcontainer's MongoDBContainer then we will have to create mongo container ourselves using generic container as mentioned in the previous comments then what is the use of testcontainer's MongoDBContainer that's only my concern.

kiview commented 2 years ago

In this case, providing a PR to extend the MongoDBContainer to support username and password might be the best approach.

krisgerhard commented 2 years ago

Can anybody guess why should MongoDBContainer use replication sets? If not let's remove it and keep the container wrapper as simple as possible.

Tagging original author for some insight maybe ;) @silaev

silaev commented 2 years ago

Hey @krisgerhard, a replica set is needed for some operations like running MongoDB transactions. The whole idea of the MongoDBContainer is to automatically init a replica set to provide full testing experience. If you don't need such experience then you can use a GenericContainer instead

krisgerhard commented 2 years ago

Okay, understood. The issue seems to be for average user this is not explicit enough. He wants to use testcontainers for mongodb, sees that there a specific module for it and uses it.

I see many different paths this issue could take

silaev commented 2 years ago

Thanks for your interest in MongoDB in general and the MongoDBContainer in particular. The MongoDB official docs state which operations require a replica set. MongoDB evolves quite rapidly these days. I'm not sure that complement documentation of the MongoDBContainer every time when MongoDB announces a new replica set feature is a good idea. Please, note that the idea of the MongoDBContainer is to cover most test cases without an end user thinking whether or not an operation requires a replica set. Regarding the MongoDBContainer doc, it explains in detail it's motivation (test MongoDB transactions) and some difficulties in initialisation a replica set manually

emincanoguz11 commented 2 years ago

Hi everyone. I try this methods but I have a problem with versions. For example if I try mongo 3.2.4 version this is work but if I try mongo 4.4 this isn't work. I take a difference errors. (I use Mongo 4.4)

First try : I try 4.4.13-focal version and I take A single node replica set was not initialized in a set timeout: 60 attempts error.

Second try: I try default image of test Container (4.0.10) this isn't work and I get this following error.

network error while attempting to run command 'isMaster' on host '127.0.0.1:27017' : connect@src/mongo/shell/mongo.js:344:17 Third try: I try latest version and its works.

Fourth try: I try 4.2.19 version and I take DBClientConnection failed to receive message from 127.0.0.1:27017 - HostUnreachable: Connection reset by peer error.

Fifth try: I try 5.0 version and its works.

So that this problem is about mongo's 4.x.x versions. Can we check this situation? Also if we try resolve this problem I want to help us.

Also I send my pseudo code.

public static final long memoryInBytes = 64 * 1024 * 1024;
public static GenericContainer<?> mongo = new MongoDBContainer(DockerImageName.parse("mongo:latest"))
            .withEnv("MONGO_INITDB_ROOT_USERNAME", "mongoadministrator")
            .withEnv("MONGO_INITDB_ROOT_PASSWORD", "secret")
            .withEnv("MONGO_INITDB_DATABASE", "test")
            .withSharedMemorySize(memoryInBytes)
            .withExposedPorts(MONGO_PORT)
            .waitingFor(Wait.forLogMessage("(?i).*Waiting for connections*.*", 1));
Thanks for helping :)

Best regards

Update: I changed code for this style and its works.

public static final long memoryInBytes = 64 * 1024 * 1024; 
public static GenericContainer<?> mongo = new GenericContainer(DockerImageName.parse("mongo:latest")) .withEnv("MONGO_INITDB_ROOT_USERNAME", "mongoadministrator")
 .withEnv("MONGO_INITDB_ROOT_PASSWORD", "secret") 
 .withEnv("MONGO_INITDB_DATABASE", "test") 
 .withSharedMemorySize(memoryInBytes) 
 .withExposedPorts(MONGO_PORT) 
 .waitingFor(Wait.forLogMessage("(?i).*Waiting for connections*.*", 1));
sergey-morenets commented 2 years ago

Hi @emincanoguz11

Thank you for the solution. But If I used it I still got an error in my integration tests:

Caused by: org.testcontainers.containers.MongoDBContainer$ReplicaSetInitializationException: A single node replica set was not initialized in a set timeout: 60 attempts
    at org.testcontainers.containers.MongoDBContainer.checkMongoNodeExitCodeAfterWaiting(MongoDBContainer.java:119)
    at org.testcontainers.containers.MongoDBContainer.initReplicaSet(MongoDBContainer.java:141)
    at org.testcontainers.containers.MongoDBContainer.containerIsStarted(MongoDBContainer.java:80)
    at org.testcontainers.containers.GenericContainer.containerIsStarted(GenericContainer.java:688)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:504)
    ... 70 more
@MicronautTest
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@Testcontainers
public abstract class BaseMongoTest {
    static final MongoDBContainer mongo;

    static final int MONGO_LOCAL_PORT;

    static {
        mongo = new MongoDBContainer("mongo:4.4").withEnv("MONGO_INITDB_ROOT_USERNAME", "test")
                .withEnv("MONGO_INITDB_ROOT_PASSWORD", "test")
                 .waitingFor(Wait.forLogMessage("(?i).*Waiting for connections*.*", 1));
        mongo.start();
        MONGO_LOCAL_PORT = mongo.getMappedPort(27017);
    }
emincanoguz11 commented 2 years ago

Hi Sergey.

You use MongoDBContainer but MongoDBContainer is not running to 4.x.x versions of Mongo. You should change MongoDBContainer to GenericContainer()

P.S: My bad. I wrongly edited my comment. Sorry about that Sergey.

silaev commented 1 year ago

Raised a PR to add auth to MongoDBContainer. @kiview pls, review

cieplakm commented 1 year ago

Maybe stupid question, but what is the default username and password for MongoDBContainer?

eddumelendez commented 1 year ago

I think most of you already figure it out but would like to summarize and provide more context about mongodb auth. MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD env vars are used with an standalone instance (no ReplicaSet), which doesn't work with MongoDBContainer because of the checks running against ReplicaSet. As mongodb docs state Authentication in MongoDB is fairly complex,. There is a PR provided by @silaev (thank you!) in order to enable authentication against a ReplicaSet. I would like to guess that most of you are interested in ReplicaSet due to is the way to go to prod instead of standalone mode.

I have created a poll in order to know about needs in the community. Looking forward to see the results.

visualscrapy commented 9 months ago

Issue Description:

I encountered an issue while trying to use MongoDBContainer with custom username, password, and database in my project. It seems that the provided configuration is not being picked up, and I couldn't connect to the MongoDB instance as expected(custom_configuration).

Solution:

I found a solution to resolve this issue. You need to set the necessary environment variables to configure the MongoDBContainer properly. Here are the steps to resolve the issue:

  1. Create a .env file in your project directory if it doesn't already exist.
  2. Add the following lines to the .env file and replace desire_username, desire_password, and desire_db with your desired values:
MONGO_INITDB_ROOT_USERNAME=desire_username
MONGO_INITDB_ROOT_PASSWORD=desire_password
MONGO_DB=desire_db
  1. Save the .env file.
  2. In your Python code, when initializing the MongoDBContainer, ensure that you use the custom configuration. You can do this as follows:
from testcontainers.mongo import MongoContainer

with MongoDbContainer("mongo:latest").with_bind_ports(27017, 27017) as mongo_container:
    details = mongo_container.env
  1. Now, when you access the details variable, you should see your updated configuration details.

This should allow you to use custom username, password, and database with MongoDBContainer successfully. Make sure to adjust the values in the .env file to match your specific requirements.

fan9704 commented 3 months ago

Same has this problem in Java