search

brianfrankcooper / YCSB

Yahoo! Cloud Serving Benchmark
Apache License 2.0
4.96k stars 2.25k forks source link

MongoDB cannot connect to cluster database [SOLVED] #1423

Closed datnod closed 4 years ago

datnod commented 4 years ago

This is what I've tried:

$ ./bin/ycsb load mongodb -s -P workloads/workloada -p mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-0/ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb Command line: -db site.ycsb.db.MongoDbClient -s -P workloads/workloada -p mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-0/ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb -load YCSB Client 0.17.0

Loading workload... Starting test. 2020-04-30 07:31:25:108 0 sec: 0 operations; est completion in 0 second mongo client connection created with mongodb://ycsb:ycsb@ycsb-shard-0/ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb DBWrapper: report latency for each error is false and specific error codes to track for latency are: [] 2020-04-30 07:31:35:078 10 sec: 0 operations; est completion in 106751991167300 days 15 hours 2020-04-30 07:31:45:077 20 sec: 0 operations; est completion in 106751991167300 days 15 hours 2020-04-30 07:31:55:079 30 sec: 0 operations; est completion in 106751991167300 days 15 hours Exception while trying bulk insert with 0 com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=ycsb-shard-0:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: ycsb-shard-0}, caused by {java.net.UnknownHostException: ycsb-shard-0}}] at com.mongodb.internal.connection.BaseCluster.getDescription(BaseCluster.java:179) at com.mongodb.internal.connection.SingleServerCluster.getDescription(SingleServerCluster.java:41) at com.mongodb.client.internal.MongoClientDelegate.getConnectedClusterDescription(MongoClientDelegate.java:136) at com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:94) at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.getClientSession(MongoClientDelegate.java:249) at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:190) at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:960) at com.mongodb.client.internal.MongoCollectionImpl.executeInsertOne(MongoCollectionImpl.java:494) at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:478) at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:472) at site.ycsb.db.MongoDbClient.insert(MongoDbClient.java:270) at site.ycsb.DBWrapper.insert(DBWrapper.java:221) at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:601) at site.ycsb.ClientThread.run(ClientThread.java:135) at java.lang.Thread.run(Thread.java:748) Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 2020-04-30 07:31:56:708 31 sec: 0 operations; est completion in 106751991167300 days 15 hours [CLEANUP: Count=1, Max=3819, Min=3818, Avg=3819, 90=3819, 99=3819, 99.9=3819, 99.99=3819] [INSERT: Count=0, Max=0, Min=9223372036854775807, Avg=?, 90=0, 99=0, 99.9=0, 99.99=0] [INSERT-FAILED: Count=1, Max=30031871, Min=30015488, Avg=30023680, 90=30031871, 99=30031871, 99.9=30031871, 99.99=30031871] [OVERALL], RunTime(ms), 31632 [OVERALL], Throughput(ops/sec), 0.0 [TOTAL_GCS_PS_Scavenge], Count, 1 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 28 [TOTAL_GCTIME%_PS_Scavenge], Time(%), 0.0885179564997471 [TOTAL_GCS_PS_MarkSweep], Count, 0 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0 [TOTAL_GCTIME%_PS_MarkSweep], Time(%), 0.0 [TOTAL_GCs], Count, 1 [TOTAL_GC_TIME], Time(ms), 28 [TOTAL_GCTIME%], Time(%), 0.0885179564997471 [CLEANUP], Operations, 1 [CLEANUP], AverageLatency(us), 3819.0 [CLEANUP], MinLatency(us), 3818 [CLEANUP], MaxLatency(us), 3819 [CLEANUP], 95thPercentileLatency(us), 3819 [CLEANUP], 99thPercentileLatency(us), 3819 [INSERT], Operations, 0 [INSERT], AverageLatency(us), NaN [INSERT], MinLatency(us), 9223372036854775807 [INSERT], MaxLatency(us), 0 [INSERT], 95thPercentileLatency(us), 0 [INSERT], 99thPercentileLatency(us), 0 [INSERT], Return=ERROR, 1 [INSERT-FAILED], Operations, 1 [INSERT-FAILED], AverageLatency(us), 3.002368E7 [INSERT-FAILED], MinLatency(us), 30015488 [INSERT-FAILED], MaxLatency(us), 30031871 [INSERT-FAILED], 95thPercentileLatency(us), 30031871 [INSERT-FAILED], 99thPercentileLatency(us), 30031871 C:\Program Files\Java\jdk1.8.0_144\bin\java -cp C:\ycsb-0.17.0\mongodb-binding\conf;C:\ycsb-0.17.0\conf;C:\ycsb-0.17.0\lib\core-0.17.0.jar;C:\ycsb-0.17.0\lib\HdrHistogram-2.1.4.jar;C:\ycsb-0.17.0\lib\htrace-core4-4.1.0-incubating.jar;C:\ycsb-0.17.0\lib\jackson-core-asl-1.9.4.jar;C:\ycsb-0.17.0\lib\jackson-mapper-asl-1.9.4.jar;C:\ycsb-0.17.0\mongodb-binding\lib\logback-classic-1.1.2.jar;C:\ycsb-0.17.0\mongodb-binding\lib\logback-core-1.1.2.jar;C:\ycsb-0.17.0\mongodb-binding\lib\mongo-java-driver-3.8.0.jar;C:\ycsb-0.17.0\mongodb-binding\lib\mongodb-async-driver-2.0.1.jar;C:\ycsb-0.17.0\mongodb-binding\lib\mongodb-binding-0.17.0.jar;C:\ycsb-0.17.0\mongodb-binding\lib\slf4j-api-1.7.25.jar;C:\ycsb-0.17.0\mongodb-binding\lib\snappy-java-1.1.7.1.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P workloads/workloada -p mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-0/ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb -load

It worked fine for doing things locally but it does not seem to work for me through the cloud, there is probably something i've done wrong but I cannot figure it despite reading documentation on how to use the string connection.

allanbank commented 4 years ago

I think your URL is a little off. You have:

mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-0/ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb

I think you want:

mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb

You had an additional ycsb-shard-0/ before the real hostnames and the driver interpreted that as specifying the hostname ycsb-shard-0 with a database name of ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb.

datnod commented 4 years ago

I think your URL is a little off. You have:

mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-0/ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb

I think you want:

mongodb.url=mongodb://ycsb:ycsb@ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb

You had an additional ycsb-shard-0/ before the real hostnames and the driver interpreted that as specifying the hostname ycsb-shard-0 with a database name of ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb.

Thanks for the kind help. I tried this but it didn't work. I also made sure to have whitelisted all ips from mongodb cluster settings. And just to be on the safe side I turned off my firewall but the issue still persists. I can't seem to make it work. And I don't feel it is right to ask for more help than this, will be sufficient with localhost for benchmarking I guess for my thesis since I never said that the research questions would focus on cloud.

datnod commented 4 years ago

Just wanted to say I made it work by adding in mongodb.url="mongodb://user:pass@shardhosts/dbname?options" in options I added this: ssl=true&replicaSet=ycsb-shard-0&authSource=admin&retryWrites=true&w=majority

Thanks for earlier help, @allanbank.

ksingh7 commented 4 years ago

@datnod can you share the exact command that worked for you. I tried your method but could not get it working on mongodb atlas cluster. I am trying this but this does not work

./bin/ycsb load mongodb -s -P workloads/workloada -s -p mongodb.url="mongodb://iuser:pwd@mongodb-prod-shard-0/mongodb-prod-shard-00-00.23onhq7.mongodb.net:27017/ycsb?ssl=true&authSource=admin&retryWrites=true&w=majority"

still getting error

Exception while trying bulk insert with 0
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=mongodb-prod-shard-0:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-prod-shard-0}, caused by {java.net.UnknownHostException: mongodb-prod-shard-0}}]
    at com.mongodb.internal.connection.BaseCluster.getDescription(BaseCluster.java:179)
    at com.mongodb.internal.connection.SingleServerCluster.getDescription(SingleServerCluster.java:41)
    at com.mongodb.client.internal.MongoClientDelegate.getConnectedClusterDescription(MongoClientDelegate.java:136)
    at com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:94)
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.getClientSession(MongoClientDelegate.java:249)
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:190)
    at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:960)
    at com.mongodb.client.internal.MongoCollectionImpl.executeInsertOne(MongoCollectionImpl.java:494)
    at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:478)
    at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:472)
    at site.ycsb.db.MongoDbClient.insert(MongoDbClient.java:270)
    at site.ycsb.DBWrapper.insert(DBWrapper.java:221)
    at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:601)
    at site.ycsb.ClientThread.run(ClientThread.java:135)
    at java.lang.Thread.run(Thread.java:748)
Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0
2020-08-09 03:17:52:188 30 sec: 0 operations; est completion in 106751991167300 days 15 hours [CLEANUP: Count=1, Max=1958, Min=1958, Avg=1958, 90=1958, 99=1958, 99.9=1958, 99.99=1958] [INSERT: Count=0, Max=0, Min=9223372036854775807, Avg=�, 90=0, 99=0, 99.9=0, 99.99=0] [INSERT-FAILED: Count=1, Max=30031871, Min=30015488, Avg=30023680, 90=30031871, 99=30031871, 99.9=30031871, 99.99=30031871]
rafbel commented 4 years ago

Hey @ksingh7, were you able to solve this issue? I am getting the same thing.

morcelicaio commented 2 years ago

Hi, @datnod , @allanbank and @rafbel

I'm trying to run ycsb on mongodb atlas cluster but not having success.

I'm using the following connection string in the 'mongodb.url' parameter:

-p mongodb.url=mongodb+srv://clusterycsb.oyia8.mongodb.net/ycsb

And I'm getting the error: ERROR: Invalid URL: 'mongodb+srv://clusterycsb.oyia8.mongodb.net/ycsb'.

Must be of the form 'mongodb://:,:/database?options'.

See http://docs.mongodb.org/manual/reference/connection-string/. Command execution failed.

Do you have any idea how I can run YCSB on MongoDB Atlas?

morcelicaio commented 2 years ago

@ksingh7

datnod commented 2 years ago

Leaving this here. I do not know if this will still work and you have to try and figure this out on your own but this worked for me two years ago (its possible these are not the same anymore so you need to check mongodb documentation): Example from the documentation example mongodb url: mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]

My command I had to make it work on YCSB + my mongodb url: ./bin/ycsb run mongodb -s -P workloads/workloada -p operationcount=10000 -p mongodb.url="mongodb://ycsb:ycsb@ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb?ssl=true&replicaSet=ycsb-shard-0&authSource=admin&retryWrites=true&w=majority"

then if you want to replace the mongodb.url with localhost instead: mongodb://localhost:27017/ycsb

I hope you manage to solve it.

datnod commented 2 years ago

@morcelicaio

morcelicaio commented 2 years ago

Thanks for feedback @datnod . Apparently it managed to make the connection, but now it returned a different error.

I got the following response using the parameter you passed me:

Scanning for projects...

Building Core YCSB 0.18.0-SNAPSHOT

--- exec-maven-plugin:1.5.0:exec (default-cli) @ core --- Command line: -db site.ycsb.AsyncMongoDbClient -s -p mongodb.url=mongodb://root:root@ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb?ssl=true&replicaSet=ycsb-shard-0&authSource=admin&retryWrites=true&w=majority -P C:\YCSB-master\core\src\main\java\site\ycsb\workloads\workloada.properties -load YCSB Client 0.18.0-SNAPSHOT

Loading workload... Starting test. 2022-04-06 20:13:19:882 0 sec: 0 operations; est completion in 0 second abr 06, 2022 8:13:20 PM com.allanbank.mongodb.MongoClientConfiguration ADVERTÊNCIA: Using the JVM default SSL Socket Factory. This may allow man-in-the-middle attacks. See http://www.allanbank.com/mongodb-async-driver/userguide/tls.html abr 06, 2022 8:13:20 PM com.allanbank.mongodb.MongoClientConfiguration INFORMAÇÕES: Not validating the replica set name is 'ycsb-shard-0'. abr 06, 2022 8:13:20 PM com.allanbank.mongodb.MongoClientConfiguration INFORMAÇÕES: Unknown property 'retrywrites' and value 'true'. mongo connection created with mongodb://root:root@ycsb-shard-00-00-whpxi.mongodb.net:27017,ycsb-shard-00-01-whpxi.mongodb.net:27017,ycsb-shard-00-02-whpxi.mongodb.net:27017/ycsb?ssl=true&replicaSet=ycsb-shard-0&authSource=admin&retryWrites=true&w=majority DBWrapper: report latency for each error is false and specific error codes to track for latency are: [] abr 06, 2022 8:13:20 PM com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory bootstrap ADVERTÊNCIA: I/O error during bootstrap to ycsb-shard-00-00-whpxi.mongodb.net:27017. java.net.UnknownHostException: ycsb-shard-00-00-whpxi.mongodb.net at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) at com.allanbank.mongodb.client.connection.socket.AbstractSocketConnection.openSocket(AbstractSocketConnection.java:808) at com.allanbank.mongodb.client.connection.socket.AbstractSocketConnection.(AbstractSocketConnection.java:172) at com.allanbank.mongodb.client.connection.socket.SocketConnection.(SocketConnection.java:121) at com.allanbank.mongodb.client.connection.socket.SocketConnectionFactory.connect(SocketConnectionFactory.java:210) at com.allanbank.mongodb.client.connection.auth.AuthenticationConnectionFactory.connect(AuthenticationConnectionFactory.java:101) at com.allanbank.mongodb.client.connection.auth.AuthenticationConnectionFactory.connect(AuthenticationConnectionFactory.java:42) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.bootstrap(BootstrapConnectionFactory.java:177) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.createDelegate(BootstrapConnectionFactory.java:290) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getDelegate(BootstrapConnectionFactory.java:268) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getClusterStats(BootstrapConnectionFactory.java:114) at com.allanbank.mongodb.client.ClientImpl.getClusterStats(ClientImpl.java:199) at com.allanbank.mongodb.client.AbstractMongoOperations.doInsertAsync(AbstractMongoOperations.java:1037) at com.allanbank.mongodb.client.AbstractMongoOperations.insertAsync(AbstractMongoOperations.java:583) at com.allanbank.mongodb.client.AbstractAsyncMongoCollection.insertAsync(AbstractAsyncMongoCollection.java:1255) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:684) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:722) at site.ycsb.AsyncMongoDbClient.insert(AsyncMongoDbClient.java:269) at site.ycsb.DBWrapper.insert(DBWrapper.java:227) at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:621) at site.ycsb.ClientThread.run(ClientThread.java:135) at java.lang.Thread.run(Thread.java:748)

abr 06, 2022 8:13:20 PM com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory bootstrap ADVERTÊNCIA: I/O error during bootstrap to ycsb-shard-00-01-whpxi.mongodb.net:27017. java.net.UnknownHostException: ycsb-shard-00-01-whpxi.mongodb.net at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) at com.allanbank.mongodb.client.connection.socket.AbstractSocketConnection.openSocket(AbstractSocketConnection.java:808) at com.allanbank.mongodb.client.connection.socket.AbstractSocketConnection.(AbstractSocketConnection.java:172) at com.allanbank.mongodb.client.connection.socket.SocketConnection.(SocketConnection.java:121) at com.allanbank.mongodb.client.connection.socket.SocketConnectionFactory.connect(SocketConnectionFactory.java:210) at com.allanbank.mongodb.client.connection.auth.AuthenticationConnectionFactory.connect(AuthenticationConnectionFactory.java:101) at com.allanbank.mongodb.client.connection.auth.AuthenticationConnectionFactory.connect(AuthenticationConnectionFactory.java:42) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.bootstrap(BootstrapConnectionFactory.java:177) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.createDelegate(BootstrapConnectionFactory.java:290) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getDelegate(BootstrapConnectionFactory.java:268) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getClusterStats(BootstrapConnectionFactory.java:114) at com.allanbank.mongodb.client.ClientImpl.getClusterStats(ClientImpl.java:199) at com.allanbank.mongodb.client.AbstractMongoOperations.doInsertAsync(AbstractMongoOperations.java:1037) at com.allanbank.mongodb.client.AbstractMongoOperations.insertAsync(AbstractMongoOperations.java:583) at com.allanbank.mongodb.client.AbstractAsyncMongoCollection.insertAsync(AbstractAsyncMongoCollection.java:1255) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:684) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:722) at site.ycsb.AsyncMongoDbClient.insert(AsyncMongoDbClient.java:269) at site.ycsb.DBWrapper.insert(DBWrapper.java:227) at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:621) at site.ycsb.ClientThread.run(ClientThread.java:135) at java.lang.Thread.run(Thread.java:748)

abr 06, 2022 8:13:20 PM com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory bootstrap ADVERTÊNCIA: I/O error during bootstrap to ycsb-shard-00-02-whpxi.mongodb.net:27017. java.net.UnknownHostException: ycsb-shard-00-02-whpxi.mongodb.net at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) at com.allanbank.mongodb.client.connection.socket.AbstractSocketConnection.openSocket(AbstractSocketConnection.java:808) at com.allanbank.mongodb.client.connection.socket.AbstractSocketConnection.(AbstractSocketConnection.java:172) at com.allanbank.mongodb.client.connection.socket.SocketConnection.(SocketConnection.java:121) at com.allanbank.mongodb.client.connection.socket.SocketConnectionFactory.connect(SocketConnectionFactory.java:210) at com.allanbank.mongodb.client.connection.auth.AuthenticationConnectionFactory.connect(AuthenticationConnectionFactory.java:101) at com.allanbank.mongodb.client.connection.auth.AuthenticationConnectionFactory.connect(AuthenticationConnectionFactory.java:42) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.bootstrap(BootstrapConnectionFactory.java:177) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.createDelegate(BootstrapConnectionFactory.java:290) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getDelegate(BootstrapConnectionFactory.java:268) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getClusterStats(BootstrapConnectionFactory.java:114) at com.allanbank.mongodb.client.ClientImpl.getClusterStats(ClientImpl.java:199) at com.allanbank.mongodb.client.AbstractMongoOperations.doInsertAsync(AbstractMongoOperations.java:1037) at com.allanbank.mongodb.client.AbstractMongoOperations.insertAsync(AbstractMongoOperations.java:583) at com.allanbank.mongodb.client.AbstractAsyncMongoCollection.insertAsync(AbstractAsyncMongoCollection.java:1255) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:684) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:722) at site.ycsb.AsyncMongoDbClient.insert(AsyncMongoDbClient.java:269) at site.ycsb.DBWrapper.insert(DBWrapper.java:227) at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:621) at site.ycsb.ClientThread.run(ClientThread.java:135) at java.lang.Thread.run(Thread.java:748)

abr 06, 2022 8:13:20 PM com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory createDelegate ADVERTÊNCIA: Could not bootstrap a connection to the MongoDB servers. com.allanbank.mongodb.error.CannotConnectException: Could not bootstrap a connection to the MongoDB servers. at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.createDelegate(BootstrapConnectionFactory.java:293) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getDelegate(BootstrapConnectionFactory.java:268) at com.allanbank.mongodb.client.connection.bootstrap.BootstrapConnectionFactory.getClusterStats(BootstrapConnectionFactory.java:114) at com.allanbank.mongodb.client.ClientImpl.getClusterStats(ClientImpl.java:199) at com.allanbank.mongodb.client.AbstractMongoOperations.doInsertAsync(AbstractMongoOperations.java:1037) at com.allanbank.mongodb.client.AbstractMongoOperations.insertAsync(AbstractMongoOperations.java:583) at com.allanbank.mongodb.client.AbstractAsyncMongoCollection.insertAsync(AbstractAsyncMongoCollection.java:1255) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:684) at com.allanbank.mongodb.client.SynchronousMongoCollectionImpl.insert(SynchronousMongoCollectionImpl.java:722) at site.ycsb.AsyncMongoDbClient.insert(AsyncMongoDbClient.java:269) at site.ycsb.DBWrapper.insert(DBWrapper.java:227) at site.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:621) at site.ycsb.ClientThread.run(ClientThread.java:135) at java.lang.Thread.run(Thread.java:748) Error inserting, not retrying any more. number of attempts: 1Insertion Retry Limit: 0 2022-04-06 20:13:20:935 1 sec: 0 operations; est completion in 106751991167300 days 15 hours [CLEANUP: Count=1, Max=56, Min=56, Avg=56, 90=56, 99=56, 99.9=56, 99.99=56] [INSERT: Count=0, Max=0, Min=9223372036854775807, Avg=?, 90=0, 99=0, 99.9=0, 99.99=0] [INSERT-FAILED: Count=1, Max=47903, Min=47872, Avg=47888, 90=47903, 99=47903, 99.9=47903, 99.99=47903]

-

[OVERALL], RunTime(ms), 1077 [OVERALL], Throughput(ops/sec), 0.0 [TOTAL_GCS_PS_Scavenge], Count, 0 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 0 [TOTAL_GCTIME%_PS_Scavenge], Time(%), 0.0 [TOTAL_GCS_PS_MarkSweep], Count, 0 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0 [TOTAL_GCTIME%_PS_MarkSweep], Time(%), 0.0 [TOTAL_GCs], Count, 0 [TOTAL_GC_TIME], Time(ms), 0 [TOTAL_GCTIME%], Time(%), 0.0 [CLEANUP], Operations, 1 [CLEANUP], AverageLatency(us), 56.0 [CLEANUP], MinLatency(us), 56 [CLEANUP], MaxLatency(us), 56 [CLEANUP], 95thPercentileLatency(us), 56 [CLEANUP], 99thPercentileLatency(us), 56 [INSERT], Operations, 0 [INSERT], AverageLatency(us), NaN [INSERT], MinLatency(us), 9223372036854775807 [INSERT], MaxLatency(us), 0 [INSERT], 95thPercentileLatency(us), 0 [INSERT], 99thPercentileLatency(us), 0 [INSERT], Return=ERROR, 1 [INSERT-FAILED], Operations, 1 [INSERT-FAILED], AverageLatency(us), 47888.0 [INSERT-FAILED], MinLatency(us), 47872 [INSERT-FAILED], MaxLatency(us), 47903 [INSERT-FAILED], 95thPercentileLatency(us), 47903 [INSERT-FAILED], 99thPercentileLatency(us), 47903

BUILD SUCCESS

Total time: 2.273 s Finished at: 2022-04-06T20:13:21-03:00 Final Memory: 10M/309M

What would this bootstrap be?

datnod commented 2 years ago

@morcelicaio I do not know. Perhaps google aorund a bit and look if at other issues in YCSB and if different versions of mongodb and YCSB are supported still. Check the documentation of YCSB aswell. Good luck.

datnod commented 2 years ago

@morcelicaio Make sure as well that your mongodb url is correct according to the documentation and you can use my example above to compare. Perhaps you have missed something.

morcelicaio commented 2 years ago

Thanks for the help @datnod.

I did some more tests and managed to run successfully.

I'll leave the settings I used for those who want to perform their tests as well.

The arguments passed were:

-db site.ycsb.MongoDbClient -s -p mongodb.url=mongodb://usercaio:xxxx@clusterycsb-shard-00-01.oyia8.mongodb.net:27017,clusterycsb-shard-00-00.oyia8.mongodb.net:27017,clusterycsb-shard-00-02.oyia8.mongodb.net:27017/ycsb?ssl=true&replicaSet=atlas-bk4da2-shard-0&authSource=admin&w=majority -P C:\YCSB-master\core\src\main\java\site\ycsb\workloads\workloada.properties -load

Where:

my username in Mongodb Atlas cluster is: usercaio the cluster access password is: xxxx

I did some more tests and managed to run successfully.

I'll leave the settings I used for those who want to perform their tests as well.

This was the output when running workload A in the load phase.

The information below is all within your cluster in MongoDB Atlas Where:

The name of the hosts that are part of the cluster are (all on port 27017): clusterycsb-shard-00-01.oyia8.mongodb.net:27017 (primary) clusterycsb-shard-00-00.oyia8.mongodb.net:27017 (secondary) clusterycsb-shard-00-02.oyia8.mongodb.net:27017 (secondary)

The replicaSet name is: atlas-bk4da2-shard-0

This was the output when running workload A in the load phase:

Scanning for projects...

Building Core YCSB 0.18.0-SNAPSHOT --- exec-maven-plugin:1.5.0:exec (default-cli) @ core --- Command line: -db site.ycsb.MongoDbClient -s -p mongodb.url=mongodb://usercaio:xxxx@clusterycsb-shard-00-01.oyia8.mongodb.net:27017,clusterycsb-shard-00-00.oyia8.mongodb.net:27017,clusterycsb-shard-00-02.oyia8.mongodb.net:27017/ycsb?ssl=true&replicaSet=atlas-bk4da2-shard-0&authSource=admin&retryWrites=true&w=majority -P C:\YCSB-master\core\src\main\java\site\ycsb\workloads\workloada.properties -load

YCSB Client 0.18.0-SNAPSHOT

Loading workload...

Starting test.

2022-04-07 10:26:25:656 0 sec: 0 operations; est completion in 0 second abr 07, 2022 10:26:26 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Cluster created with settings {hosts=[clusterycsb-shard-00-00.oyia8.mongodb.net:27017, clusterycsb-shard-00-01.oyia8.mongodb.net:27017, clusterycsb-shard-00-02.oyia8.mongodb.net:27017], mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500, requiredReplicaSetName='atlas-bk4da2-shard-0'} abr 07, 2022 10:26:26 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Adding discovered server clusterycsb-shard-00-00.oyia8.mongodb.net:27017 to client view of cluster abr 07, 2022 10:26:26 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Adding discovered server clusterycsb-shard-00-01.oyia8.mongodb.net:27017 to client view of cluster abr 07, 2022 10:26:26 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Adding discovered server clusterycsb-shard-00-02.oyia8.mongodb.net:27017 to client view of cluster mongo client connection created with mongodb://usercaio:xxxx@clusterycsb-shard-00-01.oyia8.mongodb.net:27017,clusterycsb-shard-00-00.oyia8.mongodb.net:27017,clusterycsb-shard-00-02.oyia8.mongodb.net:27017/ycsb?ssl=true&replicaSet=atlas-bk4da2-shard-0&authSource=admin&retryWrites=true&w=majority DBWrapper: report latency for each error is false and specific error codes to track for latency are: [] abr 07, 2022 10:26:26 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: No server chosen by com.mongodb.client.internal.MongoClientDelegate$1@6630ebd from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=clusterycsb-shard-00-01.oyia8.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=clusterycsb-shard-00-00.oyia8.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=clusterycsb-shard-00-02.oyia8.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Opened connection [connectionId{localValue:1, serverValue:56036}] to clusterycsb-shard-00-01.oyia8.mongodb.net:27017 abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Opened connection [connectionId{localValue:3, serverValue:55687}] to clusterycsb-shard-00-02.oyia8.mongodb.net:27017 abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Opened connection [connectionId{localValue:2, serverValue:57147}] to clusterycsb-shard-00-00.oyia8.mongodb.net:27017 abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Monitor thread successfully connected to server with description ServerDescription{address=clusterycsb-shard-00-00.oyia8.mongodb.net:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[5, 0, 6]}, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=146596600, setName='atlas-bk4da2-shard-0', canonicalAddress=clusterycsb-shard-00-00.oyia8.mongodb.net:27017, hosts=[clusterycsb-shard-00-02.oyia8.mongodb.net:27017, clusterycsb-shard-00-01.oyia8.mongodb.net:27017, clusterycsb-shard-00-00.oyia8.mongodb.net:27017], passives=[], arbiters=[], primary='clusterycsb-shard-00-01.oyia8.mongodb.net:27017', tagSet=TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='AWS'}, Tag{name='region', value='US_EAST_1'}, Tag{name='workloadType', value='OPERATIONAL'}]}, electionId=null, setVersion=11, lastWriteDate=Thu Apr 07 10:22:55 BRT 2022, lastUpdateTimeNanos=157767614738200} abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Monitor thread successfully connected to server with description ServerDescription{address=clusterycsb-shard-00-02.oyia8.mongodb.net:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[5, 0, 6]}, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=156523000, setName='atlas-bk4da2-shard-0', canonicalAddress=clusterycsb-shard-00-02.oyia8.mongodb.net:27017, hosts=[clusterycsb-shard-00-02.oyia8.mongodb.net:27017, clusterycsb-shard-00-01.oyia8.mongodb.net:27017, clusterycsb-shard-00-00.oyia8.mongodb.net:27017], passives=[], arbiters=[], primary='clusterycsb-shard-00-01.oyia8.mongodb.net:27017', tagSet=TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='AWS'}, Tag{name='region', value='US_EAST_1'}, Tag{name='workloadType', value='OPERATIONAL'}]}, electionId=null, setVersion=11, lastWriteDate=Thu Apr 07 10:22:55 BRT 2022, lastUpdateTimeNanos=157767620986900} abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Monitor thread successfully connected to server with description ServerDescription{address=clusterycsb-shard-00-01.oyia8.mongodb.net:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[5, 0, 6]}, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=157298600, setName='atlas-bk4da2-shard-0', canonicalAddress=clusterycsb-shard-00-01.oyia8.mongodb.net:27017, hosts=[clusterycsb-shard-00-02.oyia8.mongodb.net:27017, clusterycsb-shard-00-01.oyia8.mongodb.net:27017, clusterycsb-shard-00-00.oyia8.mongodb.net:27017], passives=[], arbiters=[], primary='clusterycsb-shard-00-01.oyia8.mongodb.net:27017', tagSet=TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='AWS'}, Tag{name='region', value='US_EAST_1'}, Tag{name='workloadType', value='OPERATIONAL'}]}, electionId=7fffffff000000000000006d, setVersion=11, lastWriteDate=Thu Apr 07 10:22:55 BRT 2022, lastUpdateTimeNanos=157767620992100} abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Setting max election id to 7fffffff000000000000006d from replica set primary clusterycsb-shard-00-01.oyia8.mongodb.net:27017 abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Setting max set version to 11 from replica set primary clusterycsb-shard-00-01.oyia8.mongodb.net:27017 abr 07, 2022 10:26:27 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Discovered replica set primary clusterycsb-shard-00-01.oyia8.mongodb.net:27017 abr 07, 2022 10:26:29 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Opened connection [connectionId{localValue:4, serverValue:55715}] to clusterycsb-shard-00-01.oyia8.mongodb.net:27017 2022-04-07 10:26:35:635 10 sec: 36 operations; 3,6 current ops/sec; est completion in 4 minutes [INSERT: Count=36, Max=3123199, Min=141440, Avg=253093,33, 90=202495, 99=3123199, 99.9=3123199, 99.99=3123199] 2022-04-07 10:26:45:645 20 sec: 90 operations; 5,39 current ops/sec; est completion in 3 minutes [INSERT: Count=54, Max=405503, Min=144256, Avg=184442,07, 90=241407, 99=388863, 99.9=405503, 99.99=405503] 2022-04-07 10:26:55:638 30 sec: 154 operations; 6,4 current ops/sec; est completion in 2 minutes [INSERT: Count=64, Max=204287, Min=141824, Avg=156896, 90=172543, 99=203903, 99.9=204287, 99.99=204287] 2022-04-07 10:27:05:631 40 sec: 219 operations; 6,5 current ops/sec; est completion in 2 minutes [INSERT: Count=65, Max=182911, Min=141312, Avg=151805,05, 90=161023, 99=182143, 99.9=182911, 99.99=182911] 2022-04-07 10:27:15:644 50 sec: 281 operations; 6,19 current ops/sec; est completion in 2 minutes [INSERT: Count=62, Max=219903, Min=141440, Avg=163220,65, 90=183935, 99=218239, 99.9=219903, 99.99=219903] 2022-04-07 10:27:25:644 60 sec: 326 operations; 4,5 current ops/sec; est completion in 2 minutes [INSERT: Count=45, Max=399359, Min=144512, Avg=215526,4, 90=329471, 99=399359, 99.9=399359, 99.99=399359] 2022-04-07 10:27:35:632 70 sec: 379 operations; 5,31 current ops/sec; est completion in 1 minute [INSERT: Count=53, Max=912895, Min=146688, Avg=192505,96, 90=265983, 99=284159, 99.9=912895, 99.99=912895] 2022-04-07 10:27:45:637 80 sec: 427 operations; 4,8 current ops/sec; est completion in 1 minute [INSERT: Count=48, Max=364031, Min=153600, Avg=207957,33, 90=266751, 99=364031, 99.9=364031, 99.99=364031] 2022-04-07 10:27:55:637 90 sec: 490 operations; 6,3 current ops/sec; est completion in 1 minute [INSERT: Count=63, Max=226175, Min=141696, Avg=157774,22, 90=175487, 99=188031, 99.9=226175, 99.99=226175] 2022-04-07 10:28:05:633 100 sec: 551 operations; 6,1 current ops/sec; est completion in 1 minute [INSERT: Count=61, Max=518655, Min=141568, Avg=165392,79, 90=181375, 99=204543, 99.9=518655, 99.99=518655] 2022-04-07 10:28:15:645 110 sec: 612 operations; 6,09 current ops/sec; est completion in 1 minute [INSERT: Count=61, Max=271359, Min=140928, Avg=162631,34, 90=188799, 99=251775, 99.9=271359, 99.99=271359] 2022-04-07 10:28:25:635 120 sec: 676 operations; 6,41 current ops/sec; est completion in 58 seconds [INSERT: Count=64, Max=229119, Min=140032, Avg=155758, 90=175615, 99=220415, 99.9=229119, 99.99=229119] 2022-04-07 10:28:35:633 130 sec: 744 operations; 6,8 current ops/sec; est completion in 45 seconds [INSERT: Count=68, Max=180991, Min=140160, Avg=148222,12, 90=162687, 99=177663, 99.9=180991, 99.99=180991] 2022-04-07 10:28:45:637 140 sec: 811 operations; 6,7 current ops/sec; est completion in 33 seconds [INSERT: Count=67, Max=171135, Min=140800, Avg=149707,46, 90=162047, 99=170495, 99.9=171135, 99.99=171135] 2022-04-07 10:28:55:644 150 sec: 879 operations; 6,8 current ops/sec; est completion in 21 seconds [INSERT: Count=68, Max=158591, Min=139648, Avg=146541,18, 90=152831, 99=157951, 99.9=158591, 99.99=158591] 2022-04-07 10:29:05:639 160 sec: 946 operations; 6,7 current ops/sec; est completion in 10 seconds [INSERT: Count=67, Max=179327, Min=139264, Avg=149944,36, 90=162815, 99=175231, 99.9=179327, 99.99=179327] abr 07, 2022 10:29:13 AM com.mongodb.diagnostics.logging.JULLogger log INFORMATION: Closed connection [connectionId{localValue:4, serverValue:55715}] to clusterycsb-shard-00-01.oyia8.mongodb.net:27017 because the pool has been closed. 2022-04-07 10:29:13:755 168 sec: 1000 operations; 6,65 current ops/sec; [CLEANUP: Count=1, Max=151807, Min=151680, Avg=151744, 90=151807, 99=151807, 99.9=151807, 99.99=151807] [INSERT: Count=54, Max=183679, Min=139392, Avg=147560,3, 90=155903, 99=169087, 99.9=183679, 99.99=183679] [OVERALL], RunTime(ms), 168124 [OVERALL], Throughput(ops/sec), 5.947990768718327 [TOTAL_GCS_PS_Scavenge], Count, 1 [TOTAL_GC_TIME_PS_Scavenge], Time(ms), 24 [TOTAL_GCTIME%PS_Scavenge], Time(%), 0.014275177844923985 [TOTAL_GCS_PS_MarkSweep], Count, 0 [TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0 [TOTAL_GC_TIME%PS_MarkSweep], Time(%), 0.0 [TOTAL_GCs], Count, 1 [TOTAL_GC_TIME], Time(ms), 24 [TOTAL_GC_TIME%], Time(%), 0.014275177844923985 [CLEANUP], Operations, 1 [CLEANUP], AverageLatency(us), 151744.0 [CLEANUP], MinLatency(us), 151680 [CLEANUP], MaxLatency(us), 151807 [CLEANUP], 95thPercentileLatency(us), 151807 [CLEANUP], 99thPercentileLatency(us), 151807 [INSERT], Operations, 1000 [INSERT], AverageLatency(us), 166989.632 [INSERT], MinLatency(us), 139264 [INSERT], MaxLatency(us), 3123199 [INSERT], 95thPercentileLatency(us), 223359 [INSERT], 99thPercentileLatency(us), 357119 [INSERT], Return=OK, 1000 BUILD SUCCESS Total time: 02:49 min Finished at: 2022-04-07T10:29:13-03:00 Final Memory: 9M/245M