gwnet commented 4 years ago

hello below is the error I suffer during run mongoose load on minio. what I am missing?

====error log==== [root@localhost mongoose]# sudo java -jar mongoose-4.1.4.jar --storage-driver-type=s3 --storage-net-node-port=9000 --storage-auth-uid=minioadmin --storage-auth-secret=minioadmin --storage-net-node-addrs=127.0.0.1 ################################################### mongoose v 4.1.4 ################################################### 2020-05-25T21:53:10,101 I main Available/installed extensions: dummy-mock --------------------> com.emc.mongoose.storage.driver.mock.DummyStorageDriverMockExtension Load --------------------------> com.emc.mongoose.load.step.linear.LinearLoadStepExtension PipelineLoad ------------------> com.emc.mongoose.load.step.pipeline.PipelineLoadStepExtension WeightedLoad ------------------> com.emc.mongoose.load.step.weighted.WeightedLoadStepExtension coop-mock ---------------------> com.emc.mongoose.storage.driver.coop.mock.CoopStorageDriverMockExtension net ---------------------------> com.emc.mongoose.storage.driver.coop.netty.NettyStorageDriverExtension netty-mock --------------------> com.emc.mongoose.storage.driver.coop.netty.mock.NettyStorageDriverMockExtension http --------------------------> com.emc.mongoose.storage.driver.coop.netty.http.HttpStorageDriverExtension atmos -------------------------> com.emc.mongoose.storage.driver.coop.netty.http.atmos.AtmosStorageDriverExtension s3 ----------------------------> com.emc.mongoose.storage.driver.coop.netty.http.s3.AmzS3StorageDriverExtension swift -------------------------> com.emc.mongoose.storage.driver.coop.netty.http.swift.SwiftStorageDriverExtension nio-mock ----------------------> com.emc.mongoose.storage.driver.coop.nio.mock.NioStorageDriverMockExtension fs ----------------------------> com.emc.mongoose.storage.driver.coop.nio.fs.FileStorageDriverExtension preempt-mock ------------------> com.emc.mongoose.storage.driver.preempt.mock.PreemptStorageDriverMockExtension

Warning: Nashorn engine is planned to be removed from a future JDK release 2020-05-25T21:53:10,307 I main Using the "Oracle Nashorn" scenario engine 2020-05-25T21:53:10,630 E LinearLoadStepClient main linear_20200525.215310.599: failed to init the storage driver CAUSE: com.emc.mongoose.exception.OmgShootMyFootException: Failed to create the storage driver for the type "s3 ", available types: [dummy-mock, coop-mock, netty-mock, atmos, s3, swift, nio-mock, fs, preempt-mock] 2020-05-25T21:53:10,657 E LinearLoadStepClient main linear_20200525.215310.599: failed to start the step slice "com.emc.mongoose.load.step.linear.LinearLoadStepLocal@32cb636e" CAUSE: java.lang.IllegalStateException: Failed to initialize the storage driver CAUSE: com.emc.mongoose.exception.OmgShootMyFootException: Failed to create the storage driver for the type "s3 ", available types: [dummy-mock, coop-mock, netty-mock, atmos, s3, swift, nio-mock, fs, preempt-mock] 2020-05-25T21:53:10,658 I LinearLoadStepClient main linear_20200525.215310.599: load step client started, additional nodes: [] 2020-05-25T21:53:10,701 I main linear_20200525.215310.599: await for 1 step slices done May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase shutdown WARNING: Not allowed to shutdown while state is "SHUTDOWN" May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase shutdown WARNING: Not allowed to shutdown while state is "STOPPED" May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase stop WARNING: Not allowed to stop while state is "STOPPED" May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase shutdown WARNING: Not allowed to shutdown while state is "STOPPED" May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase stop WARNING: Not allowed to stop while state is "STOPPED" May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase shutdown WARNING: Not allowed to shutdown while state is "STOPPED" May 26, 2020 5:53:10 AM com.github.akurilov.commons.concurrent.AsyncRunnableBase stop WARNING: Not allowed to stop while state is "STOPPED" [root@localhost mongoose]#

===my ext folder is=== [root@localhost ext]# ls -l total 6640 -rwxrwxrwx. 1 root root 8691 May 26 03:45 mongoose-load-step-linear.jar -rwxrwxrwx. 1 root root 12052 May 26 03:45 mongoose-load-step-pipeline.jar -rwxrwxrwx. 1 root root 12061 May 26 03:45 mongoose-load-step-weighted.jar -rwxrwxrwx. 1 root root 12270 May 26 03:45 mongoose-storage-driver-coop.jar -rwxrwxrwx. 1 root root 15045 May 26 03:45 mongoose-storage-driver-coop-netty-http-atmos.jar -rwxrwxrwx. 1 root root 578814 May 26 03:45 mongoose-storage-driver-coop-netty-http.jar -rwxrwxrwx. 1 root root 24166 May 26 03:45 mongoose-storage-driver-coop-netty-http-s3.jar -rwxrwxrwx. 1 root root 21192 May 26 03:45 mongoose-storage-driver-coop-netty-http-swift.jar -rwxrwxrwx. 1 root root 6020463 May 26 03:45 mongoose-storage-driver-coop-netty.jar -rwxrwxrwx. 1 root root 23839 May 26 03:45 mongoose-storage-driver-coop-nio-fs.jar -rwxrwxrwx. 1 root root 13625 May 26 03:45 mongoose-storage-driver-coop-nio.jar -rwxrwxrwx. 1 root root 8914 May 26 03:45 mongoose-storage-driver-preempt.jar -rwxrwxrwx. 1 root root 27022 May 26 05:40 mongoose-storage-driver-s3.jar [root@localhost ext]#

gwnet commented 4 years ago

you repo link in the readme is broken, I download from maven myself http://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-s3/

dadlex commented 4 years ago

I've double checked and both jar and docker image are working fine.

Try running the one in docker to see if your environment is wrong (docker run --network host emcmongoose/mongoose:4.1.4 --storage-driver-type=s3 --storage-net-node-port=9000 --storage-auth-uid=minioadmin --storage-auth-secret=minioadmin --storage-net-node-addrs=127.0.0.1)
Check that you are using java11+ that is a requirement

dadlex commented 4 years ago

Also: make sure that you are doing what you want to do. Right now you are using an old bundle that has a built-in s3 driver and doesn't use the new one from: https://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-s3/. If you want to use an old bundle for whatever reason - that's fine. If you expect to be using latest s3 driver, consider reading deployment part of readme and getting the base image from https://mvnrepository.com/artifact/com.github.emc-mongoose/mongoose-base

gwnet commented 4 years ago

@dadlex my environment cannot install docker. I failed to find correct to deploy this mongoose easily. could you please share me one simple guide how to deploy one mongoose to run on host OS on minio? too many jar files. :(

dadlex commented 4 years ago

@gwnet what is your java version (java --version/java -version)?

gwnet commented 4 years ago

@dadlex [root@localhost ~]# java --version openjdk 11.0.7 2020-04-14 LTS OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10-LTS, mixed mode, sharing) [root@localhost ~]#

dadlex commented 4 years ago

Great. @gwnet Now try using latest mongoose bundle: https://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-bundle/4.2.15/mongoose-bundle-4.2.15.jar. Put it somewhere on the system and from that directory run: java -jar mongoose-bundle-4.2.15.jar --storage-driver-type=s3 --storage-net-node-port=9000 --storage-auth-uid=minioadmin --storage-auth-secret=minioadmin --storage-net-node-addrs=127.0.0.1. This are the same steps you were doing before, just with a newer version. If the issue persists, we'll see what to do next. In case the error is the same, please send the logs from running the following command: java -jar mongoose-bundle-4.2.15.jar --storage-driver-type=dummy-mock --load-op-limit-count=1000

gwnet commented 4 years ago

@dadlex I will try this, and let you know the result. :) this is much simple now. :)

gwnet commented 4 years ago

@dadlex thank you so much. it works from my side.

[root@localhost mongoosebin]# java -jar mongoose-bundle-4.2.15.jar --storage-driver-type=s3 --storage-net-node-port=9000 --storage-auth-uid=minioadmin --storage-auth-secret=minioadmin --storage-net-node-addrs=127.0.0.1 ################################################## mongoose v 4.2.18 ################################################## 2020-05-28 05:19:44,032 main INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory. 2020-05-28 05:19:44,034 main INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory. 2020-05-28 05:19:44,036 main INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory. 2020-05-27T21:19:44,245 I main Available/installed extensions: Load --------------------------> com.emc.mongoose.base.load.step.linear.LinearLoadStepExtension dummy-mock --------------------> com.emc.mongoose.base.storage.driver.mock.DummyStorageDriverMockExtension PipelineLoad ------------------> com.emc.mongoose.load.step.pipeline.PipelineLoadStepExtension WeightedLoad ------------------> com.emc.mongoose.load.step.weighted.WeightedLoadStepExtension coop-mock ---------------------> com.emc.mongoose.storage.driver.coop.mock.CoopStorageDriverMockExtension net ---------------------------> com.emc.mongoose.storage.driver.coop.netty.NettyStorageDriverExtension netty-mock --------------------> com.emc.mongoose.storage.driver.coop.netty.mock.NettyStorageDriverMockExtension nio-mock ----------------------> com.emc.mongoose.storage.driver.coop.nio.mock.NioStorageDriverMockExtension http --------------------------> com.emc.mongoose.storage.driver.coop.netty.http.HttpStorageDriverExtension fs ----------------------------> com.emc.mongoose.storage.driver.coop.nio.fs.FileStorageDriverExtension atmos -------------------------> com.emc.mongoose.storage.driver.coop.netty.http.atmos.AtmosStorageDriverExtension s3 ----------------------------> com.emc.mongoose.storage.driver.coop.netty.http.s3.S3StorageDriverExtension swift -------------------------> com.emc.mongoose.storage.driver.coop.netty.http.swift.SwiftStorageDriverExtension

Warning: Nashorn engine is planned to be removed from a future JDK release 2020-05-27T21:19:44,505 I main Using the "Oracle Nashorn" scenario engine 2020-05-27T21:19:44,874 I StorageDriver main linear_20200527.211944.758: creating the storage driver instance for the type "s3" 2020-05-27T21:19:45,013 I StorageDriver main storage/driver/1/s3/216746962: use 96 I/O workers 2020-05-27T21:19:45,172 I StorageDriver main linear_20200527.211944.758: creating the storage driver instance for the type "s3" 2020-05-27T21:19:45,181 I StorageDriver main storage/driver/1/s3/1098568947: use 96 I/O workers 2020-05-27T21:19:45,219 I NettyStorageDriverBase main Adjust output buffer size: 1MB May 27, 2020 9:19:45 PM com.github.akurilov.netty.connection.pool.MultiNodeConnPoolImpl preConnect INFO: Pre-created 1 connections 2020-05-27T21:19:45,475 I LinearLoadStepClient main linear_20200527.211944.758: load step client started, additional nodes: []

Step Id	Timestamp	Op	Concurrency	Count	Step	Last Rate	Mean	Mean (last 10		type	---------------------	-------------------	Time	----------------	Latency	Duration symbols)	yyMMddHHmmss		Current	Mean	Success	Failed	[s]	[op/s]	[MB/s]	[us]	[us]
211944.758	200527211945	CREATE	0	0.0	0	0	0.002	0.0	0.0	0	0

2020-05-27T21:19:45,712 I HttpStorageDriverBase fibers-executor-#36 The bucket checking response is: 404 Not Found 211944.758|200527211955|CREATE| 1|0.95918367| 886| 0|10.002 |98.81072|98.8107| 9572| 10896 211944.758|200527212005|CREATE| 1|0.97969543| 1849| 0|20.002 |85.04016|85.0401| 9542| 10620 211944.758|200527212015|CREATE| 1|0.98648648| 2791| 0|30.002 |103.9509|103.950| 9637| 10611 211944.758|200527212025|CREATE| 1|0.98987341| 3723| 0|40.002 |101.0861|101.085| 9732| 10632 211944.758|200527212035|CREATE| 1|0.99190283| 4651| 0|50.002 |98.43436|98.4343| 9810| 10657 ^C2020-05-27T21:20:37,626 W NettyStorageDriverBase ioWorker#1 Premature channel closure CAUSE: io.netty.handler.codec.PrematureChannelClosureException: channel gone inactive with 1 missing response(s) 2020-05-27T21:20:37,690 I main linear_20200527.211944.758: await for 1 step slices done 211944.758|200527212037|CREATE| 0|0.99031007| 4869| 1|52.218 |102.1495|102.149| 9778| 10616

Results

Load Step Id: linear_20200527.211944.758 Operation Type: CREATE Node Count: 1 Concurrency:
Limit Per Storage Driver: 1 Actual:
Last: 0 Mean: 0.9903100775193798 Operations Count:
Successful: 4869 Failed: 1 Transfer Size: 4.755GB Duration [s]:
Elapsed: 52.218 Sum: 51.69101 Throughput [op/s]:
Last: 102.14956380024535 Mean: 93.63461538461539 Bandwidth [MB/s]:
Last: 102.14956379800661 Mean: 93.63461538461539 Operations Duration [us]:
Avg: 10616.350379954816 Min: 3817 LoQ: 9468 Med: 9679 HiQ: 9967 Max: 309959 Operations Latency [us]:
Avg: 9778.379954816184 Min: 347 LoQ: 8851 Med: 9046 HiQ: 9316 Max: 308834 ... [root@localhost mongoosebin]#

thank you so much again. could you please give me one example? I run create test, and it will generate one csv file, then my next load test will load from this csv file?

dadlex commented 4 years ago

Sorry for the huge delay @gwnet . This is how it can look:

 var sharedConfig = {
    "storage" : {
        "auth": {
            "uid": "user1",
            "secret": "secretKey1" 
        },
        "driver" : {
            "type" : "s3",
            "limit": {
            "concurrency": 0
            }
        },
        "net" : {
            "node" : {
                "port" : 9000,
                "addrs" : "127.0.0.1"
            }
        }      
    }
}

var writeConfig = {
    "load": {
        "op": {
            "limit": {
                "count": 1000
            }
        }
    },
    "item": {
        "output": {
            "file": "/objects.csv",
            "path": "/bucket1"
        },
        "data": {
            "size": "1KB"
        }
    }
}

var readConfig = {
    "load": {
        "op": {
            "type": "read"
        }
    },
    "item": {
        "input": {
            "file": "/objects.csv",
            "path": "/bucket1"
        }
    }
}

Load
.config(sharedConfig)
.config(writeConfig)
.run();

Load
.config(sharedConfig)
.config(readConfig)
.run();

gwnet commented 4 years ago

@dadlex hello buddy, I have run the command well now. thank you so much. when I run below read command on the same 4096k.csv files multiple times. each time, the tool will read the object inside csv by fixed same sequence? or randomly? java -jar mongoose-bundle-4.2.15.jar --storage-driver-type=s3 --storage-net-node-port=9000 --storage-auth-uid=minioadmin --storage-auth-secret=minioadmin --storage-net-node-addrs=127.0.0.1 --load-op-type=READ --item-data-size=4096k --item-input-file=4096k.csv --load-step-id=wayne-4096k --storage-driver-limit-concurrency=40 --load-op-limit-count=1000000 --load-step-limit-time=3600

dadlex commented 4 years ago

@gwnet, hi there! Mongoose will create several actual java threads and those threads will take items in the same order from the file each time. But the order in which they are completed can be different based on the fact which thread finishes its item first. So, yes, the input is the fixed sequence.

gwnet commented 4 years ago

@dadlex hello buddy, can you show me how to run this mongoose-bundle-4.2.15.jar in distributed mode, I have 4 minio server and I would like run mongoose in distributed mode. and one client to control 4 mongoose server to run. :)

gwnet commented 4 years ago

thank you so much

emc-mongoose / mongoose-storage-driver-s3

fail to run this on minio #6

Results