search

timescale / tsbs

Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
MIT License
1.26k stars 297 forks source link

MongoDB load_queries: incomplete read of message header -> i/o timeout #204

Open seybi87 opened 2 years ago

seybi87 commented 2 years ago

Hi TSBS community,

I am trying to run the TSBS against MongoDB 5.0.0 based on @gregorynoma pull request (https://github.com/timescale/tsbs/pull/173) which we have merged into our own TSBS fork (https://github.com/benchANT/tsbs).

However I am facing the following issue during the load phase:

 ...
 1652097603,47000.36,1.317000E+08,67886.57,-,-,-
1652097613,41001.36,1.321100E+08,67748.70,-,-,-
1652097623,54998.19,1.326600E+08,67683.64,-,-,-
1652097633,48001.28,1.331400E+08,67583.74,-,-,-
1652097643,42998.46,1.335700E+08,67459.56,-,-,-
1652097653,40001.58,1.339700E+08,67321.59,-,-,-
1652097663,40998.52,1.343800E+08,67189.97,-,-,-
2022/05/09 12:01:04 Bulk insert docs err: connection(172.31.16.222:27017[-64]) incomplete read of message header: read tcp 172.17.0.2:33294->172.31.16.222:27017: i/o timeout

I do not see any error message in the MongoDB logs that indicates a problem on server side and I also do not see any problem on the resource level that indicates a problem.

To reproduce the issue, execute the following commands:

/binary/tsbs/bin/tsbs_generate_queries --queries 100000 --query-type single-groupby-1-1-1 --scale 1000 --seed 123 --timestamp-end 2022-01-05T00:00:00Z --timestamp-start 2022-01-01T00:00:00Z --use-case cpu-only --format mongo --file /binary/tsbs/bin/queries
/binary/tsbs/bin/tsbs_generate_data --initial-scale 1 --scale 1000 --seed 123 --timestamp-end 2022-01-05T00:00:00Z --timestamp-start 2022-01-01T00:00:00Z --use-case cpu-only --format mongo --file /binary/tsbs/bin/data
/binary/tsbs/bin/tsbs_load_mongo --url mongodb://user:secret@172.31.16.222:27017/admin --document-per-event false  --timeseries-collection false  --retryable-writes true  --ordered-inserts true  --random-field-order true  --batch-size 1000 --do-abort-on-exist true --hash-workers false --insert-intervals 1,2 --seed 123 --workers 16 --file /binary/tsbs/bin/data

Running the same benchmark with --document-per-event true works fine. however with really slow query execution performance.

Any advice on this issue would be much appreciated!

jonatas commented 2 years ago

It also relates to #97.

jonatas commented 2 years ago

@seybi87, one suggestion is to try to use the work done on #149 that was not merged but can be useful for your research and better comprehension.