search

orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

Notification loss under heavy load #416

Closed c0c0n3 closed 3 years ago

c0c0n3 commented 3 years ago

Describe the bug

While hammering QuantumLeap with an ungodly amount of concurrent POSTs to the notify endpoint, about 4% of the posted entities didn't make it to the DB.

To Reproduce

$ source setup_dev_env.sh
$ cd src/tests/benchmark

Edit notify-load-test.sh to comment out the docker-compose down statement, then run the script

$ ./notify-load-test.sh

The script will try pumping 10,000 entities into QuantumLeap concurrently and as quickly as it can. When the script is done count how many rows are in the Crate entity table. If you're unlucky all 10,000 entities will have made it there, so you might have to run the script multiple times. When you're done, don't forget to clean up after yourself

$ docker-compose down -v

Expected behavior

All 10,000 entities should always be (eventually) inserted into the DB backend.

Additional context

This issue cropped up while benchmarking QuantumLeap, details over here:

github-actions[bot] commented 3 years ago

Stale issue message

c0c0n3 commented 3 years ago

given we're going to have a work queue implementation soon, we should reconsider this in light of that. also related: #242. Plus, for some backends like Crate we might be able to do something smart when we insert the data, like not losing a whole batch because of just a couple of rows that can't be inserted, see these comments to #481

chicco785 commented 3 years ago

pr #501 provides a queue mechanism to better handle heavy load