Payload does not work well with large csv files

artilleryio / artillery

The complete load testing platform. Everything you need for production-grade load tests. Serverless & distributed. Load test with Playwright. Load test HTTP APIs, GraphQL, WebSocket, and more. Use any Node.js module.

https://www.artillery.io

Mozilla Public License 2.0

8.02k stars 510 forks source link

Payload does not work well with large csv files #97

Open jamesfiltness opened 8 years ago

jamesfiltness commented 8 years ago

I'm trying to use Artillery with a large csv file. I've noticed that it hangs when it tries to parse the csv file. Perhaps streaming the contents of the file would be preferable over using readFileSync?

hassy commented 8 years ago

How large is the CSV file?

jamesfiltness commented 8 years ago

400,000 rows!

hassy commented 8 years ago

@jamesfiltness That is indeed a large CSV file! The fix is not as straightforward as replacing readFileSync with streaming unfortunately, artillery-core expects for the whole dataset to be in memory (it does not read the CSV files itself). Will need to have a think about this. In the meantime, do you need all of that data for your tests? Would something like https://github.com/shoreditch-ops/artillery/issues/73 be useful?

jamesfiltness commented 8 years ago

We need the data as it is in the CSV to reliably test our scenarios so chancejs isn't an option unfortunately.

steveschnepp commented 7 years ago

A nice workaround would be to use the function keyword to pick one randomly in a custom JS from that CSV.

Note that you cannot use function in a loop yet, due to #216.

A working way would be every lookup in the CSV to be done via file access.
A better way would be every lookup in the CSV to be done via file access if the file is above a certain threshold.
A very good way would be to precompile the CSV file into a temp file with an indexed format (gdbm or similar) upon first start, then have it refer it directly to it. Then clean the latter when exiting artillery.

vakil-neelin commented 3 years ago

Hi any updates on this ticket? I have a CSV over 650,000 rows which is needed for testing.