search

Fujio-Turner / sg-log-reader-demo

Parsing and Aggregating Sync Gateway Logs
https://fujio-turner.github.io/sg-log-reader-demo/
Apache License 2.0
5 stars 1 forks source link

Python Multi-Process improvement. #18

Open Fujio-Turner opened 1 year ago

Fujio-Turner commented 1 year ago

Its maxing out the single core that the script is operating on. Since it process a list of all the unique WebSocket ID and go through the IN-Memory version of the SG log file. I beat we can pass the in memory list to multiple instantiation of this script and cut of that big list into 4 smaller list and have each object do work on multiple cores.

Fujio-Turner commented 8 months ago

Read up on passing Data List to multiple cores.

https://superfastpython.com/multiprocessing-share-large-data-between-processes/