Memory usage with newer versions

[Roemer]

We used a very old version of px (the single .exe from May 20, 2019) which worked great. But we now faced some issues with a handful of urls (maybe because or https://tldr.fail/ ?) and upgraded to the latest version and those issues are fixed. But now the new version uses significantly more RAM. Initially each worker uses up to 40 mb (and later sometimes drops to about 15 mb/worker but compared to the very old version which was often less than 1 mb/worker, this is still a lot. Is there any way to optimize this a bit? We usually run 10 runners locally, is this too much and not usefull? We are on Windows and use pythonw to run it in the background.

[genotrance]

How old are we talking? v0.7.0 switched to libcurl as the backend which should be much faster and likely not need as many workers. Also, v0.9.0 upped the default threads from 5 to 32 so you likely don't need more than 2 workers which is the default. See this comment for some additional context on that.

[Roemer]

Even older, 3 releases before you even had versioning ;) We had set them to 10 workers and 20 threads. We'll probably just downscale the workers to 3 and up the threads to 50 or so. That would then total to max 120 mb or ram which is less than a normal chrome tab. So compared to before, a lot but compared to the rest, not so much.

[genotrance]

You could also experiment with just increasing threads and use a single worker. Memory usage doesn't seem to change much at startup with more threads. As they get used, they will probably use more but will be marginal since all the code will be shared.

Here's some memory usage data from my Windows machine:

• Python - 10MB
• Standard libs Px uses - 13MB
• Keyring - 12MB
• Netaddr - 2MB
• QuickJS + PSUtil - 0.5MB

This itself adds up to 37MB, rest is Px which goes up to 45MB.