Consistent OOM Kill Issues with Httpx Execution – How to Prevent Out of Memory Kills?

[hnvdie]

ISSUE DESCRIPTION:

I’m experiencing frequent OOM (Out of Memory) kills when running httpx. This occurs even with a high swap size (over 100GB), but httpx processes are still terminated by the OOM killer. im running or scan with large list its +50k-120k urls.

Httpx version: 1.6.3 (outdated)

Commands Used:

These are the commands where the OOM kills typically happen:

First Command

$ httpx -l list.txt -mr (myregex) -er (myregex) -p 80 -rl 50 -rlm 100 -threads 30 -delay 10ms -silent -o result.txt

second command

$ httpx -l directory.ts -path "media" -mr (myregex) -er (myregex) -p 80 -rl 50 -rlm 100 -threads 30 -delay 10ms -silent -o result_directory.ts

i run both command over python shortcut program with os.system like this os.system(f"{firstcommand} && {secondcommand}")

System Log Output:

Here are the relevant entries from syslog showing the OOM kill events:

2024-11-06T16:42:09.357484+00:00 rena kernel: Out of memory: Killed process 602233 (httpx) total-vm:19703332kB, anon-rss:6928784kB, file-rss:8408kB, shmem-rss:0kB, UID:0 pgtables:34632kB oom_score_adj:0
2024-11-06T16:42:09.360824+00:00 rena systemd[1]: session-4019.scope: A process of this unit has been killed by the OOM killer.
2024-11-06T17:31:09.588851+00:00 rena systemd[1]: user-0.slice: A process of this unit has been killed by the OOM killer.
2024-11-06T17:31:09.588929+00:00 rena systemd[1]: user.slice: A process of this unit has been killed by the OOM killer.
2024-11-07T12:28:21.474317+00:00 rena systemd[1]: user@0.service: A process of this unit has been killed by the OOM killer.
2024-11-07T12:28:21.478106+00:00 rena systemd[1]: user@0.service: Main process exited, code=killed, status=9/KILL
2024-11-07T12:28:21.493247+00:00 rena kernel: Out of memory: Killed process 610837 (systemd) total-vm:20356kB, anon-rss:0kB, file-rss:4404kB, shmem-rss:0kB, UID:0 pgtables:84kB oom_score_adj:100
2024-11-07T12:54:55.425016+00:00 rena kernel: Out of memory: Killed process 611595 (httpx) total-vm:21374092kB, anon-rss:7734248kB, file-rss:12432kB, shmem-rss:0kB, UID:0 pgtables:37888kB oom_score_adj:0
2024-11-07T12:54:55.429935+00:00 rena systemd[1]: session-4167.scope: A process of this unit has been killed by the OOM killer.

Question:

How can I prevent httpx from being consistently terminated by the OOM killer, given that I already have extensive swap space? Are there specific settings or adjustments I should try, such as tweaking oom_score_adj for httpx or optimizing other command parameters to reduce memory usage?

Any advice on how to manage or configure httpx to avoid these OOM kill issues would be greatly appreciated!

[hnvdie]

solved with batching proggress