funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
298 stars 45 forks source link

BUG | memleak #391

Open spirillen opened 1 week ago

spirillen commented 1 week ago

What is the problem you are experiencing?

Was trying to report this as Report a security vulnerability

but all I got is this

image

For the first time in years, I see what appears to be a memory leak, but as the system was completely unresponsive, and can't do much more than sharing a screenshot of the memory usage for until I had to increase the amount of RAM assigned to the mashine

image

How can we reproduce the problem?

To reproduce the problem:

  1. install the ProxMox virtual machine https://github.com/if-you-want-peace-prepare-for-war/proxmox/blob/master/ct/pyfunceble.sh (Not fully up to date with running version)
  2. run the command pyfunceble -c --preload --all --execution --hierarchical -f https://raw.githubusercontent.com/mypdns/matrix/refs/heads/master/source/porn_filters/explicit_content/wildcard.list
  3. Wait over the night, and this is the result.
  4. PS: the --preload are extremely slow to write in to the DB, only about 12 records per seconds, should be more like 1000-3000 depending on methods (PostgreSQL)

Do you have a screenshot?

Screenshot ![image](https://github.com/user-attachments/assets/424ae007-d950-476e-934a-6ac9f2274282)

What did you expect to happen?

Not using more than 100 mb of RAM

Is there a workaround?

Not that I've found

How did you configure PyFunceble?

cli_testing:
  inactive_db: yes
  whois_db: yes
  db_type: postgresql
lookup:
  dns: yes
  http_status_code: yes
  netinfo: yes
  special: yes
  whois: yes
  reputation: no
  platform: yes
  timeout: 2
dns:
  follow_server_order: no
  trust_server: yes
  server:
    - 9.9.9.10
    - 149.112.112.10
  protocol: UDP
  delay: 0.0

verify_ssl_certificate: no
max_http_retries: 2

platform:
  push: yes
  preferred_status_origin: recommended
  checker_priority:
    - availability
    - syntax
  checker_exclude:
    - reputation

Which Operating System did you use?

Ubuntu 24.04 LTS

Which Python Version did you use?

3.11

Which PyFunceble Version did you use?

4.3.0a4.dev

Additional context

I would like to add...

spirillen commented 6 days ago

Reproduced...

image

spirillen commented 4 days ago

It is easy to see when PyF are running both standard tests + platform

image

spirillen commented 2 days ago

This is interesting... it seems like the memleak is related to the PostdreSQL. In the following screenshot I have solely been using MariaDB

image