funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
289 stars 44 forks source link

FEATURE: meilisearch or redis support vs RamDrive #351

Open spirillen opened 5 months ago

spirillen commented 5 months ago

Description

Earlier I've documented how it is possible to setup and use a ramDrive to support the CSV test results data.

However, when you are using a SQL driven backend, you can not as easily do this, the most trustworthy way is using a RAM-driven DB like mailisearch (not elasticSearch, it is to heavy for most systems) or RedisServer as they only take up the space that is needed.

Current load over hours, after restarting @pyfunceble after broken connection to MariaDB

image

Cost: image

Possible Solution

By using a no-sql backend we can:

  1. Read out the full db to the middleman (~550.000 records is about ~35MB) 1.2. This will speed up the "--continue" and read in from source to first test
  2. for every X test results, it will keep trying writing these data to SQL backend until succees

Considered Alternative

Not for now

Additional context

Nope, be happy