search

funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
297 stars 44 forks source link

SQL interaction #208

Closed spirillen closed 2 years ago

spirillen commented 3 years ago

Is your feature request related to a problem? Please describe. The way the SQL tables and the usage of those tables, really need some works

Describe the solution you'd like Mentioned plenty of times before

If we take a peak at three GHA runs with the mariadb backend (cdb.matrix.rocks) will see that in a 45 mins cycle only 3-8 domains actually are tested

.....II.......I.....I.II.II..IIIIII.IIIIIIIIIIIIIIIIIIIIIIIIIII.IIIIIIIII..IIIIIIIIIIIIIIIIIIII..IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII.......I..II..II...I......................II....IIII..I..IIII.......IIII.I..........
.....II.......I.....I.II.II..IIIIII.IIIIIIIIIIIIIIIIIIIIIIIIIII.IIIIIIIII..IIIIIIIIIIIIIIIIIIII..IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII.......I..II..II...I......................II....IIII..I..IIII.......IIII.I..............
.....II.......I.....I.II.II..IIIIII.IIIIIIIIIIIIIIIIIIIIIIIIIII.IIIIIIIII..IIIIIIIIIIIIIIIIIIII..IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII.......I..II..II...I......................II....IIII..I..IIII.......IIII.I.......................

From: (same order)

https://github.com/dns-test/rpz.mypdns.cloud/actions/runs/512543777 https://github.com/dns-test/rpz.mypdns.cloud/actions/runs/512390971 https://github.com/dns-test/rpz.mypdns.cloud/actions/runs/512258413

funilrys commented 3 years ago

Isn't this why we now have the preload thing @spirillen ?

spirillen commented 3 years ago

And when was it #202 w might have been fixed :tongue: I'll give it a spin later, are you kind to add this one to the retest project?

spirillen commented 3 years ago

The test is committed here https://github.com/dns-test/porn.host.rpz/blob/6d0a53391bf820789b5184e1825ba834fe0d8cbc/.pyfunceble/.PyFunceble.yaml#L22-L45

First GHA run is here: https://github.com/dns-test/porn.host.rpz/actions/runs/564545645

At time of writing this have 2302 records in pyfunceble_continue (2021-02-13T23:06:21 CET)

SELECT COUNT(source)
FROM pyfunceble.pyfunceble_continue
    where `source` = "/home/runner/work/porn.host.rpz/porn.host.rpz/domains.txt";
spirillen commented 3 years ago

Another test that have been running for some time

https://github.com/dns-test/rpz.mypdns.cloud/blob/293cf28c825be312233cad1026cec802f76043db/pyfunceble/.PyFunceble.yaml#L22-L45 commit: https://github.com/dns-test/rpz.mypdns.cloud/commit/4fac7a310da88ac04bb6a5136080317bffdb9c6d#diff-7f3a4449b6c3799c7aae7dd18b925220093e68368182a58165036b031131849b

However, I would need to check the DB to see what is done between run as there are no output "logs" what what have been tested or not.

At the time of writing there are 92781 records in pyfunceble_continue for this repo (2021-02-13T22:51:47 CET)

Count script: https://www.mypdns.org/P32

funilrys commented 2 years ago

This needs some extensive tests in the future. Closing for now.

Stability is my priority right now. Performance tweak regarding SQL will come later.