HELP: Extremly poor and slow testing

[spirillen]

Description

After running this test pyfunceble --dns-lookup-only --dns 192.168.1.54:5302 --database-type csv -f https://raw.githubusercontent.com/blocklistproject/Lists/master/porn.txt for a week I have only reached to boystreaking.blogspot.fr

wc -l /tmp/pyfunceble/output/porn.txt/domains/INACTIVE/list /tmp/pyfunceble/output/porn.txt/domains/ACTIVE/list
  399957 /tmp/pyfunceble/output/porn.txt/domains/INACTIVE/list
   44642 /tmp/pyfunceble/output/porn.txt/domains/ACTIVE/list
  444599 total

cat ~/.config/PyFunceble/.PyFunceble.overwrite.yaml 
cli_testing:
  cidr_expand: true
  db_type: mariadb
  display_mode:
    all: true
    colour: true
    dots: false
    execution_time: true
    less: false
    percentage: true
    quiet: false
    simple: false
    status: all
  file_generation:
    analytic: true
    hosts: false
    merge_output_dirs: false
    no_file: false
    plain: true
    unified_results: false
  local_network: true
  sorting_mode:
    hierarchical: true
debug:
  active: false
  level: critical
lookup:
  dns: true
  http_status_code: true
  netinfo: true
  reputation: false
  special: true
  timeout: 5
  whois: true
user_agent:
  browser: firefox
  custom: null
  platform: linux
dns:
  follow_server_order: false
  trust_server: false
  protocol: UDP
  server:
    - 130.226.161.34
    - 89.233.43.71
    - 185.121.177.177
    - 163.172.168.171
    - 192.168.1.100:5302
    - 192.168.1.6
    - 9.9.9.10
    - 149.112.112.10
collection:
   push: true
   url_base: https://collection.dead-hosts.funilrys.com

cat /etc/fstab
none /tmp/pyfunceble/ ramfs noauto,user,mode=0777 0 0
df -h /tmp/pyfunceble/
Filesystem      Size  Used Avail Use% Mounted on
none               0     0     0    - /tmp/pyfunceble

How come it takes a week to test against 1 DNS server.... and how to optimize this

Even if this should be extremely slow, the absolutely max time for testing each record is 1400ms (recursor lookup time if connection is temporary slow)

[spirillen]

pyfunceble 4.1.0b11.dev (Blue Duckling: Grandiflora)

[funilrys]

If you choose to set trust_server: false, PyFunceble will go through all of the server before setting the status from the DNS Lookup.

CF: https://pyfunceble.readthedocs.io/en/dev/configuration/index.html#dns-trust-server

Description: Activates or disables the trust mode. When the trust mode is active and the first read DNS server gives us a negative response (without any error), we take it as it is.

Otherwise, when the trust mode is disabled, when the first read DNS server gives us a negative response (without any error), we still ask all other DNS servers that were given or found.

[spirillen]

https://asciinema.org/a/ZyZzw2dikMyg1tKGstDEIx33H

[spirillen]

trust_server:

Your right... forgot to set that one in my command line.... thanks for spotting this one

[spirillen]

Another question now you are online

About the --soometest-only if I add IE --dns-lookup-only --http-only then it will test both and yet only for DNS and HTTP status code or? that's not something I have lookup, but I don't recall reading about this in the merge you made back then

[spirillen]

I still think it is very slow with the CSV.... while for some reason the SQL have speed up :hand:

https://asciinema.org/a/tAT4MGPfu0vwxetpdlnNKpqQr

[funilrys]

That's the reason SQL databases exists ... CSV needs to be loaded in memory .... When SQL is mostly delegated to a optimized system and format.

[spirillen]

Yes and I do load the entire dir into tmpfs (ramdrive)