search

funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
289 stars 44 forks source link

BUG: TypeError(f"<data> should be {dict}, {type(data)} given.") #352

Closed spirillen closed 3 months ago

spirillen commented 5 months ago

Description

Process pyfunceble_producer_worker_1:
Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run
raise exception
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run
result = self.target(consumed)
^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/producer.py", line 406, in target
self.collection_query_tool.push(test_result)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 393, in wrapper
return func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 511, in push
self.__push_whois(checker_status)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 626, in __push_whois
raise TypeError(f"<data> should be {dict}, {type(data)} given.")
TypeError: <data> should be <class 'dict'>, <class 'PyFunceble.checker.availability.status.AvailabilityCheckerStatus'> given.

Configuration

cli_testing:
  autocontinue: true
  cidr_expand: false
  inactive-db: true
  db_type: mariadb
  display_mode:
    all: true
    colour: true
    dots: false
    execution_time: true
    less: false
    percentage: true
    quiet: false
    simple: false
    status: all
  file_generation:
    analytic: true
    hosts: false
    merge_output_dirs: false
    no_file: false
    plain: true
    unified_results: false
  local_network: false
  sorting_mode:
    hierarchical: true
  days_between:
    db_clean: 14
    db_retest: 7
debug:
  active: true
  level: critical
lookup:
  collection: true
  dns: true
  http_status_code: true
  netinfo: true
  reputation: false
  special: true
  timeout: 5
  whois: true
user_agent:
  browser: firefox
  custom: null
  platform: linux
dns:
  follow_server_order: false
  trust_server: true
  protocol: UDP
  server:
    - 127.0.0.1:5302
    - 9.9.9.10
    - 149.112.112.10
collection:
  preferred_status_origin: latest
  push: true
  url_base: https://collection.dead-hosts.funilrys.com
max_http_retries: 1
proxy:
  # global:
  #   http: socks5h://127.0.0.1:9050
  #   https: socks5h://127.0.0.1:9050
  rules:
    - http: socks5h://127.0.0.1:9050
      https: socks5h://127.0.0.1:9050
      tld:
        - onion

Reproduction

  1. pyfunceble -f https://gist.githubusercontent.com/okutbay/5b4974b70673dfdcc21c517632c1f984/raw/5bbdaba2927a5a5e751243daa251a9d242f3de0b/free_email_provider_domains.txt --local --dns-protocol HTTPS --dns "https://dns10.quad9.net/dns-query","https://anycast.censurfridns.dk/dns-query","https://unicast.censurfridns.dk/dns-query"

Expected behavior

Screenshots

Versions

OS: ubu 22.04

Python Version: 3.11

PyFunceble Version: 

pyfunceble --version
^CTraceback (most recent call last):
File "/home/$USER/.local/bin/pyfunceble", line 8, in <module>
sys.exit(tool())
^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/entry_points/pyfunceble/cli.py", line 1237, in tool
PyFunceble.facility.ConfigLoader.start()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/config/loader.py", line 405, in start
config = self.get_config_file_content()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/config/loader.py", line 334, in get_config_file_content
self.download_dynamic_infrastructure_files()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/config/loader.py", line 315, in download_dynamic_infrastructure_files
UserAgentsDownloader().start()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/downloader/base.py", line 285, in start
if DownloadHelper(self.download_link).download_text(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/helpers/download.py", line 235, in download_text
req = session.get(self.url, verify=self.certificate_validation)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/http/client.py", line 1374, in getresponse
response.begin()
File "/usr/lib/python3.11/http/client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/socket.py", line 705, in readinto
return self._sock.recv_into(b)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/ssl.py", line 1278, in recv_into
return self.read(nbytes, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/ssl.py", line 1134, in read
return self._sslobj.read(len, buffer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Additional context

spirillen commented 5 months ago

Frustration... I've swiched to CSV to avoif the dict issue, only to get this...

Process pyfunceble_producer_worker_1:
Traceback (most recent call last):
File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run
raise exception
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run
result = self.target(consumed)
^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/producer.py", line 406, in target
self.collection_query_tool.push(test_result)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 393, in wrapper
return func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 511, in push
self.__push_whois(checker_status)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 626, in __push_whois
raise TypeError(f"<data> should be {dict}, {type(data)} given.")
TypeError: <data> should be <class 'dict'>, <class 'PyFunceble.checker.availability.status.AvailabilityCheckerStatus'> given.
spirillen commented 5 months ago

Wondering who wrote these lines years ago... don't you blame me.... :smile: :smirk:

https://github.com/funilrys/PyFunceble/blob/dev/docs/facts/known_issues.rst#sql-missing-default-data-in-whois-table

https://github.com/funilrys/PyFunceble/blob/dev/docs/facts/known_issues.rst?plain=1#L71-L104

spirillen commented 5 months ago

There is a thing or two, that is confusing...

SELECT COUNT(*) FROM pyfunceble_whois_record
    WHERE id IS NOT NULL;

returns 58595 records (from a clean "installed" db) So which domain is causing this issue, that is missing from the log. Meaning already building the WHOIS db from scratch?

  1. Why did running the known workaround NOT solve this?
  2. Going to run with --logging-level debug
  3. Will return with, hopefully, more knowledge
spirillen commented 5 months ago

Unlike what I was believing, this issue is not related to WRITE data, but to READ data...

spirillen commented 3 months ago

This is still a issue in 4.2.7

Screenshot

```python pyfunceble --output-location ./ -f ad_servers.txt emd.txt exp.txt fsa.txt grm.txt hjk.txt mmt.txt pha.txt psh.txt pup.txt ██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗ ██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝ ██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗ ██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝ ██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗ ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝ Subject Status Source Expiration Date Registrar HTTP Code Checker ---------------------------------------------------------------------------------------------------- ----------- ---------- ----------------- ------------------------------ ---------- ------------- 005.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 006.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 007.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 04fd-74b3-b2f6-c645.reporo.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 008.free-counter.co.uk ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 06c5-dbbd-eb79-4cd4.reporo.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 060810131024.c.mystat-in.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 02gzx.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 007.go2cloud.org ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 011i5.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 0602v.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 05ni7.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 06rph.voluumtrk.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 0075-7112-e7eb-f9b9.reporo.net ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.08.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.05.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.100.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.09.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.03.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.1.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.04.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 050003.voodoo.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.06.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.10.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.101.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.07.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.103.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 08.185.87.104.liveadvert.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 0124498474f7c13ac9a2-6b191446002b31342189d56cabcf5227.r11.cf2.rackcdn.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY 050005.voodoo.com ACTIVE DNSLOOKUP Unknown Unknown Unknown AVAILABILITY Process pyfunceble_producer_worker_1: Traceback (most recent call last): File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 395, in run raise exception File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/base.py", line 373, in run result = self.target(consumed) ^^^^^^^^^^^^^^^^^^^^^ File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/cli/processes/workers/producer.py", line 406, in target self.collection_query_tool.push(test_result) File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 393, in wrapper return func(self, *args, **kwargs) # pylint: disable=not-callable ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 511, in push self.__push_whois(checker_status) File "/home/spirillen/.local/lib/python3.11/site-packages/PyFunceble/query/collection.py", line 626, in __push_whois raise TypeError(f" should be {dict}, {type(data)} given.") TypeError: should be , given. ^C Thank you for NOT using PyFunceble! ```

When can We expect it to be fixed or see the rust release?