search

funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
298 stars 46 forks source link

PyFunceble 3.0.0(.8-dev) vs Ubuntu 19 (eoan) #84

Closed spirillen closed 4 years ago

spirillen commented 4 years ago

Bug description

🌟 🎉 🍰 We now support IPv6! 🍰 🎉 🌟
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f79b5447d10>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f79b5447d10>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1125, in tool
    raise exception
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1117, in tool
    generate_all_results_only=args.generate_all_files_from_database,
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 140, in __init__
    file_path, generate_results_only, generate_all_results_only
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 178, in dispatch_file_test
    PyFunceble.output.Constructor()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 100, in __init__
    self.restore()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 548, in restore
    structure = self._get_structure()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 456, in _get_structure
    PyFunceble.LINKS.dir_structure.replace("dev", "master")
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 282, in get
    result = self.session.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 166, in send
    return super(HostSSLAdapter, self).send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f79b5447d10>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Modifications under .PyFunceble.yaml

None

A clear representation of what you changed. Updated from 18.1 to 19.04

Reproduction

pyfunceble -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301 -h --plain

pyfunceble -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301 -h

pyfunceble -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301
head /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list
porno-tube.pro
cdngangsta.com
chaturbate.com.br
lesbianporn.co.uk
nextdoorstudios.com
noirmale.com
nude-virgins.info
nudeblackporn.com
oldmomporn.com
only-amateur-porn.com

Screenshots

image

Versions

OS: Ubuntu 19.04 Eoan Python Version: Python 3.(7|8).0 PyFunceble Version: 3.0.0

Additional context

Pyfunceble-dev:

sudo -H pip3 install --upgrade pyfunceble-dev                                                  
Collecting pyfunceble-dev
  Downloading PyFunceble_dev-3.0.8-py3-none-any.whl (263 kB)
     |████████████████████████████████| 263 kB 1.3 MB/s 
Requirement already satisfied, skipping upgrade: PyMySQL in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (0.9.3)
Requirement already satisfied, skipping upgrade: domain2idna>=1.6.1 in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (1.9.1)
Requirement already satisfied, skipping upgrade: python-dotenv in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (0.10.3)
Requirement already satisfied, skipping upgrade: dnspython in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (1.16.0)
Requirement already satisfied, skipping upgrade: urllib3 in /usr/lib/python3/dist-packages (from pyfunceble-dev) (1.24.1)
Requirement already satisfied, skipping upgrade: python-box in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (3.4.6)
Requirement already satisfied, skipping upgrade: requests in /usr/lib/python3/dist-packages (from pyfunceble-dev) (2.21.0)
Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (5.2)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from pyfunceble-dev) (41.1.0)
Requirement already satisfied, skipping upgrade: cryptography in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (2.8)
Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.7/dist-packages (from pyfunceble-dev) (0.4.3)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /usr/lib/python3/dist-packages (from cryptography->pyfunceble-dev) (1.12.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.7/dist-packages (from cryptography->pyfunceble-dev) (1.13.2)
Requirement already satisfied, skipping upgrade: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi!=1.11.3,>=1.8->cryptography->pyfunceble-dev) (2.19)
Installing collected packages: pyfunceble-dev
  Attempting uninstall: pyfunceble-dev
    Found existing installation: PyFunceble-dev 2.55.0
    Uninstalling PyFunceble-dev-2.55.0:
      Successfully uninstalled PyFunceble-dev-2.55.0
Successfully installed pyfunceble-dev-3.0.8
$:tmp$ python3 pyfunceble -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301 -h --plain
/usr/bin/python3.8: can't open file 'pyfunceble': [Errno 2] No such file or directory
$:tmp$ python3 pyfunceble-dev -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301 -h --plain
/usr/bin/python3.8: can't open file 'pyfunceble-dev': [Errno 2] No such file or directory
$:tmp$ python pyfunceble -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301 -h --plain     
python: can't open file 'pyfunceble': [Errno 2] No such file or directory
$:tmp$ pyfunceble -f /var/storage01/repositories/gitlab/matrix/matrix/tmp/t2.list --dns 127.0.0.1:5301 -h --plain       

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

🌟 🎉 🍰 We now support IPv6! 🍰 🎉 🌟
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f2684eed350>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2684eed350>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1157, in tool
    raise exception
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1149, in tool
    generate_all_results_only=args.generate_all_files_from_database,
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 140, in __init__
    file_path, generate_results_only, generate_all_results_only
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 178, in dispatch_file_test
    PyFunceble.output.Constructor()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 100, in __init__
    self.restore()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 548, in restore
    structure = self._get_structure()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 463, in _get_structure
    PyFunceble.LINKS.dir_structure.replace("master", "dev")
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 282, in get
    result = self.session.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 166, in send
    return super(HostSSLAdapter, self).send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f2684eed350>: Failed to establish a new connection: [Errno -2] Name or service not known'))
pyfunceble --version
bash: /usr/local/bin/pyfunceble: No such file or directory
$:tmp$ 
$:tmp$ pyfunceble -version
bash: /usr/local/bin/pyfunceble: No such file or directory
$:tmp$ pyfunceble -v
bash: /usr/local/bin/pyfunceble: No such file or directory
$:tmp$ pyfunceble --help
bash: /usr/local/bin/pyfunceble: No such file or directory
funilrys commented 4 years ago

HI @spirillen, It looks like you are coming from PyFunceble-dev 2.55.0 which was released on Jan 24, 2020. Not many has changed since then.

May I invite you to remove/backup your .PyFunceble.yaml file and delete both .PyFunceble.yaml and .PyFunceble_production.yaml file ?

Stay safe and healthy.

funilrys commented 4 years ago

if you really get /usr/local/bin/pyfunceble: No such file or directory.

You should try: 

$ pip3 uninstall PyFunceble-dev PyFunceble
$ pip3 install PyFunceble-dev # or PyFunceble
$ PyFunceble --version
spirillen commented 4 years ago

That have been tried more than once :unamused: One reason seems to be a misc of python2.7, 3.7 and 3.8 on the same installation.

Pyfunceble default calls out for the python2.7, but is installed with 3.(7|8); as I've tested with both

But the bash: /usr/local/bin/pyfunceble: No such file or directory happens only if i do not specify the full path and python version (in this case 3.8)

Another thing that have crossed my mind is, which python module is called for the HTTPSConnectionPool ?

funilrys commented 4 years ago

@spirillen When that propper to your machine then.

What does pip3 config list gives you? And env | grep -i python ?

Cheers!

funilrys commented 4 years ago

Another thing that have crossed my mind is, which python module is called for the HTTPSConnectionPool ?

That's the requests module. It is used to download/update/overwrite all your local file with the updated one. A list of them:

Cheers!

spirillen commented 4 years ago

(sudo) pip3 config list

= Nothing

(sudo) env | grep -i python

= Nothing

image

But here is a weird thing......

$ pip3 uninstall PyFunceble-dev PyFunceble
$ pip3 install PyFunceble-dev # or PyFunceble
$ PyFunceble --version
$ pip3 uninstall PyFunceble-dev PyFunceble
WARNING: Skipping PyFunceble-dev as it is not installed.
Found existing installation: PyFunceble 3.0.0
Uninstalling PyFunceble-3.0.0:
  Would remove:
    /home/joakim/.local/bin/PyFunceble
    /home/joakim/.local/bin/pyfunceble
    /home/joakim/.local/lib/python3.8/site-packages/PyFunceble-3.0.0.dist-info/*
    /home/joakim/.local/lib/python3.8/site-packages/PyFunceble/*
Proceed (y/n)? y
  Successfully uninstalled PyFunceble-3.0.0

$ sudo pip3 uninstall PyFunceble-dev PyFunceble
WARNING: Skipping PyFunceble-dev as it is not installed.
WARNING: Skipping PyFunceble as it is not installed.

$ sudo -H pip3 install PyFunceble-dev
Collecting PyFunceble-dev
  Using cached PyFunceble_dev-3.0.8-py3-none-any.whl (263 kB)
Requirement already satisfied: dnspython in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (1.16.0)
Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (0.4.3)
Requirement already satisfied: urllib3 in /usr/lib/python3/dist-packages (from PyFunceble-dev) (1.24.1)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (5.2)
Requirement already satisfied: python-dotenv in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (0.10.3)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from PyFunceble-dev) (41.1.0)
Requirement already satisfied: PyMySQL in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (0.9.3)
Requirement already satisfied: domain2idna>=1.6.1 in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (1.9.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from PyFunceble-dev) (2.21.0)
Requirement already satisfied: python-box in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (3.4.6)
Requirement already satisfied: cryptography in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev) (2.8)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.7/dist-packages (from cryptography->PyFunceble-dev) (1.13.2)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python3/dist-packages (from cryptography->PyFunceble-dev) (1.12.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi!=1.11.3,>=1.8->cryptography->PyFunceble-dev) (2.19)
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.0.8

$ PyFunceble --version
PyFunceble 3.0.8.dev (Teal Blauwbok)

$ pyfunceble -f tmp/t2.list --dns 127.0.0.1:5302

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

🌟 🎉 🍰 We now support IPv6! 🍰 🎉 🌟

Subject                                                                                              Status      HTTP Code 
---------------------------------------------------------------------------------------------------- ----------- ----------
asiansgoporn.com                                                                                     ACTIVE      301       
asianteengirls.net                                                                                   ACTIVE      200       
asian-teen-xxx.com                                                                                   ACTIVE      301       
asiantricks.com                                                                                      ACTIVE      200       
^C
Thanks for using PyFunceble!
Share your experience on Twitter with #PyFunceble!
Have a feedback, an issue or an improvement idea?
Let us know on GitHub!

So it seems that the dev is running on ubuntu 19.04 Eoan..... (In a fresh shell) nice

PS: What's new in 3.0.7, as you know I've been a bit of the last months :unamused: :face_with_thermometer:

funilrys commented 4 years ago

You should avoid the installation of pip-packages to root unless you only work from root. But that's another debate.

That's weird.

Here is the live "tracker" since 3.0.0: https://github.com/funilrys/PyFunceble/compare/dev

spirillen commented 4 years ago

The reason I chooses to install it with sudo (root) is to add the flexibility of having a multi-user access to the program, and rather than having to keep several accounts updated with the latest version... this is a way to do it.

That said you are right in general, such packages should be install per user account.

NOTE: This comment is for other users reading this thread.