search

funilrys / PyFunceble

The tool to check the availability or syntax of domain, IP or URL.
https://pyfunceble.github.io
Apache License 2.0
298 stars 45 forks source link

Error in INVALID output #61

Closed spirillen closed 4 years ago

spirillen commented 4 years ago

Bug description

Found a few misses in the INVALID/list after testing https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/domains%20to%20check.txt

cat output/domains/INVALID/list
# File generated by PyFunceble (v2.45.0.dev) / https://github.com/funilrys/PyFunceble
# Date of generation: 2019-11-18T01:14:07.846084

a
added
and
checked
collectionofbestporn
command,
curl
did
double
file
hosts
list
needed.
not
of
properly
respond
the
which
will
www.adultcashtraffic.com
www.blog.gfrevenge.com
www.hqporner.comstudiowow-girls
www.largehdtube.comen
www.media.the-adult-company.com
www.pornblade.comcategoryanal-porn
www.porndoepremium.comcategories
www.sexdating
www.spankbang
yallainternethotnig

The errors I see is:

www.adultcashtraffic.com
www.blog.gfrevenge.com
www.media.the-adult-company.com
gfrevenge.com
   Domain Name: GFREVENGE.COM
   Registry Domain ID: 1312074137_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.eurodns.com
   Registrar URL: http://www.EuroDNS.com
   Updated Date: 2019-10-27T04:31:37Z
   Creation Date: 2007-11-02T00:31:15Z
   Registry Expiry Date: 2020-11-02T00:31:15Z
   Registrar: EuroDNS S.A.
   Registrar IANA ID: 1052
   Registrar Abuse Contact Email: legal@eurodns.com
   Registrar Abuse Contact Phone: +352.27220150
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Name Server: DNS1.P03.NSONE.NET
   Name Server: DNS2.P03.NSONE.NET
   Name Server: DNS3.P03.NSONE.NET
   Name Server: DNS4.P03.NSONE.NET
   Name Server: SDNS3.ULTRADNS.BIZ
   Name Server: SDNS3.ULTRADNS.COM
   Name Server: SDNS3.ULTRADNS.NET
   Name Server: SDNS3.ULTRADNS.ORG
   DNSSEC: unsigned
whois adultcashtraffic.com
Expired 
whois the-adult-company.com

   Domain Name: THE-ADULT-COMPANY.COM
   Registry Domain ID: 1370101655_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.safenames.net
   Registrar URL: http://www.safenames.net
   Updated Date: 2019-08-04T05:05:18Z
   Creation Date: 2008-01-03T13:29:55Z
   Registry Expiry Date: 2021-01-03T13:29:55Z
   Registrar: SafeNames Ltd.
   Registrar IANA ID: 447
   Registrar Abuse Contact Email: abuse@safenames.net
   Registrar Abuse Contact Phone: +44.1908200022
   Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
   Name Server: NS1.XMODELS-LIVE.CH
   Name Server: NS2.XMODELS-LIVE.CH
   Name Server: NS3.XMODELS-LIVE.CH
   DNSSEC: unsigned

This means two out of 3 should have been added to the ACTIVE/list

Reproduction

pyfunceble --plain -h -m -p 4 -db --database-type mariadb -f 'https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/domains%20to%20check.txt'

Expected behavior**

Only invalid entries, or invalid tld's are added to invalid/list

Versions

OS: Disco Python Version: 3.7.3 PyFunceble Version: pyfunceble -v pyfunceble 2.45.0.dev (Green Galago: Skitterbug)

Additional context

Have you seen issue at gitlab?

spirillen commented 4 years ago 
printf "www.adultcashtraffic.com\nwww.media.the-adult-company.com\n" >> output/domains/ACTIVE/list 
printf "www.adultcashtraffic.com\n" >> output/domains/INACTIVE/list
funilrys commented 4 years ago

Look at example.org :wink: I refuse to do the work of the Registry. That's why I don't necessarily ensure that the date in the past/future if it is not archived in the database.

Indeed, we both know that example.org is a special case but who knows how many "special cases" are there in the wild?

Therefor I'm not going to improve that. It will stay like that since I consider that if a Registry still provides some expiration date, it implicitly means that the subject of the record may be active.

About www.blog.gfrevenge.com, I don't bother with the WHOIS lookup as it is a subdomain. It's like that since D1 I'm not sure I'm ready to put some time into that at it will make things slower than they actually are... But I'm open to implementing that as a feature/option later.

(Could you imagine how many sub-domains are tested per hours by PyFunceble in @dead-hosts @ultimate-hosts-blacklist @Import-External-Sources ?)

What is strange is that I don't get the same results :thinking: ``` + cat issues/61.list a added and checked collectionofbestporn command, curl did double file hosts list needed. not of properly respond the which will www.adultcashtraffic.com www.blog.gfrevenge.com www.hqporner.comstudiowow-girls www.largehdtube.comen www.media.the-adult-company.com www.pornblade.comcategoryanal-porn www.porndoepremium.comcategories www.sexdating www.spankbang yallainternethotnig + PyFunceble --version PyFunceble 2.46.0.dev (Green Galago: Skitterbug) + PyFunceble --plain -m -db --database-type=mariadb -f issues/61.list ██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗ ██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝ ██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗ ██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝ ██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗ ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝ Your version is more recent! You should really think about sharing your changes with the community! Your version: 2.46.0.dev (Green Galago: Skitterbug) Upstream version: 2.45.0.dev (Green Galago: Skitterbug) 🌟 🎉 🍰 We now support IPv6! 🍰 🎉 🌟 Subject Status HTTP Code ---------------------------------------------------------------------------------------------------- ----------- ---------- added INVALID XXX command, INVALID XXX did INVALID XXX curl INVALID XXX checked INVALID XXX a INVALID XXX and INVALID XXX needed. INVALID XXX file INVALID XXX collectionofbestporn INVALID XXX double INVALID XXX list INVALID XXX hosts INVALID XXX not INVALID XXX the INVALID XXX of INVALID XXX properly INVALID XXX which INVALID XXX will INVALID XXX www.adultcashtraffic.com INACTIVE XXX www.largehdtube.comen INVALID XXX www.hqporner.comstudiowow-girls INVALID XXX respond INVALID XXX www.sexdating INVALID XXX www.porndoepremium.comcategories INVALID XXX www.pornblade.comcategoryanal-porn INVALID XXX www.media.the-adult-company.com INACTIVE XXX www.blog.gfrevenge.com INACTIVE XXX www.spankbang INVALID XXX yallainternethotnig INVALID XXX Status Percentage Numbers ----------- ------------ ------------ ACTIVE 0% 0 INACTIVE 10% 3 INVALID 90% 27 ██████╗ ██╗ ██╗███████╗██╗ ██╗███╗ ██╗ ██████╗███████╗██████╗ ██╗ ███████╗ ██╔══██╗╚██╗ ██╔╝██╔════╝██║ ██║████╗ ██║██╔════╝██╔════╝██╔══██╗██║ ██╔════╝ ██████╔╝ ╚████╔╝ █████╗ ██║ ██║██╔██╗ ██║██║ █████╗ ██████╔╝██║ █████╗ ██╔═══╝ ╚██╔╝ ██╔══╝ ██║ ██║██║╚██╗██║██║ ██╔══╝ ██╔══██╗██║ ██╔══╝ ██║ ██║ ██║ ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗ ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝ Thanks for using PyFunceble! + cat output/domains/INVALID/list # File generated by PyFunceble (v2.46.0.dev) / https://github.com/funilrys/PyFunceble # Date of generation: 2019-11-20T00:44:19.410784 a added and checked collectionofbestporn command, curl did double file hosts list needed. not of properly respond the which will www.hqporner.comstudiowow-girls www.largehdtube.comen www.pornblade.comcategoryanal-porn www.porndoepremium.comcategories www.sexdating www.spankbang yallainternethotnig + cat output/domains/INACTIVE/list # File generated by PyFunceble (v2.46.0.dev) / https://github.com/funilrys/PyFunceble # Date of generation: 2019-11-20T00:44:19.406948 www.adultcashtraffic.com www.blog.gfrevenge.com www.media.the-adult-company.com + cat output/domains/ACTIVE/list cat: output/domains/ACTIVE/list: Datei oder Verzeichnis nicht gefunden ```

P.S: For those who came late to the party: example.org have - at the time I write this - an expiration date set to 2010-08-30T04:00:00Z

dnmTX commented 4 years ago

@funilrys this is not a bug. That's why we testing each domain on whom www. is added as many are none existing and therefore the complaint here is irrelevant:

Capture

Just ignore that issue....or better yet...CLOSE IT 😈

spirillen commented 4 years ago

@funilrys I see your point... That's the good part of open thought, one remember what another didn't think of :thinking:

Building a list of exceptions, that would be quit a work.....

I put put my head in work to see if something usefully strikes me...

(Could you imagine how many sub-domains are tested per hours by PyFunceble in)

hrm hrm tooooo :speak_no_evil: :roll_eyes: :rofl:

Retest

retest results ``` # File generated by PyFunceble (v2.45.0.dev) / https://github.com/funilrys/PyFunceble # Date of generation: 2019-11-20T01:08:02.246762 a added and checked collectionofbestporn command, curl did double file hosts list needed. not of properly respond the which will www.hqporner.comstudiowow-girls www.largehdtube.comen www.pornblade.comcategoryanal-porn www.porndoepremium.comcategories www.sexdating www.spankbang www.spankbanglistaaa ```

Same as your on 3rd run....