search

caarlos0 / domain_exporter

Exports the expiration time of your domains as prometheus metrics.
MIT License
265 stars 54 forks source link

could not parse whois response from .de tld #92

Closed UstyuzhaninAV closed 1 year ago

UstyuzhaninAV commented 3 years ago

can't parse two domain holaco.de adspro.eu

fbsduser commented 3 years ago

Expiration date parsing error ERRO[0497] failed to probe "my_domain": could not parse date: -date: 2021-09-20 source="domain.go:65"

caarlos0 commented 3 years ago

.de TLD does not return any expiration times on the domains I tested at least... not much we can do...

fbsduser commented 3 years ago

So. In internal/whois/whois.go if you have returned \t in expiration date/time your domain like i had, you should separate or remove "\t" to work fine your code, and if you put that code dateStr = after(dateStr, "-date:\t") right after line "71" It will works fine. And after that i'm trying to build a container of that, and i got error says can not copy apk to /tmp/ even i build with your dockerfile. Thanks for reply.

caarlos0 commented 3 years ago

you can test building using dockerfile.dev

caarlos0 commented 3 years ago

also go test -v ./... will test it (and you can add a case specific for this problem as well)

UstyuzhaninAV commented 3 years ago

.de TLD does not return any expiration times on the domains I tested at least... not much we can do...

Some register can explain expiration date, many possible add it for default check?

namecheap & godaddy

example: whois -h whois.godaddy.com adspro.eu Domain Name: adspro.eu Registry Domain ID: adspro_eu-EURID Registrar WHOIS Server: whois.godaddy.com Registrar URL: http://www.godaddy.com Updated Date: 2020-08-20T13:06:45Z Creation Date: 2018-01-30T01:16:22Z Registrar Registration Expiration Date: 2022-01-30T22:59:59Z

GoDaddy
Make Your Own Way | GoDaddy
All the help and tools you need to grow online: Websites, Domains, Digital and Social Marketing - plus GoDaddy Guides with you every step of the way
ethaniel86 commented 3 years ago

same for certain extensions ie. .vn, returning value -1 nice if there's a config to customize whois server based on extension :) for example .de$ whois.godaddy.com

caarlos0 commented 3 years ago

Probably need to improve this RE https://github.com/caarlos0/domain_exporter/blob/master/internal/whois/whois.go#L52

so this happens: https://github.com/caarlos0/domain_exporter/blob/master/internal/whois/whois.go#L112-L115

GitHub
caarlos0/domain_exporter
Exports the expiration time of your domains as prometheus metrics. - caarlos0/domain_exporter
GitHub
caarlos0/domain_exporter
Exports the expiration time of your domains as prometheus metrics. - caarlos0/domain_exporter
la210 commented 3 years ago

I have the same issue with .kz domains Maybe adding specific whois servers to some domain zones will be a good idea.

daenney commented 3 years ago

It's not really possible to do overrides per TLD. For example, if you use whois.godaddy.com to query for europa.eu you'll get no match since GoDaddy is not its registrar (it's Proximus/Belgacom, which doesn't run a whois server so you're limited to the output EURid provides).

The same goes for other TLDs. For example the GoDaddy whois server won't respond for spiegel.de. Their registrar is cloudns, which doesn't seem to have a functional whois server either so you're limited to DENICs response which doesn't include expiration/renewal info.

In most cases, whois can determine through the IANA database if there's a more specific server that can answer the query, and will return it as a refer if you query IANA. However, these point to the whois servers of TLD registry operators, and not domain registrars (which makes sense since the latter can change often and dynamically).

$ whois -h whois.iana.org adspro.eu
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object

refer:        whois.eu

Most of these whois servers don't return much past the creation/registration of the domain or the last modification of the domain (which doesn't tell you much).

Using whois works pretty well for .net/.com/.org, but it's going to be very hit and miss for other TLDs. You'll need the ability to override the whois server used by registered domain, and hope that the registrar in question runs a whois server and that their whois server happens to expose this piece of data.

caarlos0 commented 3 years ago

maybe accept a ?whois query param would do the trick?

daenney commented 3 years ago

That would do the trick. Folks will just have to ensure they pick a TLD or registrar that makes that information available this way. Otherwise you'd have to integrate with registrar APIs directly.

AkashSivakumar-Dev commented 3 years ago

I am getting -1 as value for .in extensions. Can anyone please help me with this? Thanks in advance.

D13410N3 commented 2 years ago

Hello. Error parsing "ch" and "es" domains

11:29AM ERR failed to probe ***.ch error="could not parse whois response: \"Requests of this client are not permitted. Please use https://www.nic.ch/whois/ for queries.\\n\\n\""
ERR failed to probe ***.es error="could not parse whois response: \"\\r\\n    Conditions of use for the whois service via port 43 for .es domains\\r\\n\\r\\n    Access will only be enabled for  IP addresses  authorised  by Red.es.  A maximum of one  IP address per\\r\\n    user/organisation is permitted.\\r\\n\\r\\n    Red.es accepts  no responsibility  whatsoever  for  the availability  of access to WHOIS,  which may be\\r\\n    suspended at any time and without prior warning at the discretion of the public entity.\\r\\n\\r\\n    The service will be limited to the data established by Red.es.\\r\\n\\r\\n    The user  promises  to make use of the service and to  carry out any action derived  from the aforesaid\\r\\n    use in accordance with  current applicable  regulations, in particular with legislation on “.es” domain\\r\\n    names and personal data protection.\\r\\n\\r\\n    In particular, the user undertakes not to use  the service  to carry out abusive  or speculative domain\\r\\n    name registrations, pursuant to section 5 of the Sixth Additional Provision of Law 34/2002, of 11 July,\\r\\n    on Services of the  Information  Society and  Electronic Commerce. Likewise, the User undertakes not to\\r\\n    use the service to  obtain data, the possession  of which may  contravene the provisions of Organic Law\\r\\n    15/1999,  of  13 December,  on Personal Data Protection, and  its Regulations, or in Law 34/2002, of 11\\r\\n    July, on Services of the Information Society and Electronic Commerce.\\r\\n\\r\\n    Failure  to comply with these conditions will result in the immediate withdrawal of the service and any\\r\\n    registered domain name which breaches said conditions may be officially cancelled by Red.es.\\r\\n    -------------------------------------------------------------------------------------------------------\\r\\r\\n\\r\\n    The IP address used to perform the query  is not authorised  or  has exceeded the established limit for\\r\\n    queries.To request access to the service,complete the form located at https://sede.red.gob.es/sede/whois,\\r\\n    where you may also consult the service conditions.\\r\\n\\r\\n    -------------------------------------------------------------------------------------------------------\\r\\n    More information on each domain may be consulted at www.dominios.es.\\r\\n\\x00\""
daenney commented 2 years ago

By default for *.ch, you'll hit nic.ch, but they don't allow requests using the WHOIS protocol.

$ whois nic.ch
Requests of this client are not permitted. Please use https://www.nic.ch/whois/ for queries.

This won't work for any .ch domain by default, unless the registrar of the particular .ch domain you're interested in runs a WHOIS server you can use instead.

daenney commented 2 years ago

For Spanish domains, you'll have the same issue. WHOIS is no longer available for the TLD, unless you the registrar still runs one:

$ whois thelocal.es
This TLD has no whois server, but you can access the whois database at
https://www.nic.es/

You can get slightly more info with:

$ whois -h whois.nic.es thelocal.es

    Conditions of use for the whois service via port 43 for .es domains

    Access will only be enabled for  IP addresses  authorised  by Red.es.  A maximum of one  IP address per
    user/organisation is permitted.

    Red.es accepts  no responsibility  whatsoever  for  the availability  of access to WHOIS,  which may be
    suspended at any time and without prior warning at the discretion of the public entity.

    The service will be limited to the data established by Red.es.

    The user  promises  to make use of the service and to  carry out any action derived  from the aforesaid
    use in accordance with  current applicable  regulations, in particular with legislation on “.es” domain
    names and personal data protection.

    In particular, the user undertakes not to use  the service  to carry out abusive  or speculative domain
    name registrations, pursuant to section 5 of the Sixth Additional Provision of Law 34/2002, of 11 July,
    on Services of the  Information  Society and  Electronic Commerce. Likewise, the User undertakes not to
    use the service to  obtain data, the possession  of which may  contravene the provisions of Organic Law
    15/1999,  of  13 December,  on Personal Data Protection, and  its Regulations, or in Law 34/2002, of 11
    July, on Services of the Information Society and Electronic Commerce.

    Failure  to comply with these conditions will result in the immediate withdrawal of the service and any
    registered domain name which breaches said conditions may be officially cancelled by Red.es.
    -------------------------------------------------------------------------------------------------------

    The IP address used to perform the query  is not authorised  or  has exceeded the established limit for
    queries.To request access to the service,complete the form located at https://sede.red.gob.es/sede/whois,
    where you may also consult the service conditions.

    -------------------------------------------------------------------------------------------------------
    More information on each domain may be consulted at www.dominios.es.

Basically, you need to get whitelisted.

case commented 2 years ago

Fwiw, Switch (which operates .ch and .li) has an RDAP server:

For some reason it isn't listed in the IANA bootstrap data — https://data.iana.org/rdap/dns.json — but it's here in a different project's overrides file:

https://github.com/meeb/whoisit/blob/main/whoisit/overrides.py

serhatcetinkaya commented 1 year ago

we are having the same problem with .es TLD.

we also noticed if we use godaddy whois servers (whois.godaddy.com) we are able to get a proper response:

Click to see cli output ### output ```text ➜ ~ whois -h whois.godaddy.com $REDACTED_DOMAIN_NAME Domain Name: $REDACTED_DOMAIN_NAME Registry Domain ID: 13669014911570_DOMAIN-KEYSYS Registrar WHOIS Server: whois.godaddy.com Registrar URL: https://www.godaddy.com Updated Date: 2023-01-22T11:12:14Z Creation Date: 2012-02-17T14:38:00Z Registrar Registration Expiration Date: 2024-02-17T12:12:14Z Registrar: GoDaddy.com, LLC Registrar IANA ID: 146 Registrar Abuse Contact Email: abuse@godaddy.com Registrar Abuse Contact Phone: +1.4806242505 Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited Domain Status: clientRenewProhibited https://icann.org/epp#clientRenewProhibited Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited Registrant Organization: Registrant State/Province: NONE Registrant Country: XX Registrant Email: Select Contact Domain Holder link at https://www.godaddy.com/whois/results.aspx?domain=$REDACTED_DOMAIN_NAME Admin Email: Select Contact Domain Holder link at https://www.godaddy.com/whois/results.aspx?domain=$REDACTED_DOMAIN_NAME Tech Email: Select Contact Domain Holder link at https://www.godaddy.com/whois/results.aspx?domain=$REDACTED_DOMAIN_NAME Name Server: $REDACTED_NS Name Server: $REDACTED_NS Name Server: $REDACTED_NS Name Server: $REDACTED_NS DNSSEC: unsigned URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/ >>> Last update of WHOIS database: 2023-03-01T13:19:11Z <<< ➜ ~ whois $REDACTED_DOMAIN_NAME % IANA WHOIS server % for more information on IANA, visit http://www.iana.org % This query returned 1 object refer: whois.nic.es domain: ES organisation: Red.es address: Edificio Bronce address: Plaza Manuel Gomez Moreno address: Madrid 28020 address: Spain contact: administrative name: Alberto Martinez Lacambra organisation: Red.es address: Edificio Bronce address: Plaza Manuel Gomez Moreno address: Madrid 28020 address: Spain phone: +34 91 212 76 24 fax-no: +34 91 555 76 64 e-mail: esnic-admin@red.es contact: technical name: Juan Antonio Gutierrez Gil organisation: Red.es address: Edificio Bronce address: Plaza Manuel Gomez Moreno address: Madrid 28020 address: Spain phone: +34 91 212 76 20 fax-no: + 34 91 556 88 64 e-mail: esnic-tech@red.es nserver: A.NIC.ES 194.69.254.1 2001:67c:21cc:2000:0:0:64:41 nserver: C.NIC.ES 194.0.34.53 2001:678:44:0:0:0:0:53 nserver: G.NIC.ES 2001:500:14:7001:ad:0:0:1 204.61.217.1 nserver: H.NIC.ES 194.0.33.53 2001:678:40:0:0:0:0:53 ds-rdata: 44375 8 2 9d9858ae981aa53dd1143d93844e3d69b0fb73a9b4fe5759da39e036e754d402 whois: whois.nic.es status: ACTIVE remarks: Registration information: http://www.nic.es/ created: 1988-04-14 changed: 2022-12-19 source: IANA # whois.nic.es Conditions of use for the whois service via port 43 for .es domains Access will only be enabled for IP addresses authorised by Red.es. A maximum of one IP address per user/organisation is permitted. Red.es accepts no responsibility whatsoever for the availability of access to WHOIS, which may be suspended at any time and without prior warning at the discretion of the public entity. The service will be limited to the data established by Red.es. The user promises to make use of the service and to carry out any action derived from the aforesaid use in accordance with current applicable regulations, in particular with legislation on “.es” domain names and personal data protection. In particular, the user undertakes not to use the service to carry out abusive or speculative domain name registrations, pursuant to section 5 of the Sixth Additional Provision of Law 34/2002, of 11 July, on Services of the Information Society and Electronic Commerce. Likewise, the User undertakes not to use the service to obtain data, the possession of which may contravene the provisions of Organic Law 15/1999, of 13 December, on Personal Data Protection, and its Regulations, or in Law 34/2002, of 11 July, on Services of the Information Society and Electronic Commerce. Failure to comply with these conditions will result in the immediate withdrawal of the service and any registered domain name which breaches said conditions may be officially cancelled by Red.es. ------------------------------------------------------------------------------------------------------- The IP address used to perform the query is not authorised or has exceeded the established limit for queries.To request access to the service,complete the form located at https://sede.red.gob.es/sede/whois, where you may also consult the service conditions. ------------------------------------------------------------------------------------------------------- More information on each domain may be consulted at www.dominios.es. ```

180 is created as a solution for this

caarlos0 commented 1 year ago

on next version you'll be able to set a custom whois server per-domain.