Maintain a copy of 'whois' program database properly

relrod / whois-hs

A whois library for Haskell.

BSD 3-Clause "New" or "Revised" License

5 stars 4 forks source link

Maintain a copy of 'whois' program database properly #10

Open sshine opened 4 years ago

sshine commented 4 years ago

The data/ directory contains a WHOIS server database exported from the whois program.

[x] Write a data/README.md that explains the meaning of each of these files, what some of the keywords they use mean and how they're encoded in the library.
[ ] Come up with a way that notifies when a new version of these files is made available. (It'd be super cool if GitHub Actions or similar allowed auto-submitting a PR to this repository if a change is made to the other repository.)
[ ] Transform the compatibility tests of #9 into regular unit/prop tests for the non-IO parts and network-bound integration tests for the IO parts. Expand on the design of the integration tests first when we reach this far.

relrod commented 4 years ago

We should also confirm licensing and ensure we are able to redistribute it. :)

sshine commented 4 years ago

We should also confirm licensing and ensure we are able to redistribute it. :)

whois is GPL 2+:

https://github.com/rfc1036/whois/blob/next/whois.c

I assume that his database files are as well, but I'll enquire about that. I've meant to contact Marco anyways about some details of his implementation.

sshine commented 4 years ago

So according to rfc1036/whois#82,

Yes, they are released with the same license of the code, as much as somebody can claim copyright on a database (this depends on your jurisdiction). And yes, tld_serv_list is the most accurate database of its kind, better than IANA. :-)

As for IANA's tlds-alpha-by-domain.txt, I assume it is redistributable.

But I've written IANA and asked them, since it's not completely clear.

That way we can be clear about it.

sshine commented 4 years ago

A response from IANA says these are public domain,

The data made available through https://data.iana.org/TLD/tlds-alpha-by-domain.txt is in the public domain and has no license conditions. It is derived daily from the contents of the root zone.

As such, the data is dynamic and changes over time. Any implementation approach that takes a single version of the file and embeds it in a distribution will become quickly outdated. The implementation should take into an account an update mechanism that fetches the most recent version (e.g. daily).

For the reason explained here, I've chosen to go with an approach that does not involve compiling this file into the executable as explained in 6681a76a8803b9a643b9026cc83b21f3e2efd519. The particular approach is, of course, subject to discussion and change, but for now this just means that "new" gTLDs aren't stored unless they're in tld_serv_list for some reason, which is usually exceptional.