search

Jigsaw-Code / outline-server

Outline Server, developed by Jigsaw. The Outline Server is a proxy server that runs a Shadowsocks instance and provides a REST API for access key management.
https://getoutline.org/
Apache License 2.0
5.84k stars 789 forks source link

Add support for IPinfo's IP to Country ASN database #1614

Open abdullahdevrel opened 1 month ago

abdullahdevrel commented 1 month ago

I am requesting to add support for IPinfo's IP to Country database to the project. The database has the following features:

Database schema

Field Name Example Data Type Description
start_ip 1.0.16.0 TEXT Starting IP address of an IP address range
end_ip 1.0.31.255 TEXT Ending IP address of an IP address range
country JP TEXT ISO 3166 country code of the location
country_name Japan TEXT Name of the country
continent AS TEXT Continent code of the country
continent_name Asia TEXT Name of the continent
asn AS2519 TEXT Autonomous System Number
as_name ARTERIA Networks Corporation TEXT Name of the AS (Autonomous System) organization
as_domain arteria-net.com TEXT Official domain or website of the AS organization

Documentation: https://ipinfo.io/developers/ip-to-country-asn-database

Samples are available here: https://github.com/ipinfo/sample-database/tree/main/IP%20to%20Country%20ASN

The database can be downloaded simply by accessing the storage URI with an access token.

curl -L https://ipinfo.io/data/free/country_asn.mmdb?token=<YOUR_TOKEN> -o country_asn.mmdb

Let me know what you think. Thanks!

maddyhof commented 3 weeks ago

Thanks for reaching out! Can you share more about what capabilities you're looking for that are not already supported? For example, does this IPinfo database provide information that we don't currently offer, or is there some other reason you're interested in an alternative data source?

abdullahdevrel commented 3 weeks ago

Hey MaddyHof,

I work for IPinfo by the way. I really appreciate hearing from you.

The intention is to replace the current database here: https://github.com/Jigsaw-Code/outline-server/blob/e416938950109178462ac92ec18ef4bc2daa6fab/src/shadowbox/scripts/update_mmdb.sh#L30-L39

You are downloading a free IP database that is of reduced quality and accuracy. We can systematically prove that the IPinfo's free database is significantly more reliable and accurate than your current database provider. Our data is backed by active measurement providing the highest possible geolocation accuracy in the industry. The current database you are using is updated monthly while ours is updated daily.

The project already uses our API service, which we are very glad to see, and we thought the project deserves more better things like a free IP to Country database. It provides full accuracy, daily updates, and absolute reliability. On the licensing side, it is CC-BY-SA 4.0 (the same license as Wikipedia and Stack Overflow), which even permits commercial distribution by only providing an attribution.

Considering I am pitching a drop-in replacement here, I requested to use the IP to Country database, but I also noticed that you are using an ASN database: https://github.com/Jigsaw-Code/outline-server/blob/e416938950109178462ac92ec18ef4bc2daa6fab/src/shadowbox/server/main.ts#L46

In that case, you should use our free IP to Country ASN database. It includes both country and ASN data in a single database, so you do not need to maintain 2 separate databases for country and ASN. You can just use the one IP to Country ASN database and retrieve both country and ASN information from there: https://ipinfo.io/developers/ip-to-country-asn-database

The promise of full accuracy and daily updates remains the same as well.

Let me know what you think. I highly recommend downloading the database and exploring it a bit and sharing your thoughts with me. Looking forward to hearing from you.

fortuna commented 2 weeks ago

@abdullahdevrel it seems one needs an API TOKEN to fetch the db, which makes the deployment flow way more complex, since one would have to create an ipinfo account first.

Is that not the case?

fortuna commented 2 weeks ago

BTW, with outline-ss-server one can use whatever mmdb file they want.

We could migrate from dbip to ipinfo by default, but that requires someone spending time evaluating the change, which we unfortunately don't have, since this is not a priority compared to other tasks.

abdullahdevrel commented 2 weeks ago

@fortuna

it seems one needs an API TOKEN to fetch the db, which makes the deployment flow way more complex, since one would have to create an ipinfo account first.

You only need to create a single IPinfo account and ship the access token with the codebase, making it a repo-specific IPinfo token. Projects like Akvorado and Librespeed do this, allowing users to utilize the repo's access token without needing their account.

If you're wondering why we make this data so accessible, we’re dedicated to supporting Open-Source projects like Outline. We want to ensure that adopting our database is as easy as possible, even allowing for commercial distribution.

We could migrate from dbip to ipinfo by default, but that requires someone to evaluate the change, which we unfortunately don't have, since this is not a priority compared to other tasks.

I understand that implementing this is a request, not a priority, and I truly appreciate your consideration. We're confident that our database offers top-notch accuracy, data versatility (country + ASN) and reliability, making it a perfect fit for Outline.

I hope you’ll consider the migration whenever the timing works. In any case, I’m glad to have introduced our database as an option for future integration!

fortuna commented 2 weeks ago

Allowing use to have the api token be public is helpful, but we will probably have to have a legal review of the terms of service to ensure we are not violating them by doing so.

abdullahdevrel commented 2 weeks ago

@fortuna

Allowing use to have the api token be public is helpful, but we will probably have to have a legal review of the terms of service to ensure we are not violating them by doing so.

That is a really good point.

So, many providers actually have something called an EULA (End User License Agreement) that is added on top of the CC-BY-SA 4.0 license. Essentially, it nullifies the free distribution aspects of the free database. These providers actually charge money for free data and database distribution as part of their commercial distribution license that costs thousands.

image

But we are not like them!

We do not have an EULA. It is purely CC-BY-SA 4.0! Just give us an attribution of using our data that's all.

We recommend adding the attribution to the repo README.md or the website. Simply adding the following should be enough:

IP address data powered by IPinfo

image

Let me know what you think.