search

pelias / whosonfirst

Importer for Who's on First gazetteer
MIT License
28 stars 43 forks source link

add Pelias UserAgent to cURL requests #515

Closed missinglink closed 4 years ago

missinglink commented 4 years ago

The Geocode Earth CDN has become fairly popular, we served >15TB to Europe alone last month :rocket:

I'd like to have a little more information about who's using the CDN as I suspect that there are some systems out there downloading the data over-and-over-again in automated scripts.

This PR is pretty simple, it just adds an HTTP User Agent string to download requests made from Pelias tooling so we can distinguish the Pelias traffic from the non-Pelias traffic.

Here's an example from the logs:

Screenshot 2020-07-21 at 10 45 58

note: I unfortunately had to break the long line for the return statement in generateCommand() because the linter complained, I made it an Array to get around that but it still seems wrong to me..

Screenshot 2020-07-21 at 10 52 33
orangejulius commented 4 years ago

Looks good, this will definitely be helpful. I wonder if there's a way to include a more useful version number in the "user agent".

missinglink commented 4 years ago

Yeah good question, the 0.0.0-development version is not ideal, it's an unfortunate downside of using semantic-release. I wasn't able to think of an easy way of overcoming that problem, but if we do in the future this should 'just work'.

In the interim this still solves my issue as the software is identifiable even if the correct version isn't.