Support downloading feeds with HTTP basic authentication

[barbeau]

Summary:

I requested an API key for Metra in Chicago as part of https://github.com/CUTR-at-USF/transit-feed-quality-calculator/issues/29, and I found out that they use HTTP basic authentication for access credentials - they gave me a username and password.

Currently, we only support a URL via the CSVDownloader, which can include an API key embedded in the URL as an HTTP request parameter. However, basic authentication is accomplished via setting the header, which we currently don't support.

So, we should add an option for HTTP basic authentication for the CsvDownloader via a username and password.

Here's a Java example: https://stackoverflow.com/a/7020054/937715

Here is their developer site (which as far as I can tell doesn't document this anywhere - I had to ask via email): https://metrarail.com/developers

Here is the sample code they gave me:

// Example with C# with WebRequest:
var webRequest = WebRequest.Create(uri);
webRequest.Credentials = new NetworkCredential("userName", "password");
using (var webResponse = webRequest.GetResponse())
{
    using (var responseStream = webResponse.GetResponseStream())
    {
        return new StreamReader(responseStream).ReadToEnd();
    }
}

// Example with PHP using CURL:
$login = 'login';
$password = 'password';
$url = 'http://your.url';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_USERPWD, "$login:$password");
$result = curl_exec($ch);
curl_close($ch);  
echo($result);

Steps to reproduce:

1. Try to download Metra's feeds:

Host https://gtfsapi.metrarail.com

Raw GTFS Feeds • GET - /gtfs/raw/positionUpdates.dat • GET - /gtfs/raw/tripUpdates.dat • GET - /gtfs/raw/alerts.dat • GET - /gtfs/raw/schedule.zip

JSON GTFS Feeds • /gtfs/alerts • /gtfs/positions • /gtfs/tripUpdates

Additional API Information https://metrarail.com/developers/metra-gtfs-api

Expected behavior:

Give me a way to download the feeds using a username and password

Observed behavior:

You can't download feeds if the server requires a username and password via HTTP basic authentication

Platform:

N/A

[laidig]

Sydney / NSW use a key in the request headers, and I've hacked together support in my gtfs-rt-printer for both this and basic auth (which is also sent via headers).

docs: https://opendata.transport.nsw.gov.au/node/328/exploreapi#!/nswtrains/GetNSWTrains

curl example: curl -X GET --header 'Accept: application/x-google-protobuf' --header 'Authorization: apikey ...' 'https://api.transport.nsw.gov.au/v1/gtfs/realtime/nswtrains'

feel free to borrow code for these auth schemes from here: https://github.com/laidig/gtfs-rt-printer/blob/master/src/main/java/net/transitdata/gtfsrt/StreamUtils.java#L59

[barbeau]

Thanks @laidig!