Using Uri with Google Distance Matrix API (separator issue)

[olliefr]

I am using Uri to create a Google Distance Matrix request and i find it a bit awkward since the Google API uses pipe character "|" for separating parameter values but Uri uses commas "," and i could not find a way to override it:

utop # let x = Uri.of_string "http://www.github.com/";;
val x : Uri.t = <abstr>

utop # let y = Uri.add_query_param x ("origins", ["Bristol"; "Cambridge"; "London"]);;
val y : Uri.t = <abstr>

utop # Uri.to_string y;;
- : string = "http://www.github.com/?origins=Bristol,Cambridge,London"

Although this is RFC-compliant, it is not what the (silly) API expects.

I solved the problem by concatenating the values before passing them to Uri, which consequently performed the character escape on them. It worked with Google API, but it left me wonder – what if it would not (i.e. in case of a slightly more retarded web service)? Since Lwt and Cohttp require Uri.t, would that mean no OCaml for me in this project?

utop # let c = String.concat ~sep:"|" ["Bristol"; "Cambridge"; "Plymouth"; "London"];;
- : string = "Bristol|Cambridge|Plymouth|London"

utop # let y = Uri.add_query_param x ("origins", [c]);;
val y : Uri.t = <abstr>

utop # Uri.to_string y;;
- : string = "http://www.github.com/?origins=Bristol%7CCambridge%7CLondon"

I understand that the way Google API handles this is not RFC compliant but it is a real-world API, so maybe there is a scope for a compromise?

If Uri functions took an optional ~sep parameter, like Jane Street's Core.String.concat does, this would not be a problem.

What do you think?

[dbuenzli]

IIRC correctly RFC 3986 doesn't say anything about the structure of the query part of an URI so you can see these Uri.add_query_param as convenience function that will anyway never cover all uses. But I'd strongly advise against generating non RFC compliant URI, i.e. that possible sep separator should really be escaped if it needs to be.

In general if you think the API provider is not RFC compliant ask the provider to change not the RFC compliant implementation --- even if it's Google --- as it makes it worse for everyone otherwise. That being said, in that case, I don't think that this API is not RFC compliant (did you try a query without percent encoding ?) as it seems they expect you to URL encode your URI (cf. the comment "Distance Matrix API URLs are restricted to approximately 2000 characters, after URL encoding."). Let's say it's just unprecisely specified (or it's too obvious to these documentation writers): on paper you can write an URI with disallowed characters it's just that the standard mandates you to percent encode them when you want to feed them to a process that expects the URI non-terminal of RFC 3986 (see §1.2.1).

[olliefr]

@dbuenzli i just did telnet maps.googleapis.com 80 &> manual-request-output.txt and input

GET /maps/api/distancematrix/json?origins=Vancouver+BC|Seattle&destinations=San+Francisco|Victoria+BC&mode=bicycling&language=fr-FR&sensor=false

which yielded

Trying 2a00:1450:400c:c03::5f...
Connected to googleapis.l.google.com.
Escape character is '^]'.
HTTP/1.0 200 OK
Content-Type: application/json; charset=UTF-8
Date: Tue, 29 Oct 2013 17:44:55 GMT
Expires: Wed, 30 Oct 2013 17:44:55 GMT
Cache-Control: public, max-age=86400
Access-Control-Allow-Origin: *
Server: mafe
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic

{
   "destination_addresses" : [ "San Francisco, Californie, États-Unis", "Victoria, BC, Canada" ],
   "origin_addresses" : [ "Vancouver, BC, Canada", "Seattle, État de Washington, États-Unis" ],
   "rows" : [
      {
         "elements" : [
            {
               "distance" : {
                  "text" : "1 685 km",
                  "value" : 1685324
               },
               "duration" : {
                  "text" : "3 jours 22 heures",
                  "value" : 337245
               },
               "status" : "OK"
            },
            {
               "distance" : {
                  "text" : "136 km",
                  "value" : 135607
               },
               "duration" : {
                  "text" : "6 heures 38 minutes",
                  "value" : 23858
               },
               "status" : "OK"
            }
         ]
      },
      {
         "elements" : [
            {
               "distance" : {
                  "text" : "1 434 km",
                  "value" : 1434324
               },
               "duration" : {
                  "text" : "3 jours 7 heures",
                  "value" : 285258
               },
               "status" : "OK"
            },
            {
               "distance" : {
                  "text" : "146 km",
                  "value" : 146498
               },
               "duration" : {
                  "text" : "3 heures 10 minutes",
                  "value" : 11406
               },
               "status" : "OK"
            }
         ]
      }
   ],
   "status" : "OK"
}
Connection closed by foreign host.

So it did work and it appears that Google Distance Matrix works with both encoded and non-encoded uri's. My worry though that if it did not, i would have to abandon using OCaml for my project. I am not in position to ask Google anything.

[olliefr]

Oops, did not mean to close it! :)

[dbuenzli]

Le mardi, 29 octobre 2013 à 18:49, Ollie Frolovs a écrit :

So it did work and it appears that Google Distance Matrix works with both encoded and non-encoded uri's.

Well all is fine then, be liberal in what you accept (Google) and conservative in what you generate (ocaml-uri). Contrary to what you suggest on the mailing list there's no violation of the RFC here.

I am not in position to ask Google anything.

Why not ? Do they know too much about you ?

Daniel

[olliefr]

I was told by someone on the mailing list that it violates the RFC. If it is not, great.

In regards to Google, i do not know if it was an attempt at making a joke but i did not find it funny. If this was a genuine question – considering the fact that Google is a large company i do not believe that they are genuinely interested in what i think. Also, the time constraints that i have would make the exercise rather pointless. It is easier to write a two-liner in Python than go through all this. I raised the issue here because it came up in a real-world application i am writing and David Sheets from the mailing list has kindly encouraged me to log this issue here.

So i am leaving my suggestion here as an open issue for the Uri authors to consider.

[dbuenzli]

Le mardi, 29 octobre 2013 à 22:35, Ollie Frolovs a écrit :

I was told by someone on the mailing list that it violates the RFC. If it is not, great.

It's hard to violate anything when things are not formally specified, the bits of documentation you linked to are not sufficient to say anything about that matter. But if they would say the API accepts only RFC 3986 uri and did not accept percent encoded ones that would be a clear violation.

In regards to Google, i do not know if it was an attempt at making a joke but i did not find it funny.

Sorry, apparently we don't have the same sense of humor.

If this was a genuine question – considering the fact that Google is a large company i do not believe that they are genuinely interested in what i think.

What you think is not the point. The point is about notifying entities --- big or small --- about standards violations if they pretend to adhere to them. Adjusting standards implementations to real world quirks may sometimes be needed but should be avoided at all costs, it defeats the purpose of a standard and makes our programming lives harder. But in that case, as mentioned above, there's hardly anything to say about it. You could however suggest them to improve their documentation.

It is easier to write a two-liner in Python than go through all this. This is not a language problem. You'd have exactly the same interrogations with your two-liner python program.

Daniel