Incorrect API URLs in all the manuals

rkiyanchuk commented 9 years ago

Hello, guys!

I just faced issues with communicating to YiDB API via shell tools like curl and httpie when deployed via Tomcat server. I believe it's due to either YiDB's or Tomcat's noncompliance to RFC 3986.

When I send query to the server according to the YiDB documentation, I get 500 error from Tomcat:

$ http http://localhost:9090/cms/repositories/Asset/branches/main/query/Server[@status="Validated"].asset{*}.rack[@name="F2"]                                                                                                                                                 
HTTP/1.1 500 Internal Server Error
Connection: close
Content-Length: 2135
Content-Type: text/html;charset=utf-8
Date: Wed, 21 Jan 2015 19:22:50 GMT
Server: Apache-Coyote/1.1

<html><head><title>Apache Tomcat/6.0.24 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.IllegalArgumentException
        java.net.URI.create(URI.java:859)
        javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:95)
        com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:671)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
</pre></p><p><b>root cause</b> <pre>java.net.URISyntaxException: Illegal character in path at index 71: http://localhost:9090/cms/repositories/Asset/branches/main/query/Server[@status=ValidatedWithErrors].asset%7B*%7D.rack[@name=PHX2-F2]
        java.net.URI$Parser.fail(URI.java:2825)
        java.net.URI$Parser.checkChars(URI.java:2998)
        java.net.URI$Parser.parseHierarchical(URI.java:3082)
        java.net.URI$Parser.parse(URI.java:3030)
        java.net.URI.&lt;init&gt;(URI.java:595)
        java.net.URI.create(URI.java:857)
        javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:95)
        com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:671)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/6.0.24 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.24</h3></body></html>

In order for the query I need to force URL encoding for all non-alphanumeric characters in URL (like [, @, ]) and this can't be done with curl or httpie.

I have spent quite some time digging through various options of curl and httpie before I could figure this out, so it would be very helpful to either modify the manuals to describe the issue in detail, or modify the server-side to process queries without ? and with unencoded characters @, [, ].

Thanks for consideration!

billzwu commented 9 years ago

You can use urlencode to encode the query string first. http://meyerweb.com/eric/tools/dencoder/

Server%5B%40status%3D%22Validated%22%5D.asset%7B*%7D.rack%5B%40name%3D%22F2%22%5D

If you would like not to encode the query, just do a post with query string in the request body.

rkiyanchuk commented 9 years ago

@billzwu Yes, I have already figured it out which took plenty of time. The problem is — it's never mentioned in the documentation unless I missed it and you can point where it is.

Therefore documentation should be updated to explicitely address the issue and make it clear that one either needs to put ? before query, or encode all characters, or do a POST with the query in the request body.

Hope this will be an improvement to YiDB's usability as a cool CMS tool.

Thanks!

zhihzhang commented 9 years ago

@zoresvit

Thanks for your suggestion. I have mentioned encode part in our document now. For POST with query, it's for very long query string like "IN [...]". It's not a standard way to do a query, so we are in discussion to deprecate it. So it's not in the document now. Hope you can understand it.

Thanks & Regards Al Zhang

rkiyanchuk commented 9 years ago

@zhihzhang great, thanks for the update!

Interesting, why normal unencoded URL with [ and @ doesn't work even though it should according to the RFC. But at least encoded one works :)

rkiyanchuk commented 9 years ago

@zhihzhang @billzwu Sorry for bugging, but could actually provide a curl example of query request to YiDB that would work? I was able to do it only programmatically from Python. Is there a way to use curl on YiDB API?

billzwu commented 9 years ago

Can you try this?

curl http://localhost:9090/cms/repositories/Asset/branches/main/query/Server%5B%40status%3D%22Validated%22%5D.asset%7B*%7D.rack%5B%40name%3D%22F2%22%5D?allowFullTableScan=True

rkiyanchuk commented 9 years ago

@billzwu Are you suggesting manually perform URL encoding? It's not an acceptable solution.

billzwu commented 9 years ago

@zoresvit We use Chrome PostMan for testing etc instead of CURL. In code, you can call various uuencode methods to encode your query string and compose the URL string.

rkiyanchuk commented 9 years ago

@billzwu that's cool, but if a REST API cannot be queried via curl, there's something wrong with the API implementation, don't you find?

So are there any chances this will get fixed in future versions?

billzwu commented 9 years ago

@zoresvit When not using url encoding for the query, CMS code actually throws an exception.

java.net.URISyntaxException: Illegal character in path at index 79: java.net.URI$Parser.fail(URI.java:2810) java.net.URI$Parser.checkChars(URI.java:2983) java.net.URI$Parser.parseHierarchical(URI.java:3067) java.net.URI$Parser.parse(URI.java:3015) java.net.URI.<init>(URI.java:577) java.net.URI.create(URI.java:839)

It is pretty much a standard for a web server to expect the URI well formed. So instead of go out of the way to fix and allow URI taking non well formed string, I'd suggest you encode the query string before composing the URL.

rkiyanchuk commented 9 years ago

@billzwu As I mentioned in the first post, read carefully through RFC 3986, according to which symbols @, [ and ] do not have to be encoded, so the URI is well-formed. And that's exactly why curl leaves @ and [ unencoded — because it's compliant with the RFC.

So if YiDB is throwing 500 error to a URI compliant to RFC, it has a bug that needs to be fixed. Otherwise it's not compliant to web standards.

I'd be glad to make a pull-request with the fix myself, but unfortunately I'm not Java developer so all hope is for deliberateness of YiDB team.

billzwu commented 9 years ago

@zoresvit Well, I took a read at RFC3986 now. http://www.ietf.org/rfc/rfc3986.txt. At page 12, the symbol '[' and ']' are classified as reserved characters.

reserved = gen-delims / sub-delims

  gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

  sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
              / "*" / "+" / "," / ";" / "="

At the bottom of page 11, it said:

URIs include components and subcomponents that are delimited by characters in the "reserved" set. These characters are called "reserved" because they may (or may not) be defined as delimiters by the generic syntax, by each scheme-specific syntax, or by the implementation-specific syntax of a URI's dereferencing algorithm. If data for a URI component would conflict with a reserved character's purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.

So the RFC actually suggest to encode the URI as well when reserved string is used. Our query syntax is quite complex and there are quite lot of such characters used. So percent-encoded the query string is the way to go.

rkiyanchuk commented 9 years ago

@billzwu The RFC clearly states:

Thus, characters in the reserved set are protected from normalization and are therefore safe to be used by scheme-specific and producer-specific algorithms for delimiting data subcomponents within a URI.

Sorry guys, but if a REST API cannot be used in CLI via curl, there's something obviously wrong with such API.

Please refer discussion in one of popular http CLI clients regarding this issue for details as well: https://github.com/jakubroztocil/httpie/issues/295

billzwu commented 9 years ago

@zoresvit We will discuss internally. We may have another query API with query as a parameter in the QUERY_STRING for the URL for the GET method. Meanwhile, you can use POST call as I suggested before.

rkiyanchuk commented 9 years ago

@billzwu Ok, great, thanks for your time and consideration!

eBay / YiDB

Incorrect API URLs in all the manuals #7