Restful API return string in raw adm format instead of parsable json

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Start AsterixDB use the automatically generated configuration for single 
machine.

2. Created a datatype (UserDataType.aql) and inserted a user 
(insertSampleUserOfOurSys.aql).

3. Send query: 
"http://127.0.0.1:19002/query?mode=synchronous&query=use+dataverse+feeds%3B%0Afo
r+$u+in+dataset+usersDataSet%0Areturn+$u" with HTTP Accept header set to 
"application/json; charset=utf-8"

4. Get output in the raw adm format that is not parsable by JSON.parse():

What is the expected output? What do you see instead?

Expect to get the returned information in type "application/json" but instead 
get "text/html". For example, I got this, which is not parsable by JSON.parse() 
because of derived Asterix datatype like UnorderedList:

{"results":["{ \"userId\": \"17\", \"screenName\": \"birdy\", \"termsTracked\": 
{{ { \"term\": \"GoogleGlass\", \"dateAdded\": 
datetime(\"2014-04-01T10:10:35.000Z\"), \"belongsToWhichProductCategories\": {{ 
\"Electronics\", \"Clothing_Accessories\" }} }, { \"term\": \"APK Mini\", 
\"dateAdded\": datetime(\"2014-04-06T10:10:35.000Z\"), 
\"belongsToWhichProductCategories\": {{ \"Musical_Instruments\" }} } }} }\n"]}

What version of the product are you using? On what operating system?
--------------------------------------------------------------------
Asterix 0.8.5 on Ubuntu 12.04

Please provide any additional information below.

The reason I suspect this might be a bug is because in the Tweetbook demo 
provided on the AsterixDB's official website, there's a file called 
tweetbook.js. It uses the asterix javascript sdk but there's a line of code: 
"$.parseJSON(data.replace('rectangle(', '')
                            .replace(')', '')
                            .replace('i64',''));"

This means that the response data received can't be directly parsed as JSON 
(for example, the author replaced 'rectangle(' with '', but 'rectangle(' 
shouldn't appear if you ask the Asterix's restful API to return json.

Original issue reported on code.google.com by hencr...@gmail.com on 25 Apr 2014 at 4:11

Attachments:

• UserDataType.aql
• insertSampleUserOfOurSys.aql

[GoogleCodeExporter]

What if you only ask for stuff that's in the JSON subset of ADM?  It still 
seems to me like you are "causing" this problem.  Don't ask for things that 
JSON doesn't have and let's see if that helps?  (Or am I misunderstanding what 
you are doing?)  

Original comment by dtab...@gmail.com on 25 Apr 2014 at 5:33

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I need to use the unordered list because I need to access the list of terms 
this user account is currently tracking.

Also, I explicitly set the HTTP Accept header to "application/json; 
charset=utf-8" to notify the AsterixDB restful API that I want a response in 
the ADM format parsable by JSON, not raw ADM string. The reason for setting 
such header is because in the AsterixDB javascript SDK (more specifically 
AsterixDBConnection.prototype._api function), it sets the dataType field of 
JQuery's ajax call to 'json' (which also sets the underlying HTTP Accept 
header) to notify the server that the client side wants JSON response.

Based on: http://asterixdb.ics.uci.edu/documentation/aql/js-sdk.html, I thought 
I should receive unorderedList in the format "unorderedlist: [......]" instead 
of "{{......}}".

Finally, the reason why I thought this is a bug because in the Tweetbook demo 
provided, it uses the official Asterix javascript sdk but still need to replace 
some characters in the response like 'rectangle(' with '' before parsing it 
with JSON parser. In other words, even when using the official javascript sdk, 
the result received is not like the parsable ones described on 
http://asterixdb.ics.uci.edu/documentation/aql/js-sdk.html, but raw adm string 
that is not parsable.

Perhaps this should be described as a mismatch between documentation and the 
actual implementation instead of a bug. Or a even better scenario is that I 
misunderstood something and could you tell me how do I request for the 
"transformed" JSON-parsable response?

Thank you!

Original comment by hencr...@gmail.com on 26 Apr 2014 at 12:13

[GoogleCodeExporter]

Aha!  Thanks for the clarification.  I'm reassigning this to our API expert - 
the API is new-ish and was developed by a student as a CS299 project - our API 
expert is the owner of this because it was sort of orphaned and he had the 
knowledge to become its new "volunteer" parent among the AsterixDB student 
team.  :-)

The reason I was saying "you" were "causing the problem" was that it's 
definitely the case that ADM data types that are over/above what JSON supports 
cannot be expected to be directly parsable as JSON - and I was thinking you 
were thinking otherwise.

Let's see what Zach has to say, and/or one of the other folks on the team who 
has relevant expertise (and hopefully advice).

Original comment by dtab...@gmail.com on 26 Apr 2014 at 9:53

[GoogleCodeExporter]

Glad that clear things out. Thank you and the Asterix team for looking into 
this.

Original comment by hencr...@gmail.com on 27 Apr 2014 at 3:13

[GoogleCodeExporter]

What @hencrice is saying is correct. When our webserver sets the content-type 
to application/json, but returns ADM, this is breaking the HTTP protocol. To 
conform to the HTTP protocol, I see two options:
(1) Add support for converting ADM into JSON, though we will have to define 
what this means for complex types that are not supported in JSON (e.g. points, 
rects, datetimes, etc.)
(2) Not advertise that we are returning application/json. From RFC2616 S10.4.7 
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7), we could 
either return a 406 response when the client asks for application/json OR just 
return ADM without without setting our content-type to be application/json.

      Note: HTTP/1.1 servers are allowed to return responses which are
      not acceptable according to the accept headers sent in the
      request. In some cases, this may even be preferable to sending a
      406 response. User agents are encouraged to inspect the headers of
      an incoming response to determine if it is acceptable.

Doing (1) will take time, so to immediately unblock you, I'd suggest ignoring 
the fact that the head says the content-type is application/json and sure just 
assume that you'll be receiving ADM.

Original comment by zheilb...@gmail.com on 27 Apr 2014 at 8:56

[GoogleCodeExporter]

So basically, you mean what's described on 
http://asterixdb.ics.uci.edu/documentation/aql/js-sdk.html is not yet 
implemented?

Also, the content-type header I received is actually text/HTML. In other words, 
the server doesn't "pretend" it's sending json, it just ignores my Accept 
header that my client wants json.

Original comment by hencr...@gmail.com on 27 Apr 2014 at 9:10

[GoogleCodeExporter]

I'm confused. How are you getting text/html? In the REST API server, we always 
set the content-type to application/json as far as I can tell:

...
public void handleRequest(HttpServletRequest request, HttpServletResponse 
response, String query)
            throws IOException {
        response.setContentType("application/json");
        response.setCharacterEncoding("utf-8");
...

To be sure, I just tried your query on the latest release of asterix and I 
receive application/json as the content-type of the response. Can you please 
tell me what version of asterix you are using?

Also, as I pointed out in the RFC, it is acceptable for the server to ignore 
your "Accept" header.

With resepct to (2) in my previous comment, I just looked through our codebase 
and found that we actually do convert complex ADM types into JSON primitives. 
So as far as I can tell, what's described on 
http://asterixdb.ics.uci.edu/documentation/aql/js-sdk.html is implemented 
(after all, those are results from actual executions of the demo). The reason 
you are not seeing it is due to a subtle bug. 

The subtle bug is as follows:

        String contentType = request.getContentType();
        if ((contentType == null) || (contentType.equals("text/plain"))) {
            format = DisplayFormat.TEXT;
        } else if (contentType.equals("application/json")) {
            format = DisplayFormat.JSON;
        }

When displayformat JSON is selected, complex types will be serialized as JSON 
primitives. When TEXT format is selected, you will see ADM as it is. This much 
is correct! However, the issue here is that the "content-type" of a request is 
being confused with the "Accept" header of a request. This code needs to be 
changed to look at the Accept header rather than content-type. You can verify 
this by manually setting your content-type to application/json.

Original comment by zheilb...@gmail.com on 27 Apr 2014 at 10:12

[GoogleCodeExporter]

Original comment by zheilb...@gmail.com on 27 Apr 2014 at 10:13

• Changed state: Accepted

[GoogleCodeExporter]

@Till also pointed out that the ADM -> JSON conversion is not entirely correct 
since field names are not quoted, so you may run into issues there if you run 
it through a JSON parser.

Original comment by zheilb...@gmail.com on 27 Apr 2014 at 10:55

[GoogleCodeExporter]

Thanks Zach.

About receiving text/HTML, I was wrong. The result I received is indeed json 
(i.e an ADM string embedded in the json returned).

So tentatively for now you will suggest I set the content-type to 
application/json to retrieve something that is almost json parsable, then use 
something like regular expression to fill in the missing quotes. Is that right?

Original comment by hencr...@gmail.com on 27 Apr 2014 at 11:41

[GoogleCodeExporter]

Unfortunately, yes... At least until we can add a fix for you. What version of 
Asterix are you using: the standard release on the website or one of Raman's 
branches?

Original comment by zheilb...@gmail.com on 28 Apr 2014 at 4:25

[GoogleCodeExporter]

I'm currently using the standard version 0.8.5 provided on the Asterix website.

Original comment by hencr...@gmail.com on 28 Apr 2014 at 4:42

[GoogleCodeExporter]

Original comment by zheilb...@gmail.com on 15 May 2014 at 12:59

[GoogleCodeExporter]

FYI, I have committed changes some time ago that fix the missing-quotes part of 
this problem. Also, I switched it to check the "Accept" header rather than 
"Content-type". So, now if you send your request with "Accept: 
application/json", the contents of the "results" array will be strings that 
are, in turn, parseable JSON. This should be in 0.8.7 (it might have been in 
0.8.6 even).

I am working on more sweeping changes at the moment. Most notably, when you 
request JSON from the server, in a non-error case the output should be the 
actual result - ie, the wrapper JSON Object with its "results" field will be 
gone. I'm also continuing to tweak how query results get translated into JSON.

Original comment by c...@lambda.nu on 8 Aug 2014 at 7:57

[GoogleCodeExporter]

I have completed the changes mentioned above: 
http://fulliautomatix.ics.uci.edu:8443/#/c/123/

When you request JSON, what you will get will be the JSON form of the actual 
results - no more "results" wrapper. It should always be fully-correct 
parseable JSON. In order to ensure that it is always a single JSON text, the 
output will in fact always be a JSON array with zero or more values (even if 
the query returns a single object).

There are more changes coming - right now, for instance, I believe it's 
impossible to get JSON from a deferred-results API call (ie. ASYNC). I'm 
pondering introducing an "application/adm" output format for ADM results and 
eliminating the "results" wrapper JSON object there as well. However, that's 
going outside the scope of this issue, so I am resolving this one and will open 
others according to plan.

Original comment by c...@lambda.nu on 21 Aug 2014 at 6:57

[GoogleCodeExporter]

Original comment by c...@lambda.nu on 21 Aug 2014 at 6:57

• Changed state: Fixed