ChuckConnell
commented
11 years ago Jrask -- I am also working with this SerDe, and would very much like to get it running. There is a typo in your post though. You said
But when I write hive> select payload.params from fresspayload where payload.params.session.id.value == "xxxx";
but you did not say what happens in this case...??
One issue to keep in mind is that there is not (I am fairly sure) a one-to-one correpondence between JSON datatypes and Hive datatypes. So it is sometimes tricky to match types.
jrask
rcongiu
Hi, I am using this project to see if I can use it to query many large nested documents. So far I have only tried with small documents and I have a question.
Schema
create table fresspayload ( payload struct<id:string,jsonrpc:string, params:struct< device_info:struct,
session:struct<
id:struct<type:string,value:string
Json document:
{ "payload": { "jsonrpc": "2.0", "id": "0cd0f2f8-31b6-4038-9864-1bcc01dee188", "params": { "device_info": { "model": "HTC Desire", "device_id": "xxxx" }, "session": { "id": { "type": "msisdn", "value": "xxxxx" }, "type": "app" }, "received_at": "2012-08-28T00:07:21.858Z" }, "method": "connect" }, "protocol_version": "1.1", "route": { "source": { "id": { "type": "msisdn", "value": "xxxx" }, "type": "app" } } }
Once imported I run:
hive> select payload.params from fresspayload; Total MapReduce CPU Time Spent: 2 seconds 70 msec OK {"device_info":{"model":"HTC Desire"},"session":null} Time taken: 30.232 seconds
It seems like "session" is null
But when I write a Where clause with the session id it returns a hit, see below. hive> select payload.params from fresspayload where payload.params.session.id.value == "xxxx"; Total MapReduce CPU Time Spent: 2 seconds 230 msec OK {"device_info":{"model":"HTC Desire"},"session":null} Time taken: 30.117 seconds
But when I run with nonexisting id I do not get any matches. hive> select payload.params from fresspayload where payload.params.session.id.value = "non_existing"; MapReduce Total cumulative CPU time: 1 seconds 910 msec Ended Job = job_201209041503_0096 MapReduce Jobs Launched: Job 0: Map: 1 Accumulative CPU: 1.91 sec HDFS Read: 0 HDFS Write: 0 SUCESS Total MapReduce CPU Time Spent: 1 seconds 910 msec OK
Since the WHERE clause matches there must be something right with this... do you have any input concerning this?
Kind regards /Johan Rask