search

IBMStreams / streamsx.json

Toolkit for working with JSON in SPL applications.
http://ibmstreams.github.io/streamsx.json/
Other
3 stars 19 forks source link

JSONToTuple incorrectly parsing event payloads from streamsx.iotf.DeviceEvents #52

Closed millerbryan closed 8 years ago

millerbryan commented 8 years ago

I am ingesting device events using streamsx.iotf.DeviceEvents which returns

<rstring deviceType, rstring deviceId, rstring eventId, rstring jsonString>

However, when I stream to JSONToTuples (1.1.1_dev) it parses most of the JSON correctly but fails to parse a few items. I used createTypes.sh to generate the list of types.

I have attached my code and a sample JSON event payload. zoneID, storeID, and the customer ID list are the items failing to parse correctly.

Thanks, Bryan

MQTTtoAnayticsTest.txt MQTTtoAnayticsTestJSON.txt

ddebrunner commented 8 years ago

Did you try the operator in the streamsx.iotf toolkit that was intended for this purpose?

DeviceEventExtractData

http://ibmstreams.github.io/streamsx.iotf/doc/spldoc/html/tk$com.ibm.streamsx.iotf/spl$com.ibm.streamsx.iotf$extract.html#spldoc_compilationunit__composite_operator__DeviceEventExtractData

ddebrunner commented 8 years ago

I don't think the JSONToTuple operator is doing anything incorrectly, with your SPL application it is looking for JSON keys at the top level of storeId, customer, and zoneID, but they are not present in the jsonString, because IoTF puts them in a sub-object with key "d".

DeviceEventExtractData from streamsx.iotf was designed to make this simple for you and not to have to worry about how JSONToTuple needs to be setup to cope with the IoTF device event format.

millerbryan commented 8 years ago

I concur. I took the output from createTypes.sh and created a new type in my application and it is parsing correctly now.

millerbryan commented 8 years ago

DeviceEventExtractData is very useful! Thank you for the pointer.