search

IBMStreams / streamsx.json

Toolkit for working with JSON in SPL applications.
http://ibmstreams.github.io/streamsx.json/
Other
3 stars 19 forks source link

Convert to map in JSONToTuple operator #121

Closed rrea closed 3 years ago

rrea commented 4 years ago

The JSONToTuple operator in the streamsx.json toolkit currently only supports converting nested objects to nested tuples. This requires the operator to always know the structure of the JSON and does not allow for flexibility within the schema.

Supporting conversion to a map will allow content within the JSON structure to differ between submissions and support more generic operations.

From Aha! Idea: https://bigblue.aha.io/ideas/STREAMSOP-I-3

markheger commented 3 years ago

Streams schema even with map of type is not flexible enough, for nested types also. Structured streams schema with map has a fixed layout (key, values and nested structure, for example map of map).

The only flexible schema in Streams in CommonSchema.Python when using Python operators.

rrea commented 3 years ago

looks like a good issue to resolve ...

rnostream commented 3 years ago

If one needs more generic flexibility regarding Streams schemas one has to use Python operators. Between Python operators any schema (being serializable by Python) can be send and transparently handled by receiving operator while using only those element which are known. I think being more generic in Streams schemas between legacy operators would need some effort in platform too, this is not only question of a single toolkit. How should legacy operators/functions make use of this generic?

JSON is integral part of Python. We don't have the possibility to extend the streamsx.JSON toolkit (which is C++) to any possible future scenarios relating from different JSON usages. As we actually focus on going Python, why not using Python operators for handling those schema generics.