search

cube2222 / octosql

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
Mozilla Public License 2.0
4.75k stars 201 forks source link

JSON data type #268

Open Allam76 opened 2 years ago

Allam76 commented 2 years ago

First or all, fantastic work! I really love this tool. It really fills a void in the ecosystem. Also kudos for the rewrite!

I would suggest to add a json data type. Then all json processing can be in a dedicated type and not assumed to be an object that can be converted to map[string]interface{}

For me it works great with the gjson/sjson libraries that work directly on strings and then store the data simply as Str in the octosql value type.

I can send a PR when you feel that the rewrite is sufficiently stable.

go has fantastic json support and would be great to leverage that rather than transforming the json to a different octosql type.

Cheers Martin

cube2222 commented 2 years ago

Hey!

Thanks for raising this issue!

I'll be working on JSON and object support soon, so will keep this use case in mind - it totally makes sense.

Do you have a specific reason for not wanting to decode the JSON types into map[string]interface{} maps - that is, if proper operators for working with them would be supplied? Or is it just that you like the interface/ergonomics of the packages you mentioned?

Allam76 commented 2 years ago

Well map[string]interface{} implies json objects and call me old-fashioned but I often work with top level json arrays :-)

This could of course be addressed with standard go json marshalling but then the top level json would need to be known (array or object).

These libraries are dirt simple and come with some serious batteries includes, such as json path. The actual data is just a string. So it took me 30 minutes to smack together with the existing type system.

So why handle json processing in the octosql core when not needed. Building a federated sql engine is hard enough. Outsource what can be outsourced.

When you feel octosql is stable enough, give me a shout and I'll smack together the json functions we are used to and love from postgres.