Option to keep order of keys

[adrienaury]

Hello,

It would be nice to have a KeepKeyOrder option such that key order will always be preserved.

row2 := oj.MustParseString(`{"b":0, "a":1, "c":2}`, &ojg.Options{KeepKeyOrder: true})

This could be done by using [][2]any (a list of key/value pairs) instead of map[string]any to store objects.

Is it possible ? if yes, and if someone can point me to the right direction I could try to submit a pull request for that.

Thank you for your answer :)

Adrien

[ohler55]

It is not possible right now. If I understand correctly you would like an option to use something other than map[string]any for JSON objects created during parsing. I suppose the more general case would be to provide types or functions for all types or maybe a callback parser. Is that right?

[adrienaury]

Yes exactly :)

[ohler55]

Using alternatives for each element type is a rather specialized case. Another approach that would give you what you want would be a callback parser like SAX but for JSON. That I would consider adding although I'm not sure how different it would be to the oj.Tokenizer. What do you think?

[adrienaury]

Maybe let me be more specific, I would like to be able to continue using jsonpath expressions on the resulting object, and preserving key order, at all levels.

row := oj.MustParseString(`{"b":0, "a":1, "c":2}`, &ojg.Options{KeepKeyOrder: true})
query := jp.MustParseString("$.b")
fmt.Println(query.Get(row)) // output 0
fmt.Println(oj.JSON(row)) // output : {"b":0, "a":1, "c":2}

With the current version of the library, the last output would give a random order of keys

fmt.Println(oj.JSON(row)) // output : {"a":1, "c":2, "b":0} or any other order of keys

I'm not sure how different it would be to the oj.Tokenizer.

I tried to use a custom oj.Tokenizer, but it didn't work because, JSONPath queries stopped working

[ohler55]

Maps are unordered but JSONPath understand them. JSONPath does not know about new collection types like the one you are proposing. So the second part of going from parsing to access would have to be something that looked for a match on an interface that allowed for indexing the collection by either a string or an integer. That might be the enhancement you are looking for. Is that correct?

[adrienaury]

I don’t know if I understand correctly, but the order should not be part of the data. When you mention a match on an interface, is it about a way for the jsonpath engine to understand that the key order is important so it must adapt its algorithm to know how to read an objects with its keys? In this case, I think yes it respond to what I am looking for, because it would be an internal detail about how to « scan » for objects keys.

Let me know if I didn’t understand

[adrienaury]

To complete my previous response:

I think the parser could use this struct when the KeyOrder option is active, in place of a map[string]any

type object struct {
  m map[string]any // object keys / values map
  keys []string // order in witch keys should be accessed
}

The the JSONPath engine could access object keys like this when it detects an object struct

obj.m[key] // instead of obj[key]

Finally, the conversion to JSON could use a range over the keys instead of over the map, when it detects an object struct

for _, key := range obj.keys: // instead of : for key, value := range obj:
   value := obj.m[key] // process to output key/value pair as JSON

[ohler55]

Almost but not quite. Let me try providing an example.

First the collection or map alternative:

type OrderedMap struct {
    m    map[string]any
    keys []string
}

func (om *OrderedMap) ValueForKey(key string) any {
    return om.m[key]
}

func (om *OrderedMap) Keys() []string {
    return om.keys
}

The interface that the jp package would look for would be:

type Keyed interface {
    ValueForKey(key string) any
    Keys() []string
}

How that would work internally is that when evaluating the JSONPath (jp.Expr) the calls to walk over the OrderMap would use the interface calls to get keys and values. This was just my first thoughts so the actual interface might be different.

[adrienaury]

This is even better I agree.

Thank you for taking time with me :)

Do you think that solution would be easy to implement without breaking anything on the current version ?

[ohler55]

It will be an enhancement so nothing exiting will change from the user perspective. I've get it in this weekend but might take a few extra days for tests.

[ohler55]

If you would like to see how this would work take a look at the "keyed-indexed" branch and the end of the get_test.go file. Just a few expressions have been implemented so far and the test demonstrates them. Not that I combined the map and slice alternatives into one type. That is not required but only to simplify testing.

[adrienaury]

I see that if I parse the JSON into objects that implements the Keyed interface, then jp expressions will work fine. This is already a huge step toward my goal! Thank you @ohler55 :)

The parser should understand the Keyed (and Indexed) interface to output keys in the correct order, but for this part I know I can do it with a custom code.

I will be able do test it extensively in the next few weeks because it will be use on a new tool we are working on in my organisation

[ohler55]

Good to hear. I still haven't finished the set and modify calls yet and filters have not been done either but that should happen over the next few days.

[ohler55]

Tests completed. If you haven't run into any issues I'll make a release.

[adrienaury]

Thank you, I did some tests yesterday, everything went fine with Get and Set. So far so good :) you can launch the release 🚀

[ohler55]

Released v1.18.2