pybindJSONDecoder.load_json with overwrite

[tonic12]

I'm try overwrite data in object:

data = json.loads('''{
    "interface": {
        "LAN9": { "ip-address": "9.9.9.9" },
        "LAN8": { "ip-address": "8.8.8.8" }
    }
}''')
pybindJSONDecoder.load_json(data, None, None, obj=model_obj, overwrite=True)

and recived error:

Traceback (most recent call last):
  File "src/lib/ymodels.py", line 65, in set_data_to_model_object
    pybindJSONDecoder.load_json(data, None, None, obj=model_obj, overwrite=True)
  File "/home/arch/.local/lib/python3.7/site-packages/pyangbind/lib/serialise.py", line 666, in load_json
    for child_key in chobj:
RuntimeError: OrderedDict mutated during iteration
{
    "test-router:interface": [
        {
            "id": "LAN2",
            "ip-address": "2.2.2.2"
        },
        {
            "id": "LAN3",
            "ip-address": "3.3.3.3"
        },
        ...
    ]
}

When i change for child_key in chobj: to for child_key in [*chobj]: in file lib/serialize.py it works fine. Data overwritten is correct. This is bug or i doing some thing wrong?

Then i check load_ietf_json,

data = json.loads('''{
"test-router:interface": [
    { "id": "LAN9", "ip-address": "9.9.9.9" },
    { "id": "LAN8", "ip-address": "8.8.8.8" }
]
}''')
pybindJSONDecoder.load_ietf_json(data, None, None, obj=model_obj, overwrite=True)

and recived error:

Traceback (most recent call last):
  File "src/lib/ymodels.py", line 63, in set_data_to_model_object
    pybindJSONDecoder.load_ietf_json(data, None, None, obj=model_obj, overwrite=True)
  File "/home/arch/.local/lib/python3.7/site-packages/pyangbind/lib/serialise.py", line 791, in load_ietf_json
    for i in existing_keys:
RuntimeError: OrderedDict mutated during iteration
{
    "test-router:interface": [
        {
            "id": "LAN2",
            "ip-address": "2.2.2.2"
        },
        {
            "id": "LAN3",
            "ip-address": "3.3.3.3"
        },
        ...
    ]
}

I go to lib\serialize.py to line 790 and change existing_keys = this_attr.keys() to existing_keys = [*this_attr.keys()]. I think now all be ok, but it's not! Print the object pybindJSON.dumps(model_obj, filter=False, mode="ietf"):

{
    "test-router:interface": [
        {
            "id": "LAN8",
            "ip-address": "8.8.8.8"
        }
    ]
}

Where part { "id": "LAN9", "ip-address": "9.9.9.9" }?

Yang model:

module test-router {
    yang-version 1;
    namespace "urn:test:router";
    prefix test-router;

    description "Test krutilok";

    revision "2019-07-01" {
        description "Initial version.";
    }

    list interface {
        key id;

        leaf id {
            type string;
        }

        leaf ip-address {
            type string;
        }
    }

    container device {
        config false;
        description "Device identification information.";

        leaf vendor {
            type string;
            description "Device vendor identificator.";
        }

        leaf model {
            type string;
            description "Device model.";
        }

        leaf serial {
            type string;
            description "Device serial number.";
        }

        leaf hwrev {
            type string;
            description "Device hardware revision.";
        }

    }
}

[tonic12]

I found where lost { "id": "LAN9", "ip-address": "9.9.9.9" }. It because we delete items every iteration, even when we add it. That's list and we have no key for check what need to delete. That's mean we need save start state and check current state with starting state.

My fix lib\serialize.py line 781:

            elif isinstance(d[key], list):
                start_state = None # fix overwrite
                for elem in d[key]:
                    # if this is a list, then this is a YANG list
                    this_attr = getattr(obj, "_get_%s" % safe_name(ykey), None)
                    if this_attr is None:
                        raise AttributeError("List specified that did not exist")
                    this_attr = this_attr()
                    if hasattr(this_attr, "_keyval"):
                        if overwrite:
                            existing_keys = [ *this_attr.keys() ]
                            if not start_state: # fix overwrite
                                start_state = existing_keys # fix overwrite
                            for i in existing_keys:
                                if i in start_state: # fix overwrite
                                    this_attr.delete(i)

Now i have result:

{
    "test-router:interface": [
        {
            "id": "LAN9",
            "ip-address": "9.9.9.9"
        },
        {
            "id": "LAN8",
            "ip-address": "8.8.8.8"
        }
    ]
}

[tonic12]

Init state:

{
    "test-router:interface": [
        { "id": "LAN4", "ip-address": "1.1.1.1" },
        { "id": "LAN2", "ip-address": "2.2.2.2" },
        { "id": "LAN3", "ip-address": "3.3.3.3" },
        { "id": "WAN", "ip-address": "4.4.4.4" },
        { "id": "LAN1", "ip-address": "5.5.5.5" }
    ],
    "test-router:device": {
        "vendor": "apple"
    }
}

I'm try overwrite data in object:

data = json.loads('''{
            "test-router:interface": [
                { "id": "LAN9", "ip-address": "9.9.9.9" },
                { "id": "LAN1", "ip-address": "8.8.8.8" }
            ],
            "test-router:device": {
                "model": 1432.1
            }
        }''')
pybindJSONDecoder.load_ietf_json(data, None, None, obj=model_obj, overwrite=True)

and result:

{
    "test-router:interface": [
        {
            "id": "LAN9",
            "ip-address": "9.9.9.9"
        },
        {
            "id": "LAN1",
            "ip-address": "8.8.8.8"
        }
    ],
    "test-router:device": {
        "vendor": "apple",
        "model": "1432.1"
    }
}

1) container 'device' not overwritten; (only for load_ietf_json method, load_json works correct)

Go to lib/serialise.py line 765 and replace
                chobj = attr_get()
                if hasattr(chobj, "_presence"):
                    if chobj._presence:
                        chobj._set_present()

                # fix overwrite 
                if overwrite:
                    for elem in chobj._pyangbind_elements:
                        unsetchildelem = getattr(chobj, "_unset_%s" % elem)
                        unsetchildelem()
                # fix overwrite end

                pybindJSONDecoder.check_metadata_add(key, d, chobj)
                pybindJSONDecoder.load_ietf_json(
                    d[key],
                    None,
                    None,
                    obj=chobj,
                    path_helper=path_helper,
                    extmethods=extmethods,
                    overwrite=overwrite,
                    skip_unknown=skip_unknown,
                )

2) container 'device' - has property config false; and it's can't be written.

It's mean when JSON apply to object, yang model restrictions not check.

[JoseIgnacioTamayo]

Hi,

Could you please try again with recent versions of pyangbind?

Thanks.

[JoseIgnacioTamayo]

Closing issue without recent updates.