ygot.Diff don't respect presence containers

[JonasKs]

Hi,

Given two structs (generated with -yangpresence=true flag), the ygot.Diff() do not diff based on presence or not.

Given the struct below, where AllowImportedVpn is a presence container, and sending it should set the presence:

*Vrf {
    AddressFamily: *Vrf_AddressFamily {
        Ipv4: *Vrf_AddressFamily_Ipv4 {
            Multicast: *Vrf_AddressFamily_Ipv4_Multicast nil, 
            Unicast: *Vrf_AddressFamily_Ipv4_Unicast {
                Export: *Vrf_AddressFamily_Ipv4_Unicast_Export {
                    To: *Vrf_AddressFamily_Ipv4_Unicast_Export_To {
                        DefaultVrf: *Vrf_AddressFamily_Ipv4_Unicast_Export_To_DefaultVrf nil, 
                        Vrf: *Vrf_AddressFamily_Ipv4_Unicast_Export_To_Vrf {
                            AllowImportedVpn: *Vrf_AddressFamily_Ipv4_Unicast_Export_To_Vrf_AllowImportedVpn {} // <-- This is a presence container
                        }
                    }
                },
            }
        }, 
        Description: *"Default", 
        VrfName: *"TestVRF"
    }
}

with this object:

*Vrf {
    AddressFamily: *Vrf_AddressFamily {
        Ipv4: *Vrf_AddressFamily_Ipv4 {
            Multicast: *Vrf_AddressFamily_Ipv4_Multicast nil, 
            Unicast: *Vrf_AddressFamily_Ipv4_Unicast {
                Export: *Vrf_AddressFamily_Ipv4_Unicast_Export {
                    To: nil. // <-- Presence container is nil
                },
            }
        }, 
        Description: *"Default", 
        VrfName: *"TestVRF"
    }
}

where the entire To key is set to nil, creates no diff. As far as I understand, this should remove the presence, and therefor create an Update-entry?

Here's the YANG model: https://github.com/YangModels/yang/blob/main/vendor/cisco/xr/772/Cisco-IOS-XR-um-router-bgp-cfg.yang#L565C1-L571

[robshakir]

This looks like something that needs to be added to ygot.Diff -- since OpenConfig does not use presence containers then not all functionality that pertains to them may be supported today.

I'm happy to help with a contribution to add this to the library -- it needs some consideration as to where Diff finds out that a particular container is a presence container.

edit: It looks like this should be relatively simple, as the Diff code traverses the struct then we need to do a IsYANGPresence check and then if it returns true, then produce output for the container.

[JonasKs]

I looked into this briefly before I created the issue, but it is a pretty big code base to grasp.

Would you just add an if-else here? https://github.com/openconfig/ygot/blob/master/ygot/diff.go#L310

I'm out for the holidays, but I'd love to tive this a shot next weekend if no one beats me to it. We definitely need this for our Crossplane provider.

Thanks for the great library and the quick response 😊

[JonasKs]

I had another quick look at this today. Would love some pointers/tips, as I were not successful.

Adding this as a non-optional, paired with ygot.BuildEmptyTree would also probably be breaking, as it would set all these presence containers?

[wenovus]

The logic may be added here: https://github.com/openconfig/ygot/blob/4dcc65ef1230778675d989ce9a4d194512974f17/ygot/diff.go#L316C41-L316C41

I think you want to do a if on util.IsValueStructPtr(ni.FieldValue), deal with the orderedMap case (it's not relevant), and in a separate case check for IsPresence.

I think the tricky part comes in how you would indicate presence containers in the return value of ygot.Diff. In the out variable you might want to store some sentinel value to indicate to the comparison code that a particular path is a presence container. In the gpb.Notification output, one option is to use a nil update value to indicate a presence container.

[JonasKs]

Thanks @wenovus, I'll give it a shot this or next weekend.

I would of course appreciate if anyone else got time to look at an implementation. 😊

[robshakir]

w.r.t backwards compatibility -- this should be done by providing some DiffOpt to ygot.Diff to say that you care about presence containers IMHO.

[JonasKs]

Any chance OpenConfig will add support for this, or is this relying on outside contributors (me?)?

[robshakir]

As with most open source projects, the feature set of ygot tends to be driven by contributions where there is interest in a feature -- and by having those with interest contribute code.

The core maintainers (@wenovus these days) try to ensure that the library is complete for users, but especially with a language as complex as YANG where there are many ways to express things, it's not really possible for us to commit to cover every feature. Equally, we have found that the way that developers expect particular YANG features to be represented varies, and having a real user helps with that.

So, the answer is "it might happen, but it will certainly happen if you contribute code". :-)

[JonasKs]

I gave GitHub Copilot Workspace a go at this issue, and safe to say it didn't really work. It focused more on removing licensing headers, hehe. (Commit)

I still need this, and will revisit. Do you guys have a slack/IRC/Discord for potential questions? Last time I have this a shot I did not succeed.