How to replace data in the operational datastore?

[samuel-gauthier]

Hi Michal,

With sysrepo 2, the possibility to "replace" the content of the operational datastore with sr_edit_batch was removed, leaving us only with "merge". We are in the process of migrating our application to sysrepo2, and we were using this feature in the previous sysrepo version. Would you know of a way to do something similar with the new version?

Thanks, Samuel

[michalvasko]

You mean that, for example, you have some subscribed running data that are copied into operational and you want to replace these with some other data? You cannot because these data would then depend on the base data they are replacing, which could 1) lead to errors or 2) be adjusted on the base data changes, which is too costly to check.

[samuel-gauthier]

Thanks for your answer, but I am not sure to understand your comment... Let me try to clarify what we do.

Our application registers a callback for oper data to return the system state. In order to get yang push on-change notifications, our application also writes data in the operational datastore, for the same modules. On some triggers (netlink, or dbus) , a local cache of the system state in the application is updated. The cache is then written to the operational datastore with edit_batch. Then we can be notified if we subscribe to yang push notifications of changes on the operational datastore. When a netconf request comes to get the state of the operational datastore, the data written in the datastore is merged with the data retrieved by callback, thanks to the oper_merge flag in the oper data subscription.

At least it was my understanding, and it seemed to work well. With the previous version, I was able to be notified of the changes in the operational datastore when subscribing to the changes of this operational datastore with sysrepo, and the state retrieved was correct.

Replacing the data is really what we want to do here, as we are writing our local cache to the datastore. Sysrepo sees the differences, and triggers the right notifications.

Am I misunderstanding something?

Regards, Samuel

[michalvasko]

In short, the way it worked before was causing the internals to be too complex and inefficient, maintaining these operational data, so I have decided to simplify them at the cost of removing some functionality. So they were changed to not depend on the underlying data, which leaves you with the remove and merge operations that can never fail and their expected changes (delete and create) can safely be reported to the subscribers. Despite them not having to actually result in these operations (if the node to be merged exists, it will not be created).

Anyway, have you tried combining remove and merge to get the functionality of replace?

[samuel-gauthier]

Thanks for your explanation. Do you mean using sr_delete_item to remove, then sr_edit_batch, and then apply the changes?

[michalvasko]

You will have to use sr_oper_delete_item_str(), then sr_edit_batch() and then sr_apply_changes().

[samuel-gauthier]

It does not work, I get this error "There are already some session changes." when calling sr_edit_batch. My understanding is that sr_oper_delete_item_str takes the edit context lock, and sr_edit_batch won't work in this case.

[michalvasko]

Right, there is no lock but it will not work this way, I have only partly corrected what you wrote. I meant sr_oper_delete_item_str() with sr_apply_changes() and then sr_edit_batch() with sr_apply_changes(). You just need to be careful to actually delete something so I am not sure how usable this would be.

For example, if you have

list l1 {
  key "k1";
  leaf k1 {
    type string;
  }
  list l2 {
    key "k2";
    leaf k2 {
      type string;
    }
  }
}

and want to replace an instance of l2 then you must not delete /l1[k1='a'] and merge /l1[k1='a']/l2[k2='b'], which would overwrite the previous edit. Instead, delete /l1[k1='a']/l2[k2='a'] and merge /l1[k1='a']/l2[k2='b'].

[samuel-gauthier]

Basically, our application calls a callback when something changes in the system (the trigger could be dbus, netlink, or inotify for instance). The callback queries the system to deduce its current state and sends it to another part that writes it to the sysrepo datastore. So we don't know what changed, only that something changed. We were relying on sysrepo for the diff. Having a hierarchical yang model, we have lists at several levels, in different modules, which does not help. Also, doing things in two apply changes is not too nice, as a query coming between the two changes could return partially wrong data.

I'll try to see how to accommodate for this new sysrepo behavior, although I must say I'm quite pessimistic...

Anyway, thanks for your time and your help with this problem. Samuel

[michalvasko]

Just note that the operational datastore is a tricky one and the stored oper data are among the most complex features of sysrepo. Even when implementing it I was hoping it will not be used for anything too wild because I knew there would be problems. I was mistaken and it was used a lot which expectedly led to serious problems and the whole feature had to be reduced to the current functionality.

[samuel-gauthier]

I could find a workaround. Thanks!