lyd_validate_all takes ~6 seconds to run

[donaldsharp]

Running libyang 2.0.7, in the FRR project. When lyd_validate_all is called with aproximately ~3k lines of config it is taking ~6 seconds to validate the configuration.

2023/05/03 18:05:08.699713 BGP: [M7Q4P-46WDR] vty[32]@(config)# route-map FOOZIP permit 10
2023/05/03 18:05:08.700330 BGP: [VFJ68-Y6TG7] About to validate all
2023/05/03 18:05:13.911460 BGP: [JK895-Y959X] Finished validating all

About to and Finished lines are above and below the lyd_validate_all function, so timings are accurate from my perspective. I do not even know where to begin to figure out what I need to be doing to speed this validation up to a reasonable value. What data can I gather for you to help me figure out where the problem is?

[michalvasko]

I am surprised that this is the first problem you encountered with the libyang version you are using, it is really old and v2 was still immature then. So the first thing you should do is update to the latest release v2.1.55.

[donaldsharp]

I have upgraded to 2.1.55 and am now seeing ~.8 seconds to run validation. Definitive improvement but more would be a huge help

2023/05/04 07:27:13.162532 BGP: [M7Q4P-46WDR] vty[31]@(config)# route-map foozagi permit 10
2023/05/04 07:27:13.163154 BGP: [VFJ68-Y6TG7] About to validate all
2023/05/04 07:27:13.910913 BGP: [JK895-Y959X] Finished validating all

What data could I gather to help here?

[michalvasko]

Well, 3k lines of config is not exactly few so this may be it. Nevertheless, I can take a look if you can provide the data and the YANG module(s) needed to load them. You can even analyze the performance yourself if you want to, either using perf or what I personally use, valgrind tool callgrind.

[donaldsharp]

frr-yang-models.tar.gz

Here is the frr yang models as well as a flamegraph of ~1 minute of run time of the 3k file read in. ( this takes ~2 1/2 minutes to read in with the libyang 2.1.55 )

As a side note 3k lines is not a large FRR config at all. We have operators with route-maps and prefix-lists that total well over 100k lines. We need to be able to work with those as well.

[michalvasko]

Thanks for the modules but you have not provided the data, I cannot try it on my own without them. Nevertheless, the graph shows that majority of the time is spent on validating some when conditions and our XPath evaluator is quite optimized (for some expressions, it can never be for all) so I am not sure there will be much to be done. If nothing else, once I identify the exact conditions, it may be possible to rewrite them to be more efficient.