Performance issue caused by references when validating a big bundle

[darena-antonw]

Hello, wanted to report a performance issue

the containing bundle contains ~8000 observation resources that we'd like to insert as part of a batch operation

validating this bundle takes over a minute.

I've done some investigation and basically the gist is:

• for every reference the ReferencedInstanceValidator will try to find the referenced resource locally within the bundle looping all elements.
• this check causes the enumeration of PocoElementNode.Children of wrapping elements which in turn causes:
  • enumeration of NamedChildren
  • creating location strings
  • creation of PocoElementNode

all of this happens for each entry for each reference, so by the end this logic runs more than x10 million times...

Anything you can suggest as a short term solution or a way to disable this?

Also (I didn't dig far enough) why is the validator trying to find the referenced resource? what will it do with it when it finds it?

sample.json

var fileStream = File.OpenRead("sample.json");
var bundle = await JsonSerializer.DeserializeAsync<Bundle>(fileStream, new JsonSerializerOptions().ForFhir());
var result = validator.Validate(bundle);

[mmsmits]

MIght have to be improved in ScopedNode

[mmsmits]

Timeboxed for 4 hours for research

[Kasdejong]

Findings:

• ScopedNode has a lot of redundancy and GC right now. Improving this could consist of multiple steps.
• We should probably improve this in the SDK, not in the validator, unless we want to keep an in memory reference tree in the validation state. I suggest we do this in scoped node, or some other helper class in the SDK instead.
• For big bundles and contained resources, we can provide temporary relief by using a dict instead of a list (See FirelyTeam/firely-net-sdk#2845
• Overall, there is a lot of work to be done, and anything we improve now will help overall performance in SDK 6 (when scoped node will get a different role)

Proposed actions:

• Pull FirelyTeam/firely-net-sdk#2845 before next release
• See new issue FirelyTeam/firely-net-sdk#2846

[Kasdejong]

Closed in favour of https://github.com/FirelyTeam/firely-net-sdk/issues/2846. We will address this properly in SDK 6