search

samcragg / sharpkml

SharpKML is an implementation of the Open Geospatial Consortium (OGC) KML 2.2 standard developed in C#, able to read/write both KML files and KMZ files.
MIT License
158 stars 51 forks source link

Memory improvements #10

Closed sylvaneau closed 6 years ago

sylvaneau commented 6 years ago

My company uses actively your lib to manage KML files. We are currently running into memory issues with large files (>20k features). I made some "improvements" that I would like to share with you and to get your feedback. Each commit contains a specific memory improvement. Our reference file contains 36000 placemarks associated with some metadata (file size 67.5 MB).

We use Visual Studio 2017 performance profiler to measure memory usage, with our test file and the stock library memory usage is : 847.96 MB

1bcc653 : Element inner text is stored into a StringBuilder, this object is quite heavy and it seems we do not need to build complex strings, so I suggest to use a basic string instead. Memory usage : 785.72 MB

010a179 : XmlNamespaceManager is a complex object and we do not use all its features. Basicaly what we need is a dictionnary storing the differents namespaces and a way to retreive them excluding the default xml namespace.

Memory usage : 377.23 MB (:+1:)

4a0cb0b : The first two changes were quite simple. This one is a bit more complex and slightly changes the library behaviour. In the current implementation, valid child types are registered for each instances of an Element which implies having a huge number of Dictionaries in memory to store these associations. My suggestion here is to register valid child types in a static manner, so there is only one dictionnary containing an entry for each KML type. This way valid child types are registered once (through the static constructor of each type).

Memory usage : 311.14 MB

I hope I made theses changes the right manner, feel free to contact me if you wan't to discuss anything about this code.

Regards

Sylvain

samcragg commented 6 years ago

That's great work, thanks so much for spending time investigating and submitting the improvements!

I like what you've done and it looks OK so will merge in over the weekend, as I want to get rid of the empty constructors (nice idea to use the static constructors, I've wanted to move to removing the registrations from the constructors as well but was thinking of attributes but the way you've done it is more performant so happy with that)

Thanks again for the PR

sylvaneau commented 6 years ago

Cool ! :-)

I might send you another PR in the next few days (but probably not this week) about KMZ file handling, same aim, saving memory ;-)