search

mbr / simplekv

A simple key-value store for binary data.
http://simplekv.readthedocs.io
MIT License
152 stars 50 forks source link

Skip over the uninteresting parts of the XML for iter_keys #88

Closed xhochy closed 5 years ago

xhochy commented 5 years ago

Instead of parsing the full XML response and then creating Python object that contain the same information of the XML, we only parse the names from the XML response.

In our measurements this changes the duration of an iter_keys call from 20-30s to 3-4s. The request itself is now the main waiting time. Before the change we spent 90% of the time at full CPU load, now the CPU usage is negligible.

xhochy commented 5 years ago

Note: There is a PR over at https://github.com/Azure/azure-storage-python/pull/545 that would integrate this into azure-storage-python which would get rid of the internal API usage here.

criemen commented 5 years ago

While I can see the performance gain, I'm not happy with the code living in simplekv.

Please ping again if msft doesn't get the PR into azure-storage-python, but I don't think the code is really suitable for simplekv.

xhochy commented 5 years ago

Superseeded by https://github.com/mbr/simplekv/pull/89