Comcast / sirius

A distributed system library for managing application reference data
http://comcast.github.io/sirius/
Apache License 2.0
298 stars 49 forks source link

Use BufferedInputStream for read operations #136

Closed HaloFour closed 5 years ago

HaloFour commented 5 years ago

Change Sirius read ops to use a BufferedInputStream over the underlying Uberstore files to improve performance, particularly around bootstrapping.

Rudimentary benchmark running XtvAPI locally on my Macbook Pro:

Sirius 1.2.6:

Finished firing up Session Sirius on HQSML-1685301:2553 in 501986 ms
Finished firing up Metadata Sirius on HQSML-1685301:2552 in 1237428 ms

Sirius 1.2.8-SNAPSHOT (with file op mods):

Finished firing up Session Sirius on HQSML-1685301:2553 in 187282 ms
Finished firing up Metadata Sirius on HQSML-1685301:2552 in 224490 ms
HaloFour commented 5 years ago

Testing on a prod-sized XtvAPI instance with full metadata and session Uberstores:

Sirius 1.2.6:

Session BootstrapTime  : Some(Some(296165)) - 4.9 minutes
Metadata BootstrapTime  : Some(Some(2054482)) - 34 minutes

Sirius 1.2.8-SNAPSHOT (with buffering enabled, 8k buffer):

Session BootstrapTime  : Some(Some(166132)) - 2.8 minutes
Metadata BootstrapTime  : Some(Some(1375710)) - 22.9 minutes