search

MatrixAI / js-encryptedfs

Encrypted Filesystem for TypeScript/JavaScript Applications
https://polykey.com
Apache License 2.0
10 stars 3 forks source link

Loading and storing raw data buffers in the database #52

Closed scottmmorris closed 2 years ago

scottmmorris commented 2 years ago

Description

This PR will change the data buffers to be loaded and stored into the db using the raw parameter to avoid the unnecessary cost of JSON serialisation and deserialisation

Tasks

  1. [x] Introduce the raw parameter into the data transaction calls to the db
  2. [x] Run the new benchmark tests with the update
CMCDragonkai commented 2 years ago

All tests passed and lintfixed?

scottmmorris commented 2 years ago

Yep, linted and all tests passing with the latest commit

CMCDragonkai commented 2 years ago

Ok please squash as well. I'll add my bench marks on top.

scottmmorris commented 2 years ago

Have squashed

CMCDragonkai commented 2 years ago

New benchmarks!

image

image

image

CMCDragonkai commented 2 years ago

Compared to the master:

image

image

image

CMCDragonkai commented 2 years ago

Looks like it's about up to 10% improvement in get and put speeds for smaller block sizes. Negligible difference for larger block sizes. I would have expected more performance improvement for larger blocks. Maybe JSON serialisation isn't a big deal for larger blocks, but the storage usage should decrease!

CMCDragonkai commented 2 years ago

Sorry I just realised that the benchmarks shouldn't change much because we don't actually have EFS benchmarks yet. That's still pending due to #50. So ignore the above screencaps.