search

MatrixAI / js-db

Key-Value DB for TypeScript and JavaScript Applications
https://polykey.com
Apache License 2.0
5 stars 0 forks source link

Introduce Snapshot Isolation OCC to DBTransaction #19

Closed CMCDragonkai closed 2 years ago

CMCDragonkai commented 2 years ago

Derived from #18

Description

This implements the snapshot isolation DB transaction.

This means DBTransaction will be automatically snapshot isolated, which means most locking will be unnecessary.

Instead when performing a transaction, there's a chance for a ErrorDBTransactionConflict exception which means there was a write conflict with another transaction.

Users can then decide on their discretion to retry the operation if they need to (assuming any non-DB side-effects are idempotent, noops or can be compensated). This should reduce the amount of locking overhead we need to do in Polykey. We may bubble up the conflict exception to the user, so the user can re-run their command, or in some cases, in-code we will automatically perform a retry. The user in this case can be the PK client, or the another PK agent or the PK GUI.

There is still one situation where user/application locks are needed, and that's where there may be a write-skew. See snapshot isolation https://en.wikipedia.org/wiki/Snapshot_isolation for more details and also https://www.cockroachlabs.com/blog/what-write-skew-looks-like/.

In the future we may upgrade to SSI (serializable snapshot isolation) which will eliminate this write-skew possibility.

~Additionally this PR will also enable the keyAsBuffer and valueAsBuffer options on the iterators, enabling easier usage of the iterators without having to use dbUtils.deserialize<T>(value) where it can be configured ahead of time.~ - already merged

See this https://www.fluentcpp.com/2019/08/30/how-to-disable-a-warning-in-cpp/ as to how to disable warnings in C++ cross platform.

Also see: https://nodejs.github.io/node-addon-examples/special-topics/context-awareness/

Issues Fixed

Tasks

Final checklist

CMCDragonkai commented 2 years ago

Bringing in the new changes from TypeScript-Demo-Lib-Native. But without the application builds because this is a pure library. And also removing deployment jobs.

CMCDragonkai commented 2 years ago

Ok it's time to finally bring in the leveldb source code and start hacking C++.

CMCDragonkai commented 2 years ago

I'm also going to solve the problem with key path here and probably prepare it for merging by cherry picking into staging, this can go along with a number of other CI/CD changes too.

@emmacasolin

CMCDragonkai commented 2 years ago

@emmacasolin I'm changing the DBIterator type to be like this and introducing DBIteratorOptions:

/**
 * Iterator options
 * The `keyAsBuffer` property controls
 * whether DBIterator returns KeyPath as buffers or as strings
 * It should be considered to default to true
 * The `valueAsBuffer` property controls value type
 * It should be considered to default to true
 */
type DBIteratorOptions = {
  gt?: KeyPath | Buffer | string;
  gte?: KeyPath | Buffer | string;
  lt?: KeyPath | Buffer | string;
  lte?: KeyPath | Buffer | string;
  limit?: number;
  keys?: boolean;
  values?: boolean;
  keyAsBuffer?: boolean;
  valueAsBuffer?: boolean
};

/**
 * Iterator
 */
type DBIterator<K extends KeyPath | undefined, V> = {
  seek: (k: KeyPath | string | Buffer) => void;
  end: () => Promise<void>;
  next: () => Promise<[K, V] | undefined>;
  [Symbol.asyncIterator]: () => AsyncGenerator<[K, V]>;
};

This means now KeyPath becomes pre-eminent, and anywhere I have KeyPath | Buffer | string, the Buffer or string is intepreted as a singleton KeyPath.

CMCDragonkai commented 2 years ago

This also allows the key returned by the iterator to be later used by seek or the range options.

This will impact downstream EFS and PK usage though. But find and replace should be sufficient.

CMCDragonkai commented 2 years ago

I've updated the DB._iterator to use the new DBIteratorOptions and DBIterator. I haven't tested yet, only type checked.

CMCDragonkai commented 2 years ago

I've also created a utils.toKeyPath function that can be used to easily convert possible keypaths into keypaths. This can be used in our get, put, del functions to Buffer and string into KeyPath.

The keyAsBuffer option now means that the returned KeyPath is converted to an array of string compared to an array of buffers as it would normally be.

tegefaulkes commented 2 years ago

If the problem is double encoding then couldn't this be solved by having clear barriers to where and when the encoding is applied? We need the encoded form internally but the the user needs the un-encoded form. Doesn't this mean encoding conversion only needs to happen when we pass to and from the user?

CMCDragonkai commented 2 years ago

If the problem is double encoding then couldn't this be solved by having clear barriers to where and when the encoding is applied? We need the encoded form internally but the the user needs the un-encoded form. Doesn't this mean encoding conversion only needs to happen when we pass to and from the user?

There are clear barriers. It's only applied when user passes input into the system. The problem is only for the iterator. Because iterator returns your the "key" that you're supposed to use later for other operations. Then the solution is not encode the key then again while inside the iterator. Solving the double escaping problem.

CMCDragonkai commented 2 years ago

Something I recently discovered the effect of the empty key.

That is if you have a KeyPath of [] this becomes [''].

Therefore if you use the empty key, in master I believe you would not be able to see this key when iterating.

To solve this, I had to change the default iterator option to instead of using gt to use gte.

For example:

    if (options_.gt == null && options_.gte == null) {
      options_.gte = utils.levelPathToKey(levelPath);
    }

This now ensures that the empty key shows up within the level during iteration.

CMCDragonkai commented 2 years ago

Extra test cases were added for empty keys now, and this actually resolves another bug involving empty keys.

describe('utils', () => {
  const keyPaths: Array<KeyPath> = [
    // Normal keys
    ['foo'],
    ['foo', 'bar'],
    // Empty keys are possible
    [''],
    ['', ''],
    ['foo', ''],
    ['foo', '', ''],
    ['', 'foo', ''],
    ['', '', ''],
    ['', '', 'foo'],
    // Separator can be used in key part
    ['foo', 'bar', Buffer.concat([utils.sep, Buffer.from('key'), utils.sep])],
    [utils.sep],
    [Buffer.concat([utils.sep, Buffer.from('foobar')])],
    [Buffer.concat([Buffer.from('foobar'), utils.sep])],
    [Buffer.concat([utils.sep, Buffer.from('foobar'), utils.sep])],
    [Buffer.concat([utils.sep, Buffer.from('foobar'), utils.sep, Buffer.from('foobar')])],
    [Buffer.concat([Buffer.from('foobar'), utils.sep, Buffer.from('foobar'), utils.sep]),
    ],
    // Escape can be used in key part
    [utils.esc],
    [Buffer.concat([utils.esc, Buffer.from('foobar')])],
    [Buffer.concat([Buffer.from('foobar'), utils.esc])],
    [Buffer.concat([utils.esc, Buffer.from('foobar'), utils.esc])],
    [Buffer.concat([utils.esc, Buffer.from('foobar'), utils.esc, Buffer.from('foobar')])],
    [Buffer.concat([Buffer.from('foobar'), utils.esc, Buffer.from('foobar'), utils.esc])],
    // Separator can be used in level parts
    [Buffer.concat([utils.sep, Buffer.from('foobar')]), 'key'],
    [Buffer.concat([Buffer.from('foobar'), utils.sep]), 'key'],
    [Buffer.concat([utils.sep, Buffer.from('foobar'), utils.sep]), 'key'],
    [Buffer.concat([utils.sep, Buffer.from('foobar'), utils.sep, Buffer.from('foobar')]), 'key'],
    [Buffer.concat([Buffer.from('foobar'), utils.sep, Buffer.from('foobar'), utils.sep]), 'key'],
    // Escape can be used in level parts
    [Buffer.concat([utils.sep, utils.esc, utils.sep]), 'key'],
    [Buffer.concat([utils.esc, utils.esc, utils.esc]), 'key'],
  ];
  test.each(keyPaths.map(kP => [kP]))(
    'parse key paths %s',
    (keyPath: KeyPath) => {
      const key = utils.keyPathToKey(keyPath);
      const keyPath_ = utils.parseKey(key);
      expect(keyPath.map((b) => b.toString())).toStrictEqual(
        keyPath_.map((b) => b.toString()),
      );
    }
  );
});
CMCDragonkai commented 2 years ago

Needs rebase on top of staging now.

CMCDragonkai commented 2 years ago

Time to rebase.

ghost commented 2 years ago
👇 Click on the image for a new way to code review - Make big changes easier — review code in small groups of related files - Know where to start — see the whole change at a glance - Take a code tour — explore the change with an interactive tour - Make comments and review — all fully sync’ed with github [Try it now!](https://app.codesee.io/r/reviews?pr=19&src=https%3A%2F%2Fgithub.com%2FMatrixAI%2Fjs-db)

Review these changes using an interactive CodeSee Map

Legend

CodeSee Map Legend

CMCDragonkai commented 2 years ago

I was looking at 2 codebases to understand how to integrate leveldb.

  1. https://github.com/Level/classic-level
  2. https://github.com/Level/rocksdb

It appears the classic-level is still using leveldb 1.20 which is 5 years old. The rocksdb is bit more recent.

The leveldb codebase has a bit more of a complicated build process.

The top level binding.gyp includes a lower level gyp file:

    "dependencies": [
      "<(module_root_dir)/deps/leveldb/leveldb.gyp:leveldb"
    ],

The deps/leveldb/leveldb.gyp file contains all the settings to actually compile the leveldb as a shared object.

Note that the binding.cc does import leveldb headers like:

#include <leveldb/db.h>

These headers are not specified by the binding.gyp, it's possible that by specifying the dependencies, the inclusion headers made available to the top-level target.

The leveldb.gyp also specifies a dependency in snappy:

    "dependencies": [
      "../snappy/snappy.gyp:snappy"
    ],

These are all organised under deps. These are not git submodules. Except for the snappy.

I think for us, we should just copy the structure of deps, as well as the submodule configuration. We can preserve the leveldb.gyp and snappy.gyp, and then just write our own binding.gyp that uses it. Then things should proceed as normal.

So it seems that leveldb has alot of legacy aspects, rocksdb compilation is alot cleaner. In fact lots of tooling has stopped using gyp file. We can explore that later.

CMCDragonkai commented 2 years ago

The presence of the snappy submodule means cloning has to be done now with git clone --recursive. If you have already cloned, setup the git submodule with git submodule update --init --recursive. It should bring in data into deps/snappy/snappy.

CMCDragonkai commented 2 years ago

While porting over the binding.gyp from classic-level, we have to be aware of: https://github.com/MatrixAI/TypeScript-Demo-Lib/pull/38#issuecomment-1124494772

The cflags and cflags_cc both apply to g++ and g++ is used when the file is cpp.

However both c and cpp files may be used at the same time, so we should be setting relevant flags for both cflags and cflags_cc.

I'm not sure if this is true for non-linux platforms. The upstream classic-level does not bother with cflags_cc. However we will set it just so that we can get the proper standard checks.

CMCDragonkai commented 2 years ago

I found out what cflags+ means. It is based on:

If the key ends with a plus sign (+), the policy is for the source list contents to be prepended to the destination list. Mnemonic: + for addition or concatenation.

So cflags+ will prepend rather than appending as normal. This sets the visibility=hidden to be true.

This is required due to: https://github.com/nodejs/node-addon-api/blob/main/doc/setup.md. The reason this is required is documented here: https://github.com/nodejs/node-addon-api/pull/460#issuecomment-544143695. This should go into TS-Demo-Lib-Native too.

CMCDragonkai commented 2 years ago

I am using this .vscode/c_cpp_properties.json as configuration, so that vscode finds the right headers, there's no automatic way to generate this atm, it's based on paths that is inside the nix-shell:

{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**",
                "/nix/store/zl4bvsqfxyx5vn9bbhnrmbmpfvzqj4gd-nodejs-16.14.2/include/**"
            ],
            "defines": [],
            "compilerPath": "/nix/store/58pwclg9yr437h0pfgrnbd0jis8fqasd-gcc-wrapper-11.2.0/bin/gcc",
            "cStandard": "c99",
            "cppStandard": "c++17",
            "intelliSenseMode": "${default}"
        }
    ],
    "version": 4
}

Paths are from:

which gcc
which node
echo $npm_config_nodedir
CMCDragonkai commented 2 years ago

Noticed that the sync option for PutOptions and DelOptions should be switched to true. This ensures that when the transaction is committed, it is infact committed to disk. Which means if the OS crashes, we won't be losing data.

Probably makes sense that the DBTransaction should be using the internal write batch. I'm not sure if they are journaling the write batch, but it probably isn't, and it's in-memory. The only issue is that you cannot dump what's in the batch, which limits its debuggability.

CMCDragonkai commented 2 years ago

It is possible or the DB to be corrupted in the case of power failure. In that case the repair_db should be called. There's no guarantee that the DB will return to normal state, but we should have this available in our pk agent start command and auto-repair in case we cannot open the DB. This is also why it is important to always backup PK agent state to other PK agents.

CMCDragonkai commented 2 years ago

I believe all the types for src/leveldb is done.

I've also discovered a nice way of doing phantom types. It relies on the same mechanism as Opaque type constructor.

Note that I've updated the Opaque type to use a unique symbol as the differentiating key.

This only exists at compile-time and is elided at runtime.

Think of it as the "tag" in Tagged Unions used to implement algebraic data types.

So for example:

/**
 * Iterator object
 * A `napi_external` type
 */
type Iterator<
  K extends string | Buffer = string | Buffer,
  V extends string | Buffer = string | Buffer,
> = Opaque<'Iterator', object> & {
  readonly [brandIteratorK]: K;
  readonly [brandIteratorV]: V;
};
declare const brandIteratorK: unique symbol;
declare const brandIteratorV: unique symbol;

Now the K and V types are phantom types.

Phantom types don't work normally in TS due to structural typing system, but by telling TS that there would be a special key differentiating it, TS stops complaining about the fact that we aren't using the type parameter. But in reality we never use the type parameter and just use casting to create the specific Iterator type.

CMCDragonkai commented 2 years ago

Should update the Opaque usage in PK to be:

/**
 * Opaque types are wrappers of existing types
 * that require smart constructors
 */
type Opaque<K, T> = T & { readonly [brand]: K };
declare const brand: unique symbol;

@emmacasolin

CMCDragonkai commented 2 years ago

With this, we should be able to proceed with integrating src/leveldb and replace our usage of leveldown completely.

However the purpose of this is to also get snapshot isolation transactions. I'll do this after first prototyping substitution of leveldown for our internal src/leveldb.

CMCDragonkai commented 2 years ago

LevelDB corruption errors are represented in JS as an exception with LEVEL_CORRUPTION code. It was introduced here: https://github.com/Level/classic-level/commit/39cad430930e1196a66197de04448392f7e40ef1.

This should mean we can detect if this is the exception and attempt an automatic repair.

CMCDragonkai commented 2 years ago

It uses https://nodejs.org/api/n-api.html#napi_create_error, which should create an Error object with a code property of "LEVEL_CORRUPTION.

So we should be able to do:

if (e.code === 'LEVEL_CORRUPTION') { 
  // attempt repair before opening
}

Not sure how this would be tested though.

CMCDragonkai commented 2 years ago

The src/leveldb now exports both:

All leveldb types are also prefixed with LevelDB making it easier to differentiate them from our DB's types.

CMCDragonkai commented 2 years ago

I've augmented the promisify function now with additional types that preserve typing info. It however still won't work well on overloaded function signatures.

/**
 * Convert callback-style to promise-style
 * If this is applied to overloaded function
 * it will only choose one of the function signatures to use
 */
function promisify<
  T extends Array<unknown>,
  P extends Array<unknown>,
  R extends (T extends [] ? void : (T extends [unknown] ? T[0] : T)),
>(
  f: (...args: [ ...params: P, callback: Callback<T> ]) => unknown
): (...params: P) => Promise<R> {
  // Uses a regular function so that `this` can be bound
  return function (...params: P): Promise<R> {
    return new Promise((resolve, reject) => {
      const callback = (error, ...values) => {
        if (error != null) {
          return reject(error);
        }
        if (values.length === 0) {
          (resolve as () => void)();
        } else if (values.length === 1) {
          resolve(values[0] as R);
        } else {
          resolve(values as R);
        }
        return;
      };
      params.push(callback);
      f.apply(this, params);
    });
  };
}

This can be ported to EFS and PK @tegefaulkes @emmacasolin but low priority.

CMCDragonkai commented 2 years ago

I've just had a look at the rocksdb interface, it appears the exported C++ functions are exactly the same.

NAPI_INIT() {
  NAPI_EXPORT_FUNCTION(db_init);
  NAPI_EXPORT_FUNCTION(db_open);
  NAPI_EXPORT_FUNCTION(db_close);
  NAPI_EXPORT_FUNCTION(db_put);
  NAPI_EXPORT_FUNCTION(db_get);
  NAPI_EXPORT_FUNCTION(db_get_many);
  NAPI_EXPORT_FUNCTION(db_del);
  NAPI_EXPORT_FUNCTION(db_clear);
  NAPI_EXPORT_FUNCTION(db_approximate_size);
  NAPI_EXPORT_FUNCTION(db_compact_range);
  NAPI_EXPORT_FUNCTION(db_get_property);

  NAPI_EXPORT_FUNCTION(destroy_db);
  NAPI_EXPORT_FUNCTION(repair_db);

  NAPI_EXPORT_FUNCTION(iterator_init);
  NAPI_EXPORT_FUNCTION(iterator_seek);
  NAPI_EXPORT_FUNCTION(iterator_end);
  NAPI_EXPORT_FUNCTION(iterator_next);

  NAPI_EXPORT_FUNCTION(batch_do);
  NAPI_EXPORT_FUNCTION(batch_init);
  NAPI_EXPORT_FUNCTION(batch_put);
  NAPI_EXPORT_FUNCTION(batch_del);
  NAPI_EXPORT_FUNCTION(batch_clear);
  NAPI_EXPORT_FUNCTION(batch_write);
}

However some of the types may be slightly different.

Furthermore rocksdb does have support for block level encryption. https://github.com/facebook/rocksdb/search?q=encryption

Seems like it should be relatively simple to port over.

I do want to point out something about the indexing work:

Point is, if you stay at the key-value level, you'd need a deterministic order-preserving encryption scheme to be flexible. There's alot of research on this atm like: https://arxiv.org/abs/1706.00324. However it's quite speculative, I believe it is possible, but it seems having block encryption is just simpler and will solve the problem directly. Order preserving can still be useful if you want others to be able to "work" on the encrypted data without know the plaintext values.

CMCDragonkai commented 2 years ago

The DB.ts is only using these functions:

  NAPI_EXPORT_FUNCTION(db_init);
  NAPI_EXPORT_FUNCTION(db_open);
  NAPI_EXPORT_FUNCTION(db_close);
  NAPI_EXPORT_FUNCTION(db_put);
  NAPI_EXPORT_FUNCTION(db_get);
  NAPI_EXPORT_FUNCTION(db_del);

  NAPI_EXPORT_FUNCTION(iterator_init);
  NAPI_EXPORT_FUNCTION(iterator_seek);
  NAPI_EXPORT_FUNCTION(iterator_close);
  NAPI_EXPORT_FUNCTION(iterator_nextv);

  NAPI_EXPORT_FUNCTION(batch_do);

I'm not currently using:

  NAPI_EXPORT_FUNCTION(db_clear);

Because in C++ it appears to be doing the same thing that I'm already doing by relying on the iterator.

CMCDragonkai commented 2 years ago

I've ported all of those to DB.ts now.

CMCDragonkai commented 2 years ago

Note that having ios/android builds of the native addon would only relevant if we intend to stay in the node ecosystem when on mobile. If node itself won't be embeddable (as a library or separate runtime) on mobile, then there's no need for ios/android builds of this addon.

CMCDragonkai commented 2 years ago

Hit this problem again: https://github.com/facebook/jest/issues/12814.

So will need to use toEqual when comparing buffers coming out of the leveldb, not toStrictEqual.

I'm not sure but leveldown and classic level doesn't seem to have this problem, but I'm pretty sure they didn't wrap their buffers. So I wonder how I'm able to test like this.

CMCDragonkai commented 2 years ago

The DB.test.ts is working now directly against the native addon.

However I discovered that boolean options that have a default value of false in the C++ code, if passed as undefined would instead actually be asserted to being true.

It appears that the BooleanProperty would:

This cause some confusion on the reverse option. This is because I've starting assigning the options even if they are not set, which would set it as undefined.

This would affect any option that is meant to default to false, and we would want to ensure that if passed as undefined, the option should be cast explicitly to true with !!(options?.[OPTION]).

CMCDragonkai commented 2 years ago

If you're logging stuff out from C++, make sure to use STDERR otherwise it will clobber console.log outputs:

#include <ios>
#include <iostream>

    fprintf(stderr, "BOOLEAN RESULT: %s %d\n", key, result);
    std::cerr << std::boolalpha << result << "\n";
CMCDragonkai commented 2 years ago

Note that napi_get_value_bool will end up casting undefined values into a number. It's important to ensure that options are strictly defined with optionality. This means x?: boolean cannot be x: undefined when passed to the native addon.

I've created a filterUndefined utility function to remove any undefined properties to ensure that these properties are removed prior to passing into the native addon.

CMCDragonkai commented 2 years ago

Before attempting the other tasks, let's see what happens if we use rocksdb with src/rocksdb as we may be able to get free SI (and free block encryption) if we just integrate at the C++ level.

CMCDragonkai commented 2 years ago

I found a fork of leveldb's rocksdb that provides a guide on how to get SI and other advanced functionality out of rocksdb: https://github.com/lu4/rocksdb-ts. Would be worth investigating so potentially we can just use https://github.com/facebook/rocksdb/blob/main/examples/transaction_example.cc for free.

CMCDragonkai commented 2 years ago

Added a new test iterating sublevels with range it demonstrates how to use the range options which can be quite tricky.

CMCDragonkai commented 2 years ago

I tried benchmarking the current code.

It looks like this:

[nix-shell:~/Projects/js-db]$ npm run bench

> @matrixai/db@4.0.5 bench
> rimraf ./benches/results && ts-node -r tsconfig-paths/register ./benches

Running "DB1KiB" suite...
Progress: 100%

  get 1 KiB of data:
    43 276 ops/s, ±3.66%   | fastest

  put 1 KiB of data:
    28 117 ops/s, ±2.99%   | 35.03% slower

  put zero data:
    35 105 ops/s, ±2.27%   | 18.88% slower

  put zero data then del:
    18 127 ops/s, ±2.42%   | slowest, 58.11% slower

Finished 4 cases!
  Fastest: get 1 KiB of data
  Slowest: put zero data then del

Saved to: benches/results/DB1KiB.json

Saved to: benches/results/DB1KiB.chart.html
Running "DB1MiB" suite...
Progress: 100%

  get 1 MiB of data:
    1 932 ops/s, ±4.57%    | 93.21% slower

  put 1 MiB of data:
    19 ops/s, ±135.25%       | slowest, 99.93% slower

  put zero data:
    28 456 ops/s, ±4.25%   | fastest

  put zero data then del:
    13 221 ops/s, ±6.31%   | 53.54% slower

Finished 4 cases!
  Fastest: put zero data
  Slowest: put 1 MiB of data

Saved to: benches/results/DB1MiB.json

Saved to: benches/results/DB1MiB.chart.html

Everything looks similar to before EXCEPT the put 1 MiB of data.

That is now around 20 ops/s.

Whereas previously on staging branch that's currently around 467 ops/s.

How did it drop so much. Firstly I think it's because the sync option is true (which I set to false by default). But I've made it false, but it doesn't change anything.

This is quite strange since all other ops are basically the same.

Need to find out why my put is so slow. Especially since we are just calling leveldbP.put. I'm going to try the callback first.

CMCDragonkai commented 2 years ago

Because I pulled this out of classic-level and not leveldown, the slowness in put might have been introduced by classic-level. The current staging is still using leveldown. The native addon appear to be incompatible.

CMCDragonkai commented 2 years ago

I tried with classic level, it's not that. So now I need to think if it's now due to our promise implementation or something else. Very strange. Maybe there's optimisation flags that aren't turned on.

CMCDragonkai commented 2 years ago

It's got nothing to do with the C++ code.

I installed the classic-level node package, and then copied their prebuilt binary to prebuilds/linux-x64.

So the JS code is then using their library, but for some reason, by directly using the JS/TS that I'm wrtiting, I'm getting 19 ops/s, ±145.79%.

Not only is the 19 number strange, but it's also the margin of error is so high as well. Perhaps there's something about the way the database is being setup that is different.

CMCDragonkai commented 2 years ago

It's also not the src/leveldb code. Benching that gives us 471 ops/s. So it's something that's happening inside DB.ts.

CMCDragonkai commented 2 years ago

It turns out its the iterator.

If the iterator is left open, it appears to cause performance issues for the 1MiB benchmark. This is quite surprising that the existence of an open iterator can cause such issues.

We have a bug where the iterator is not being closed.

I need to refactor the next call inside the iterator so that it is actually being closed.

CMCDragonkai commented 2 years ago

The await iterator.close() must not be called twice. If you call it twice, it's an error.

CMCDragonkai commented 2 years ago

Ok so there's some missing logic in the iterator implementation atm:

  1. We need to block calls to seek, next, and close. That is these calls cannot be run concurrently.
  2. The ready decorator currently has a block parameter that enables blocking these calls during the async lifecycle. So we should use a lock.
  3. Upstream classic-level and variants just throw exceptions if they are called at the same time, we can use our async-locks to ensure the methods are called.
  4. The close should based on our CreateDestroy system.
  5. The existence of the cache must be considered as well...

So I'm thinking we create a class called DBIterator to take care of this so it's clearer and fits within our existing async class design.

The DBTransaction is a bit more complicated since it has to maintain 2 iterators, however this might change once we bring in rocksdb since it might already have its own transaction state that we can get for free.

CMCDragonkai commented 2 years ago

It is essential to run git submodule update --init --recursive now if we using submodules. Also use git submodule status to see what the status of the submodules are.

Currently:

[nix-shell:~/Projects/js-db]$ git submodule status
 09c7e96eac1ab983f97ce9e0406730b8014b3398 deps/rocksdb/rocksdb (v5.8-3265-g09c7e96ea)
 b02bfa754ebf27921d8da3bd2517eab445b84ff9 deps/snappy/snappy (1.1.7)
CMCDragonkai commented 2 years ago

We now have DBIterator, this has taken over the functionality that was specified inside DB.ts before.

The benchmarks are back to how they were before, and we no longer have the iterator leak. We've made some changes to how the iterator next and seek calls work. Concurrent calls to next will block rather than throwing an error, but if a seek call is made while a next is in progress, that will result in ErrorDBIteratorBusy.

CMCDragonkai commented 2 years ago

Although DBIterator is an CreateDestroy, it doesn't actually have an asynchronous creation at all, and it is just created from the constructor. This is because the iterator() call is supposed to just return the iterator object without wrapping it as a promise.