Structured Change Detection for Mutation Events and Schema Compliance

MatrixAI / js-encryptedfs

Encrypted Filesystem for TypeScript/JavaScript Applications

https://polykey.com

Apache License 2.0

10 stars 3 forks source link

Structured Change Detection for Mutation Events and Schema Compliance #55

Open joshuakarp opened 2 years ago

joshuakarp commented 2 years ago

Specification

Whilst considering vault schemas, it became apparent that it would be extremely beneficial to implement some kind of filesystem event-watching API integrated into the EFS, such that we can track changes before and after they're made directly to the files in the EFS. That is, before and after changes are made to files, we can generate an accumulating list of these changes. Then, we can provide hooks for these pre-edit and post-edit changes.

This would allow us to solve 3 problems in one:

enforce the vault schema before a file change is made (i.e. pre-edit vs. having to do a post-edit check)
easier automatic commit message generator
accurate elimination of dirty commits

The latter 2 problems are currently being solved by a hacky, post-edit solution through recursive scans over the EFS (see the top-level comment from @scottmmorris here).

Additional context

vault schemas epic https://github.com/MatrixAI/js-polykey/issues/222

Some further discussion of pre-edit and post-edit from the vaults refactoring MR:

Tasks

CMCDragonkai commented 2 years ago

@scottmmorris can you expand on the relevant items in the vaultsrefactoring MR https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/205 and why is this issue important.

joshuakarp commented 2 years ago

There's been some further discussion of this with VaultInternal here https://vimeo.com/manage/videos/652700328

Specifically, we'd be able to significantly reduce the complexity of VaultInternal.commit by implementing this.

Currently, commit performs a recursive search over all files within the vault directory (in the EFS), utilising the iso-git-provided statusMatrix to deduce the changes made. This would be completely unnecessary with the addition of this event-watching API.

It would also allow us to solve https://github.com/MatrixAI/js-polykey/issues/260 (single digit modification is not detected by isomorphic-git's status call) because we'd no longer need to rely on the statusMatrix (which is most likely the cause of this "mystery bug").

CMCDragonkai commented 1 year ago

Interesting usage of git for finding changed files: https://github.com/facebook/jest/tree/main/packages/jest-changed-files.

Relies on shelling out to the git command. Obviously pre-edit and post-edit change events is more efficient, but interesting to compare.