feat(sltt-app): store vcrs and docs per client with multiple items in each file

[ericpyle]

What issue(s) is this trying to resolve?

• feat(sltt-app): store vcrs and docs per client with multiple items in each file #31

How does it all work?

• PROBLEM1: vcrs store per-client data that should not be shared across other clients
• SOLUTION1: add clientId to api parameters so that vcrs get stored per clientId
• PROBLEM2: storing mod doc per-file led to great filesystem inefficiences when transferring thousands and thousands of files.

• SOLUTION2:

  • store remote mod docs in single remote/remote.sltt-docs file, appending data to end of file, and use a {clientId}.sltt-spots file to keep track of the last spot each {clientId} read from that file. Furthermore, the remote Seq number is always appended to the end of each line so each client only needs to read the last 9 bytes from the file to know the latest remote seq stored in the file.

  • Likewise, store local mod docs in per client local/{clientId}.sltt-docs each client having its own local/{clientId}.sltt-spots file to keep track of the last place each client read from each client file. The client is expected to handle merging and sorting of data, and to enforce winner vs. loser logic that stores/applies only the latest modDate changes per mod doc _id.

  • Likewise, store all vcr docs per portion, and use bottleneck batches to collect all the writes to the file within a span of 1 second.

What particularly has changed?

• create blobs.d clients.d and connections.d ts types. updated docs.d ts types
• add connections.ts for probing and connecting to lan (or local) drive
• refactored blobs api to blobs.ts
• add clients.ts to handle updating clients/{clientId}.sltt-users file
• remove V0 docs api code
• change lanStoragePath base path to be mutable by connection url
• add listVcrFiles api for getting all the vcr files so client can load each project's portion vcrs at once.
• update tests
• add packages bottleneck, fs/extra and lodash dev types

Steps for testing

1. run unit tests
2. run client initialization and make new changes online and offline. Do Clear Video Cache!!! and Refetch local database... while offline
3. try configuring shared drive where multiple computers could connect and run sltt-app

ticket: https://github.com/ubsicap/sltt-app/issues/31 commit-convention: https://www.conventionalcommits.org/en/v1.0.0/

[AllisonPyle]

Notes so far on test step 2: I used a recently installed version of SLTT-App and did not manually set up any folders or tell it to use local storage.

I did the following while offline: -recorded video for Acts 3B -created "New Passage" after that -added "New Document" (body: "here") in New Passage -uploaded new video to replace previous one for Acts 3

Here's a screenshot showing pending uploads (with the video I dragged and dropped visible):

Then I clicked on "Clear Video Cache!!!" and restarted the app. It DOES NOT show any pending uploads, and the videos I recorded or uploaded ARE NOT loading. It DOES still have the new passage and new passage document I created.

Screenshot after restarting:

I reconnected to the internet, and the behavior is the same.

Is this expected?

I have not yet tested with "Refetch local database..." I wanted to make notes on these steps for now.