Feedback: How to access records with unreplicated changes?

[MrChadMWood]

I am looking for a way to check on pending pushes. Essentially, track in real-time (on the client-side) which how many documents are needing to be replicated to the server-side still.

I want to implement a counter with this functionality, so the user knows if some of their work is still client-side only (and how much). I.e., a user makes a change to a document, and they see the counter increment by one; once the document replicates to the server, the counter simply decrements. Aside from that, I also want to implement the option to download the unreplicated documents as a .json file.

Simply put, I'm just looking for:

• A way for the client to know how many docs still need to be pushed to server
• A way to know which docs those are

The idea is that users can see if the counter isn't decrementing, and in that case they know something is potentially wrong. They can download their work as a last resort, if my app has any issues. I don't want to be in a predicament where the data is lost simply because they had no options to download it from their browser.

Can it be done in a way that works regardless of how the change was made? Or, do I need to create observables for each way of modifying the collection? Thank you for any feedback!

[pubkey]

For the unreplicated pulls, your server has to tell the client the number. For pushes there is no official feature. But you could get the push-checkpoint from the replication metadata and then run a query against the storageInstance of the collection.

[MrChadMWood]

I will try the approach you've given. Where's the push-checkpoint located on the replicationState object?

[MrChadMWood]

@pubkey Just following up, how should the push-checkpoint be accessed?

[pubkey]

This is not documented. You can check the replication internals to fetch the checkpoint.

Edit: In the replication-protocol subpackage there is the method getLastCheckpointDoc which can be used to fetch the latest checkpoint in both directions (pull and push).

[MrChadMWood]

Thanks that seems like it is what I need.

[MrChadMWood]

@pubkey I'm not able to get a response with direction set to "up"

14:29:33.033 await getLastCheckpointDoc(replicationState.internalReplicationState, 'up')
14:29:33.045 undefined 

Using "down" works though. Any idea what might cause this?

[pubkey]

Maybe there is not checkpoint stored yet when the replication is started?

[MrChadMWood]

I had already pushed some docs to the backend, which I assume is what triggers a checkpoint creation.

Do you know if something like this can work instead?

// Subscribes to sent$ to keep last pushed doc as custom /push checkpoint
// to allow for comparing 'last-push'.doc.updated_at and unreplicatedDoc.updated_at
// Should work out that unreplicatedDoc.updated_at > 'last-push'.doc.updated_at

replicationState.sent$.subscribe(async doc => {
  replicationState.collection.upsertLocal('last-push', { doc: doc });
});

I think the only need here is that RxDB would /push docs out in the same order that they were changed. What do you think, and does the queue even work that way?

[MrChadMWood]

@pubkey Any chance you;d be able to show a minimal example of observing a query containing unreplicated documents on the client side (as in, unreplicated from client to server)? I've made some attempts, but this is a tricky one especially as there is no formal documentation on how to do this.

Attempt 1: I utilized the getLastCheckpointDoc but the 'up' direction continues producing an undefined result.

14:29:33.033 await getLastCheckpointDoc(replicationState.internalReplicationState, 'up')
14:29:33.045 undefined 

To troubleshoot this, I tried making some /push replications and checking only after they'd completed. The result was still undefined.

Attempt 2: I tried synchronizing the schema's local document ( last-push) with the replicationState.sent$.subscribe observable, then I observed changes to that local document while subscribed to a query for documents with a newer updatedAt value. Whenever the local document last-push value changed, I unsubscribed to update the query with the new docs updatedAt and subscribed to this new query. I wasn't able to get consistent results with this approach and haven't been able to diagnose why just yet.

[pubkey]

No sorry, this is not an official feature. You have to dive into the internals. I do not think you can use a normal RxQuery and observe that as these will never show deleted documents in their results.

[MrChadMWood]

No problem, I'll keep looking into it. My use case will never have a deleted document or a new document, just changes to existing docs. Hopefully that'll simplify things a bit in my case.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed soon. Please update it or it may be closed to keep our repository organized. The best way is to add some more information or make a pull request with a test case. Also you might get help in fixing it at the RxDB Community Chat If you know you will continue working on this, just write any message to the issue (like "ping") to remove the stale tag.

[stale[bot]]

Issues are autoclosed after some time. If you still have a problem, make a PR with a test case or to prove that you have tried to fix the problem.