Syncing with server overrides local changes

[intrepidlemon]

If you remove, rename, or edit a project or frame locally in any way after syncing to a server (crick.io), those changes are lost on the next sync.

To reproduce:

# configure server 
watson start discoverDNA
watson stop
watson sync 
watson rename project discoverDNA discover-dna

watson sync

watson projects # discoverDNA (should be discover-dna)

[jmaupetit]

Hi @intrepidlemon! Thanks for the report. A similar issue has been reported for Crick.

We must re-consider our synchronization strategy which —at the time of writing— overrides local changes between two sync events.

PS: that does not matter at all, but there is a typo in your example, the rename command usage should be:

watson rename project discoverDNA discover-dna

[dungerdunger]

I find this is happening with frame editing and deletion as well. Say I pull a frame down from crick during a sync and then edit that frame to change the date. When I sync again, watson again pulls down the initial un-edited frame locally. I then try to delete the re-pulled original frame locally but everytime I sync it pulls it down again. Halp!

[DnOberon]

same here - not sure the best direction in solving it though

[shulard]

Hello,

I tried to update the lastSync parameter to force push all my frames directly from the code 😄. It works because I got the following message :

$ watson sync
Received 0 frames from the server
Pushed 57 frames to the server

But it seems that Crick API does not update mapping with project / tag if the frame already exists.

So I think that this issue is related to the two projects (or I missed something). Adding a watson push command seems relatively easy (maybe with a parameter --last-sync for a bit of control) then we must be sure that the API fully handle the corresponding data.

[shulard]

Hello,

I've checked inside Crick API code and the frames/bulk API endpoint work as append only. I think it's a wanted behaviour but this block updating a previously created frame.

This append in the file : https://github.com/TailorDev/crick/blob/master/api/handlers/frames.go#L68

Maybe a new endpoint must be created ? Maybe the current one must be updated ? It depend on the final API design...

[willdurand]

So, I don't really know how to solve this, because, so far, we prevent data loss by being append only. Of course, if you sync, make changes and then sync again, your changes will be lost.

I think we could introduce a --force option but it might create new issues. The force option would imply a full synchronisation (deleting everything and then re-creating everything), otherwise we cannot detect deleted frames. Ideally, the force option should ask the user for confirmation and should discourage her to use it.

This option might make sense though because our idea was to make Watson the main tool (standalone) and Crick an optional tool that enhances Watson.

[shulard]

Hello, maybe another solution is to add an option in Crick to drop all data, specific frames or a project. I think it's really important for Crick to be append only within the sync workflow.

To be able to implement the --force Crick must be updated to allow removing all / part of the data so I'm not sure that option must belongs to Watson directly.

[acidjunk]

Is there a way to delete stuff in Crick? So I can correct a faulty synced setup?

[shulard]

As far as I know there isn’t a way to remove details from crick. The sync process can only push details.

Le 3 avr. 2019 à 07:43, Rene Dohmen notifications@github.com a écrit :

Is there a way to delete stuff in Crick? So I can correct a faulty synced setup?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[intrepidlemon]

In general, one strategy that I've seen append-only databases use is to capture, "add", "delete" and "modify" actions in an action log. This action log can be thought of as a lower level data structure. Instead, the current append-only design is at too high of a level and doesn't capture intentions or actions. Thus, you can't create a sequence of events that you play back to recreate the database.

As others have said, a refactor is necessary here.