Cljs coverge but clj does not

[maacl]

Running the cljs/clj demo everything appeared to work at first but after having let it run overnight now only the cljs instances are converging and not clj which appears to have missed an operation (ie to browser tabs have have counter as 13 while @val-atom produces 11. Adding 2 from a browser leaves the server 2 behind. Please see logs in this gist: https://gist.github.com/maacl/3755445d3ec9563d6ec2d0e2d40d3f5b

[whilo]

Thanks for reporting the issue! I can reproduce it sometimes, but I am not sure yet when it happens. I assume it is in the streaming code, can you verify that

(<?? (head-value server-store
                 eval-fns
                 ;; manually verify metadata presence
(:state (get @(:state server-store) ["mail:eve@replikativ.io" cdvcs-id]))))

produces the right value? This at least verifies that the system is converging correctly and the bug is in stream-into-atom!.

[maacl]

I get the following:

replikativ-demo.core=> (<?? (head-value server-store

_=> eval-fns

              #_=>                  ;; manually verify metadata presence
              #_=> (:state (get @(:state server-store) ["

mail:eve@replikativ.io" cdvcs-id]))))

ExceptionInfo Cannot transact. clojure.core/ex-info (core.clj:4617)

On Fri, Aug 26, 2016 at 9:38 PM Christian Weilbach notifications@github.com wrote:

Thanks for reporting the issue! I can reproduce it sometimes, but I am not sure yet when it happens. I assume it is in the streaming code, can you verify that

(<?? (head-value server-store eval-fns ;; manually verify metadata presence (:state (get @(:state server-store) ["mail:eve@replikativ.io" cdvcs-id]))))

produces the right value? This at least verifies that the system is converging correctly and the bug is in stream-into-atom!.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/replikativ/replikativ-demo/issues/1#issuecomment-242831492, or mute the thread https://github.com/notifications/unsubscribe-auth/AAD6XIk8p_9iF1tfJ0S1SEfcIk_2Fhh1ks5qj0CwgaJpZM4Jst8B .

Martin Clausen Mob. +45 61 55 50 95 https://www.vyte.in/martin.clausen

[whilo]

Hmm, this is unrelated to streaming missing some of the events. I have worked on the codebase a lot over the last week, so I am not sure which version of replikativ you have used. Could you pull the new version of the demo projects and retry? I still miss some streaming events in the backend in some scenario, but I don't get exceptions even in the stress test scenario (last comment block). Thanks again for taking the time to report back. I hope I find time to improve streaming (also for the new CRDTs) in the next two weeks.

[maacl]

I believe I have been using replikativ-0.2.0-20160812.154150-2.jar

I will update the demo repos, try again and report back.

On Sat, Aug 27, 2016 at 6:38 PM Christian Weilbach notifications@github.com wrote:

Hmm, this is unrelated to streaming missing some of the events. I have worked on the codebase a lot over the last week, so I am not sure which version of replikativ you have used. Could you pull the new version of the demo projects and retry? I still miss some streaming events in the backend in some scenario, but I don't get exceptions even in the stress test scenario (last comment block). Thanks again for taking the time to report back. I hope I find time to improve streaming (also for the new CRDTs) in the two weeks.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/replikativ/replikativ-demo/issues/1#issuecomment-242927318, or mute the thread https://github.com/notifications/unsubscribe-auth/AAD6XHYx6MmYsGGXkDpebISl5Ek3P-Ojks5qkGgHgaJpZM4Jst8B .

Martin Clausen

https://www.vyte.in/martin.clausen

[whilo]

Also CDVCS conflicts if you do concurrent transactions, the demo project is only to play around (without automatic merging) and does not reflect the support for normal CRDTs in replikativ 0.2.0 yet. If set or map semantics are ok for your application, have a look at the new short example at the beginning of the replikativ README.md. If you have any particular needs in that direction (like streaming support for these CRDTs), let me know.

[maacl]

I updated both the clj and cljs demos. I can force them out of sync just by submitting changes from one cljs instance (no concurrency) rapidly. Is that suppose to be possible?

On Sat, Aug 27, 2016 at 7:57 PM Christian Weilbach notifications@github.com wrote:

Also CDVCS conflicts if you do concurrent transactions, the demo project is only to play around (without automatic merging) and does not reflect the support for normal CRDTs in replikativ 0.2.0 yet. If set or map semantics are ok for your application, have a look at the new short example at the beginning of the replikativ README.md https://github.com/replikativ/replikativ/blob/master/README.md. If you have any particular needs in that direction (like streaming support for these CRDTs), let me know.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/replikativ/replikativ-demo/issues/1#issuecomment-242931389, or mute the thread https://github.com/notifications/unsubscribe-auth/AAD6XFx1bqFRUIF8hVv2UpMEM59jlpyGks5qkHqGgaJpZM4Jst8B .

Martin Clausen

https://www.vyte.in/martin.clausen

[whilo]

Yes, the different tradeoffs in terms of the CAP theorem are exposed to the programmer. In case of CDVCS you need to do conflict management then (use replikativ.crdt.cdvcs.stage/merge!). You can do so on any replica (difference to git), since the network should converge to the same conflicting state. If you don't need strong sequential semantics (linearizability = strong consistency in fact), you can get away without conflicts with normal CRDTs, e.g. a Grow-Set (grow only) or an OR-Map (observe and remove map).

As long as you are prototyping, I think CDVCS is nice, because it allows you to trace and debug your changes through the explicit linearized history: https://youtu.be/KV8JcVhQHxw?t=3299 (the project is here: https://github.com/replikativ/flechtwerk). Once you need to scale or allow concurrent edits, it makes sense to explore different CRDTs. I explicitly decided so far not to abstract the tradeoffs away, as they are real and otherwise difficult to reason about.

[maacl]

I am not sure I get this. How could a conflict occur if only one instance is sequentially submitting transactions?

[whilo]

I misunderstood you, sorry. I have added a console.info to the cljs project statement and for me it is the original bug in streaming, the real head-value is converging. Can you pull it and verify that this converges for you?

So you just don't see the events properly (which still is very annoying, I need to fix this).

[maacl]

This is getting weirder and weirder.

I now see "Counter 1370" in the browser, "Current value from store (to check against streaming): 1375" in the browser console and @val-atom 1374 in the clj repl.

[whilo]

The val atoms missed events due to a race condition. Important is that the head-values read from the store agree. The streaming is just there to incrementally realize the value in memory without doing the whole history reduction on every update. I fixed the race condition and all val atoms agree with me now on fast writes. I have pushed a new SNAPSHOT.

[maacl]

Concur. The val atoms now agree.

[whilo]

Good :). Ping back if you have more issues or interesting applications.