Using SocialCalc in ZeGenie

[ramezrafla]

I have to say, I am very much impressed with this project.

In short, we have developed the first full solution for the digital classroom (more info here: http://www.zegenie.com). We have already implemented collaborative document editing, and next we need to include spreadsheet and presentation functionalities.

So we were thinking of using SocialCalc (we only need client-side, the server side is handled with pub-sub using Meteor). We are looking for guidance on how to build an Excel-like clone with SocialCalc. Of course, anything we create we'll happily push back to the community.

Can we brainstorm with someone? Even on a consulting basis would be appreciated.

[eddyparkinson]

we only need client-side, the server side is handled with pub-sub

Please give more details, am not sure I understand.

did you look at:

Ethercalc Ethercalc is the google-docs style version http://ethercalc.net/ https://github.com/audreyt/ethercalc

App builder features sheet.cellmaster.com.au/examples

[ramezrafla]

Thanks, @eddyparkinson We only need the client-side, this is why SocialCalc seesm to fit the bill. Will look at the examples link now. Our interest is in interfacing with SocialCalc via API -- is there a doc somewhere?

[ramezrafla]

Forgot to answer the question:

pub-sub (or publication-subscription) is the modern way of pushing ajax data from server to client. The client subscribes to data on the server (which checks for permissions) and sends data via websockets or HTTP REST to the client. We also use server-side methods, which means we call the server with data (say when user updates a cell) to keep the stored version in DB up to date.

I seems the next big thing in data communication is GraphQL (by Facebook), which allows the client to fine tune the data it needs, but we're not there yet :)

[marcelklehr]

Hey @ramezrafla! Are you looking to implement collaborative editing for spreadsheets as well? Ethercalc currently implements this using pub-sub and faces a lot of issues with conflict resolution, the obvious solution being OT or CRDT. I think that's why @eddyparkinson was confused: Collaborative editing using pub-sub only is almost certainly a bad idea, you will need a conflict resolution algorithm.

[ramezrafla]

Got it, thanks @marcelklehr

We are using OT already in our implementation of docs, so have the needed tools to properly diff and merge, including handling intermittent connections (we diff against specified versions). We use server methods to send the OT diffs, and reactive pub-sub to send the diffs to the other clients, which then merges the diffs.

I'll be happy to share code once we implement it with SocialCalc.

We are looking for SocialCalc as the front-end only, hence the need for better understanding and for an API (e.g. to detect changes, alter cells, nothing fancy). But I can see some extra work to be done to prevent diffs of computed cells (e.g. SUM, AVG etc.) -- we should trigger the clients to do it based on underlying data, so need somewhere to tell SocialCalc that a cell changed to recompute if current cell is dependency of other cells.

[ramezrafla]

BTW, if you look into Meteor, you will be able to get EtherCalc completely migrated to NodeJS very quickly and use the latest tools of the JS community. We can help with that if interested (since this is what we are doing anyway).

[marcelklehr]

Ah, ok. Thanks for the clarification :)

Ethercalc is very mature and easy to integrate in that regard: It has a data format for persisting spreadsheets and can serialize commands. ot-socialcalc utilizes this to provide an OT type for socialcalc commands. Sadly, there is very little documentation for socialcalc atm, so you'll have to dig through the codez a little. gulf-editor-socialcalc is my solution to collaborative editing with socialcalc. You may find it useful as inspiration and documentation of useful APIs in that context ;)

EtherCalc is already based on Node.js, afaik (they're running atop of a fork with thread support, though, I think).

[audreyt]

Re node-webworker-threads it is an optional module (same way that Redis is optional) — ethercalc functions just fine using the built-in vm module in Node.js.

[marcelklehr]

Ah, thanks @audreyt for clearing that up.

@ramezrafla that being said. We always appreciate support in our undertakings, so every contribution is welcome and appreciated :)

[ramezrafla]

Excellent! Will start digging into ot-socialcalc and gulf-editor-socialcalc to get the API hammered out. Maybe our first contribution is in the docs. Are you ok if we PR the main repo readme for that?

Second, if we get that far, we need to minimize the number of server calls for images and resources (and allow UI customization in the process). We may use SVG with symbols (take a look at trumbowyg) or just plain CSS.

[audreyt]

Of course - pull requests welcome and thanks in advance for your contribution!

Ramez Rafla notifications@github.com 於 2017年2月12日 00:35 寫道：

Excellent! Will start digging into ot-socialcalc and gulf-editor-socialcalc to get the API hammered out. Maybe our first contribution is in the docs. Are you ok if we PR the main repo readme for that.

Second, if we get that far, we need to minimize the number of server calls for images and resources (and allow UI customization in the process). We may use SVG with symbols (take a look at trumbowyg or just plain CSS).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[marcelklehr]

@ramezrafla just fyi, there's a (not so easy to find, admittedly) demo of my shenanigans with socialcalc: https://beta.hivejs.org/documents/4

[ramezrafla]

Thanks @marcelklehr

So here is where we are. We are done with the UI changes (we started hard-coding Semantic UI into it). Looks much nicer than the outdated icons. We used Icomoon to generate the missing icons, including generating our own symbols from graphics.

If you are interested, we can push the font somewhere.

Second, in terms of OT: following the KISS principle, upon a change in the sheet, we use CreateSheetSave (super fast!) to generate a text representation of sheet data (in fact, as we improve our process, we can pass a range to it so to only generate for the cells that changed). We use JSDiff (the line diff option) to find the difference with the current date and we push that to the server (which in turn pushes to all listeners, who simply call ParseSheetSave to parse single lines that changed). Works really well, and is super fast.

There is room for improvement by optimizing the range further, but we'll get to it later.

I'll go ahead and close this issue, but feel free to comment and I'll respond.

**Thanks for all your help*** and pls advise what I can do to help.

[marcelklehr]

@ramezrafla Is there a way to contact you via email?

[ramezrafla]

I just sent you an email using your obfuscated email on github.

If you don't get it (check junk), http://www.zegenie.com and click on contact -- sorry for not putting my email publicly, get so much spam

[eddyparkinson]

@ramezrafla favour please. Do you plan to test against ethercalc? If no, can you try to make it easy for us to cherry pick the git changes. This will make easy to check compatibility issues.