search

remotestorage / remotestorage.js

⬡ JavaScript client library for integrating remoteStorage in apps
https://remotestoragejs.readthedocs.io
MIT License
2.33k stars 141 forks source link

consider merging/converging with SyncIt #787

Closed michielbdejong closed 9 years ago

michielbdejong commented 10 years ago

Looking at https://github.com/forbesmyester/SyncIt it looks like they built exactly what we built with what was introduced as "the sync lib" and then became part of remotestorage.js starting with version 0.6. They will have a different set of supported backends, a different app-facing API.

Probably, in some things, our sync will be better, in others, theirs will be. In any case, it would make sense to compare notes.

Say we add a remoteStorage backend to SyncIt and the result is better than what we have now, then we don't really need to continue developing the sync part of our library separately. We could use SyncIt as our sync engine, which would come with all backends supported, get our discovery from webfinger.js, and then all we need to maintain ourselves is basically the widget (and the modules repo on top of the whole thing, of course).

On the other hand, if we add a remoteStorage backend to SyncIt, and the result is worse than what we have in-house, then we can send them a bunch of pull requests to get them up to date with stuff like multi-thread async sync and IndexedDB commit cache. Then once those land, we could still switch to SyncIt.

Unfortunately, I don't have a lot of time the coming six months, but at least I think it's good if we have a ticket open about this. :)

cc @forbesmyester

silverbucket commented 10 years ago

Hi @michielbdejong and @forbesmyester - I had a look at the code, the repository seems pretty unorganized and scattered, and a codeclimate review rates it at 1.6 (https://codeclimate.com/github/forbesmyester/SyncIt) not that codeclimate is the end-all-be-all, there are some things it does which I tend to ignore, but it's generally a good way to gauge - at least superficially - code complexity.

@forbesmyester - would you be willing to address the hotspot areas of the code and maybe create some basic order in the repository (put things in folders, etc.)?

Also, I was reading over the documentation. After the non-technical overview, It jumps in at the deep end, without covering the basics of how to include it in your code, initialize it, assign it to a data set, etc. So I didn't spend too much time on it at this point.

Does it support AMD, node & general browser use all 100%?

forbesmyester commented 10 years ago

Hi Nick / Etc.

Yeah the docs do go into the deep end a bit... I tried an attempt to make it more accessible at http://forbesmyester.github.io/SyncIt/index.html but that user level of documentation is not my strong point TBH... I much prefer an API doc than a How To myself...

It did generally score badly on Code Climate but noticed it complained a lot about really long functions, which actually include lots of smaller functions within, so I am not sure how valid the score is in this case. Certainly I am willing to look at suggestions of code organisations etc given pointers on what direction to go.

Forbes Lindesay (of Jade fame) has recently started a fork of SyncIt and it might be worth including him in the conversation as he has his own ideas on direction, which, as much as I have heard of them, I agree with. How do we add him to this conversation?

Thanks

Matt On 11 Oct 2014 14:40, "Nick Jennings" notifications@github.com wrote:

Hi @michielbdejong https://github.com/michielbdejong and @forbesmyester https://github.com/forbesmyester - I had a look at the code, the repository seems pretty unorganized and scattered, and a codeclimate review rates it at 1.6 (https://codeclimate.com/github/forbesmyester/SyncIt) not that codeclimate is the end-all-be-all, there are some things it does which I tend to ignore, but it's generally a good way to gauge - at least superficially - code complexity.

@forbesmyester https://github.com/forbesmyester - would you be willing to address the hotspot areas of the code and maybe create some basic order in the repository (put things in folders, etc.)?

Also, I was reading over the documentation. It jumps it at the deep end, without much initial explanation of how to include it in your code, initialize it, assign it to a data set, etc. So I didn't spend too much time on it at this point.

Does it support AMD, node & general browser use all 100%?

Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/787#issuecomment-58749871 .

forbesmyester commented 10 years ago

BTW it's mostly a Browserify project, so can be built to run anywhere.

On 11 October 2014 22:28, Matthew Forrester < matthew.forrester@speechmarks.com> wrote:

Hi Nick / Etc.

Yeah the docs do go into the deep end a bit... I tried an attempt to make it more accessible at http://forbesmyester.github.io/SyncIt/index.html but that user level of documentation is not my strong point TBH... I much prefer an API doc than a How To myself...

It did generally score badly on Code Climate but noticed it complained a lot about really long functions, which actually include lots of smaller functions within, so I am not sure how valid the score is in this case. Certainly I am willing to look at suggestions of code organisations etc given pointers on what direction to go.

Forbes Lindesay (of Jade fame) has recently started a fork of SyncIt and it might be worth including him in the conversation as he has his own ideas on direction, which, as much as I have heard of them, I agree with. How do we add him to this conversation?

Thanks

Matt On 11 Oct 2014 14:40, "Nick Jennings" notifications@github.com wrote:

Hi @michielbdejong https://github.com/michielbdejong and @forbesmyester https://github.com/forbesmyester - I had a look at the code, the repository seems pretty unorganized and scattered, and a codeclimate review rates it at 1.6 (https://codeclimate.com/github/forbesmyester/SyncIt) not that codeclimate is the end-all-be-all, there are some things it does which I tend to ignore, but it's generally a good way to gauge - at least superficially - code complexity.

@forbesmyester https://github.com/forbesmyester - would you be willing to address the hotspot areas of the code and maybe create some basic order in the repository (put things in folders, etc.)?

Also, I was reading over the documentation. It jumps it at the deep end, without much initial explanation of how to include it in your code, initialize it, assign it to a data set, etc. So I didn't spend too much time on it at this point.

Does it support AMD, node & general browser use all 100%?

Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/787#issuecomment-58749871 .

forbesmyester commented 10 years ago

And to use it you'd probably want to look at the SyncItBootstrap project. On 11 Oct 2014 22:35, "Matthew Forrester" matthew.forrester@speechmarks.com wrote:

BTW it's mostly a Browserify project, so can be built to run anywhere.

On 11 October 2014 22:28, Matthew Forrester < matthew.forrester@speechmarks.com> wrote:

Hi Nick / Etc.

Yeah the docs do go into the deep end a bit... I tried an attempt to make it more accessible at http://forbesmyester.github.io/SyncIt/index.html but that user level of documentation is not my strong point TBH... I much prefer an API doc than a How To myself...

It did generally score badly on Code Climate but noticed it complained a lot about really long functions, which actually include lots of smaller functions within, so I am not sure how valid the score is in this case. Certainly I am willing to look at suggestions of code organisations etc given pointers on what direction to go.

Forbes Lindesay (of Jade fame) has recently started a fork of SyncIt and it might be worth including him in the conversation as he has his own ideas on direction, which, as much as I have heard of them, I agree with. How do we add him to this conversation?

Thanks

Matt On 11 Oct 2014 14:40, "Nick Jennings" notifications@github.com wrote:

Hi @michielbdejong https://github.com/michielbdejong and @forbesmyester https://github.com/forbesmyester - I had a look at the code, the repository seems pretty unorganized and scattered, and a codeclimate review rates it at 1.6 ( https://codeclimate.com/github/forbesmyester/SyncIt) not that codeclimate is the end-all-be-all, there are some things it does which I tend to ignore, but it's generally a good way to gauge - at least superficially - code complexity.

@forbesmyester https://github.com/forbesmyester - would you be willing to address the hotspot areas of the code and maybe create some basic order in the repository (put things in folders, etc.)?

Also, I was reading over the documentation. It jumps it at the deep end, without much initial explanation of how to include it in your code, initialize it, assign it to a data set, etc. So I didn't spend too much time on it at this point.

Does it support AMD, node & general browser use all 100%?

Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/787#issuecomment-58749871 .

raucao commented 10 years ago

@silverbucket Yup, have a look at https://github.com/ForbesLindesay/SyncIt/commits/master -- saw lots of clean-up work in there.

michielbdejong commented 10 years ago

Forbes Lindesay (of Jade fame) has recently started a fork of SyncIt and it might be worth including him in the conversation as he has his own ideas on direction, which, as much as I have heard of them, I agree with.

Cool! So let's wait for that, then, and maybe we can use his fork as a dependency in our library.

How do we add him to this conversation?

Like this! :) CC @ForbesLindesay < /hofstadter-moment>

ForbesLindesay commented 10 years ago

Hi,

Here's a quick update on where I'm headed with this:

My first goal with SyncIt was to convert it to a purely promise based API and to make the code as readable as possible (to my eyes). I then want to come up with a simpler idea of what the API looks like that you have to implement in order to be a back end. I want to do this by defining a REST API, so that using the client would look something like:

var sync = require('sync-it');
var collection = sync('http://example.com/my-backend');

As part of this, I want to support some form of partial sync as well as some notion of security rules. I also want a reference backend (built on top of node.js) and would like to provide a hosted service.

The end result of this, I want to release as bicyclejs, which would be an open source organisation that I'd love you all to be a part of :)

ForbesLindesay commented 10 years ago

I essentially want to build an open source version of firebase

forbesmyester commented 10 years ago

Hi Forbes/All,

Looks like we're going the same way. I originally did not want to do Promises for portability reasons but as time/JS progress has gone on that viewpoint has become a bit null and void.

The actual network portion of SyncIt is certainly the weakest part of the project but it would almost certainly be worth reviewing the problems I had getting anything working as well as exposure to new ideas, it's a really tough problem. Have you started on this side of the problem yet?

I still need to go through the remotestorage project to understand what they are doing there and where things may sit together...

I have just changed jobs so I am a bit short on time but SyncIt has been too much time investment for it not to be a major priority wherever the project(s) are going.

Matt On 14 Oct 2014 11:18, "Forbes Lindesay" notifications@github.com wrote:

I essentially want to build an open source version of firebase https://www.firebase.com/

Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/787#issuecomment-59018631 .

michielbdejong commented 9 years ago

Instead of trying to merge remotestorage.js and SyncIt, I will play around with combining localForage + SyncIt + ludbud.

forbesmyester commented 9 years ago

Hi Michiel,

I always intended to do the localForage + SyncIt, I'm actually confident I can do that super quickly. Would that help? I'll have to look into ludbud and what it does...

Thanks

Matt

On 13 March 2015 at 10:11, Michiel de Jong notifications@github.com wrote:

Instead of trying to merge remotestorage.js and SyncIt, I will play around with combining localForage + SyncIt + ludbud.

Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/787#issuecomment-78899586 .

michielbdejong commented 9 years ago

@forbesmyester OK, so this project, remotestorage.js consists more or less of the following parts:

Now, ludbud does just webfinger+oauth+wireclient, in 300 lines of code.

It's a light-weight client for the remoteStorage protocol that does no sync etc., it only does the actual requests to the server. You can see an example on https://ludbud.5apps.com/.

forbesmyester commented 9 years ago

So does localForage + SyncIt + ludbud possibly replace everything except the UI components?

On 13 March 2015 at 10:44, Michiel de Jong notifications@github.com wrote:

@forbesmyester https://github.com/forbesmyester OK, so this project, remotestorage.js consists more or less of the following parts:

  • UI widget
  • helper functions for dealing with domain modules
  • local store (we wrote our own one, but it's equivalent to localForage)
  • sync engine (this part is equivalent to SyncIt, as I understand)
  • it uses webfinger.js for discovery
  • simple OAuth client for obtaining the access token
  • what we call the "wire client" (doing the actual calls to the server's API)
  • experimental wire clients for Dropbox and Google Drive support.

Now, ludbud does just webfinger+oauth+wireclient, in 300 lines of code.

It's a light-weight client for the remoteStorage protocol that does no sync etc., it only does the actual requests to the server. You can see an example on https://ludbud.5apps.com/.

Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/787#issuecomment-78912215 .

michielbdejong commented 9 years ago

Pretty much, yes. UI components and also https://github.com/remotestorage/modules would still be missing. I'll try to build it! :)

forbesmyester commented 9 years ago

Hi All,

I've made progress on this... see the original bug we had ( https://github.com/forbesmyester/SyncIt/issues/5 ) for details. Basically I have the LocalForage version passing mostly all the relevant tests in SyncIt. I'm pretty happy it's fully compatible with the old LocalStorage based on but I'd like to get my TodoMVC app using it just to be sure.

The code is at https://github.com/forbesmyester/SyncIt/tree/localforage

forbesmyester commented 9 years ago

LocalForage branch merged into Master. TodoMVC project hosted at https://github.com/forbesmyester/SyncItBootstrap/tree/todomvc ported to use LocalForage (and it works fine). So SyncIt backed by LocalForage is good to go... see https://github.com/forbesmyester/SyncItBootstrap/blob/master/public/js/SyncItFactory.js#L63 for usage.

michielbdejong commented 9 years ago

\o/ will check it out soon! I've been working on https://github.com/Decentralized-Sharing-Working-Group/server-to-server lately. Announced it on the "unhosted web apps" Google Group though, and tweeted about it! :)