Add WiFi/Multipeer cloud file system

[drewmccormack]

Should have a local cache on each device. Would locate another device using bonjour, and should go through a pairing procedure. Each device would then store a 4 digit number that is used to handshake.

The sync procedure would simply involve each device sending the files that it currently has. The other device would then send back any new files it has. This sync would not necessarily have to happen at the same time as a ensembles merge. Usually, it would be wise to do the file sync first, and after that trigger a merge.

[cflorion]

Hi ! Ensembles rocks so far thanks !

I'm implementing a local sync via Multipeer Connectivity to synchronize files. The prototype seems to work. But I want to be the closest to realtime possible (I don't want a sync button), so here's the question. I want to send data files generated by Ensembles each time the EventStore writes to disk. So I have 2 questions:

• I searched the code but I can't find how I can be notified when a new file has been written to disk. Is there a callback somewhere ?
• I could observe disk changes but this would be called each time a file is written, which is too often. So, when the EventStore writes to disk, could I be notified by batches instead of each file creation ?

Thanks ! ;)

[drewmccormack]

When you say 'writes to disk', I guess you mean each time it generates an event in its event store. This only happens

1. When it leeches
2. When your main context saves
3. When you call the merge... method

In other words, there shouldn't be any need to monitor files. Just monitor saves. You basically want to fire a merge after a save, correct?

To be sure you are starting the merge after ensembles has stored the data for the save, monitor the CDEMonitoredManagedObjectContextDidSaveNotification notification, not the usual NSManagedObjectContextDidSaveNotification. The problem with the latter is that you don't know the order the observers get it. With the CDE notification, it is definitely posted after ensembles has processed the data. You can just call merge... in the observer method.

[cflorion]

Thanks!

Argh, I think I didn't quite get the philosophy of your framework. What I did for now is:

• I use CDELocalCloudFileSystem as is.
• At each local save, a device send its header to the other devices. What I call a header is basically the list of the filepaths it has.
• "MainStore/events/xxxxxxxx0002707DF9B236_5.cdeevent",
• "MainStore/events/38_1xxxxxxx8AE-A785-48B4N_27.cdeevent"
• I have another object called DirectoryManager that is initialized with the same root directory as CDELocalCloudFileSystem and that take care of returning the list of filepaths it has. It also takes care of returning a zip file containing the files asked by another device. Also, it takes care of importing zip files containing the files sent by other devices.
• I have a MultipeerManager object that deals with communicating with other devices and sending/receiving zip files.

So my code is not interacting with Ensembles at all, but only take care of syncing the actual files. And I think I'm going in the wrong direction. I think I should use something CDEMultipeerCloudFileSystem that would replace my DirectoryManager (and maybe MultipeerManager as well?). But I can't figure out how to respect CDECloudFileSystem protocol with my logic. Indeed, when a device is missing some files (relatively to another device) it receives a zip containing them (a diff). I use Zip so I can keep the files hierarchy (/Mainstore/baselines for i.e) but I'm sure there's a better solution . So I am dealing with group of files and I think CDECloudFileSystem protocol does it file by file.

Am I wrong ?

[drewmccormack]

You are on the right track, but I would do it a bit differently.

If I were you, I would copy CDELocalCloudFileSystem, since it does most of what you need already. Call the new class CDEMultipeerCloudFileSystem.

I would then build the code from the DirectoryManager and MultipeerManager directly into the new class. So you end up with 1 class that does the same stuff as the current local cloud file system, but additionally has methods that can trigger communications/exchange of files with other devices.

The existing methods from CDELocalCloudFileSystem, which are needed to satisfy the CDECloudFileSystem protocol, can probably remain exactly as they are. They just access the local file system, and get files from that.

The multi peer stuff would probably be pretty orthogonal. You could either add a method that can be called to look for peers, and sync up the files, or you could just have that happen automatically when a device is found. (I haven't used multipeer yet, so not sure what the apis are like. I have used bonjour.)

I think sending a zip file is a good solution to transferring files.

The only thing you might find is that files that get deleted come back when you sync up the files, because they exist on the other device. One way around this might be to simply keep a plist that lists the deleted file paths, and keep that up-to-date when files are deleted by Ensembles. But I think you could make it work even without this, it just wouldn't be as efficient (i.e. it would keep files around, and transfer them, after they are no longer needed).

[drewmccormack]

Small point: It would definitely be useful to have your class have a delegate, and to have a delegate method that is called after multipeer file sync occurs. That would be a good time to trigger a merge... operation.

[drewmccormack]

How is this going? Or did you give up? Anything I can help with?

[cflorion]

Sorry, I didn't give up at all, I had other subjects to deal with :) I'll give it another try in the next few days but here is where I am for now:

• I integrated your suggestions (all in CDEMultipeerCloudFileSystem)
• the whole concept of syncing files in working well
• I was actually struggling with one problem: it's seems that once every fews times, the zip file gets corrupted. I mean the files's md5 is different at emission (a device) and reception (another device). My first assumption was that I was trying to send the zip file too soon, before it was fully zipped, but it seems like it's not it. Now I need to check that the problem doesn't come from MultiPeer framework (hope not...). or my implementation of it...

But in the meantime maybe I should commit this first version on Idiomatic ?

[cflorion]

Another point, I am using ZIP since I think it's the easiest solution to encapsulate a directories and files structure. But is there a solution (I tried to avoid using 3rd part libraries, but with ZIp I don't have a choice). I don't see any other alternative, do you ? On my opinion sending one file at a time would be a bad solution since I think that in terms of network performance it's best to send a big files than a lot of files. Still, I would need to find a solution to encapsulate each file to keep the files trees...

[drewmccormack]

I agree, something like zip or gzip is the way to go.

There should be plenty of zip libraries on github. I think this looks promising: https://github.com/AgileBits/objective-zip

Sam Soffes is also good: https://github.com/soffes/ssziparchive

When you are reasonably happy, you can send a pull request, and I'll take a look.

[drewmccormack]

I suppose this effort died out, correct? If you would like me to take a look, and possibly push it through, you could send me the files or issue a pull request.

[barrettj]

Now I need to check that the problem doesn't come from MultiPeer framework (hope not...)

I've experienced multipeer sending incorrect data as well, so it's not just you. I get a file of a different size than the file I was expecting

[drewmccormack]

Yuck. Doesn't sound good.

[cflorion]

Oh well... Thanks for sharing this ! Good news is that I'm not dumb, bad news is that it looks tricky. Unfortunately I haven't had time to spend another minute on this since last time, but I'll be working on this within two weeks I hope. I'll share my code. Let's keep in touch about this ! ;)

[drewmccormack]

This is now available.