Implement auto synchronization

rynffoll commented 7 years ago

Please, add auto synchronization for remote repositories. Manual synchronization isn't comfy. also sometimes it could be cause of synchronization conflicts.

ghost commented 7 years ago

If you have tasker and a rooted phone, try running this in tasker via a shell command (it'll force a sync): su-c 'am startservice com.orgzly/com.orgzly.android.sync.SyncService' To stop the service (may need to do this before starting again): su-c 'am stopservice com.orgzly/com.orgzly.android.sync.SyncService'

nevenz commented 7 years ago

If you have tasker and a rooted phone, try running this in tasker via a shell command (it'll force a sync): su-c 'am startservice com.orgzly/com.orgzly.android.sync.SyncService'

Service can be made "exported", so it doesn't require root to start, if that would help, short(-ish)-term.

ghost commented 7 years ago

It would be lovely to have this exported (even with auto-sync). Thanks.

anderspapitto commented 7 years ago

I'd like to provide my usecase and desired UX as an example

I run orgzly against a folder in local storage, which (without any orgzly integration) I have synced to my computer with Syncthing. I'm careful to not make conflicting updates - I have one org file which I update from my phone. The rest I update only on my computer, and orgzly is read-only for those files. So, I never get conflicts, but I do have to press the sync button every so often.

Ideally (for me) orgzly would just attempt a sync every so often (say hourly). If it works, great. If it would produce a conflict, skip the sync entirely (i.e. don't make the multiple conflicted versions), and just give a notification that says "automatic sync failed. push the sync button whenever you want and resolve your conflicts manually"

ghost commented 7 years ago

Anderspapitto, I am currently using tasker to do a periodic sync as you suggest and foldersync to handle the remote sync and file conflict resolution. So in at least 1 form, what you are asking for can be done already. :)

nevenz commented 7 years ago

I've also added actions for better control, so what happens doesn't depend on current status:

am startservice -a com.orgzly.intent.action.SYNC_START com.orgzly/com.orgzly.android.sync.SyncService
am startservice -a com.orgzly.intent.action.SYNC_STOP com.orgzly/com.orgzly.android.sync.SyncService

If sync is already running, sending SYNC_START does nothing. If not, sending SYNC_STOP does nothing.

Toggling remains the default if no action is specified or recognized.

ghost commented 7 years ago

This is great! It is very useful to do this without needing root. Anyone having tasker that wants autosync should be very happy. Do you know when this may be released? Not pushing, just curious.

nevenz commented 7 years ago

Do you know when this may be released?

Week or two from now probably.

alphapapa commented 7 years ago

Would it be possible to implement a kind of on-demand auto-sync, so that when a file is loaded or when Orgzly regains focus, it would quickly check to see if the remote copy has been modified and sync that file if necessary?

As it is now, I have several Org files synced to Dropbox, and Orgzly takes a long time to sync all of them. If I have to wait 1-2 minutes for it to sync every file before I can load the one file I need, it means that there are lots of situations in which Orgzly simply isn't useful. But if it could just sync the one file I'm loading, it would be much faster.

Thanks.

ghost commented 7 years ago

I think the golden standard for autosync (for me anyway) is Simplenote. I never have to do anything, and everything stays in sync.

In fact, "getting out of people's way" is something Simplenote does really well. Notice the difference in workflows:

In Orgzly:

Click Notebook
Click Note
Click "Edit Content" *
Type
Click "Close" (w/Save -- the checkmark) *
Click Sync *

In SimpleNote:

Click Note
Type

Big difference!

* Why do I have to do this?

I wish Simplenote and Orgzly would have a baby. It would be the most beautiful child in the world to me, and it's uncle Emacs would definitely support it!

nevenz commented 7 years ago

Click "Edit Content" * * Why do I have to do this?

For the same reason we have "Write" and "Preview" here (GitHub comments editor). 😄

Currently, only links are supported ([[link][description]]), but in the future, we'll have *bold*, /italic/, lists, etc.

"Write" mode is default when you're creating new note, "Preview" when editing the existing one, as a attempt to speedup things.

Suggestions for improving UX welcome. Perhaps the button can be renamed for start?

ghost commented 7 years ago

If we are not using a rooted phone, how can we run these commands?

am startservice -a com.orgzly.intent.action.SYNC_START com.orgzly/com.orgzly.android.sync.SyncService am startservice -a com.orgzly.intent.action.SYNC_STOP com.orgzly/com.orgzly.android.sync.SyncService

licaon-kter commented 7 years ago

Use an automation tool like Tasker, you don't need root for these commands.

Be aware that it might complain about the user this runs on, so make it say: am startservice --user 0 -a ... and the same for stop.

ghost commented 7 years ago

Oh, thanks, @licaon-kter ! The user error was the issue I was having! (I was running this from termux.)

licaon-kter commented 7 years ago

@nevenz After testing that in Termux (start and then stop), I've look at the app to see that it synced, it did, but after exiting the app I got 2 crashes: https://gist.github.com/licaon-kter/346c8f7012dc61b7db30f99401808e4e

I'll try to look at the bigger log soon, maybe see what happened before the actual crash.

nevenz commented 7 years ago

@licaon-kter Thanks, fixed.

timoc commented 7 years ago

I would prefer to offer a different strategy - Git branching. Whenever you make changes, it is committed to an orgzly branch, and visa versa. This implementation allows for more fun things, like having a limited subest of a larger git repo being under orgzly. Whenever there is a conflict, it can automatically create a new branch. Its then up to the repo-owner to tidy everything up.

mkaito commented 7 years ago

If you were to use git as a backend, which is an idea I definitely like, it can probably solve a lot of conflicts itself.

ghost commented 7 years ago

I like the idea of git for merge conflict resolution.

I use termux. I have it running orgzly sync via cron (stop then start). I also have setup inotifyd to watch my repository directory, and when notes change, it runs a script that commits to git, which I also setup in that directory. I am using syncthing to sync the whole directory between my machines. Finally, I am using rclone via cron to 'backup' (end-to-end encrypted) the directory (actually the parent with lots of other non-note files in it) to gdrive, and, using a sneaky trick on my always-on desktop of sharing directories, duplicating that on yandesk.disk and mega.

Sorry. TMI, I guess.

Anyway, I want concurrency, privacy, security, redundancy, integrity, and as much automation as possible in my setup. So, automating sync would still help.

angrybacon commented 7 years ago

This discussion is becoming a little long, I'm not sure someone has mentioned Drive Sync already.

https://play.google.com/store/apps/details?id=com.ttxapps.drivesync

I like the way each folder can have its own strategy. More details here https://metactrl.com/userguide/?app=drivesync#folder-pair (all the options are detailed at the end).

alphapapa commented 7 years ago

This discussion is becoming a little long, I'm not sure someone has mentioned Drive Sync already.

https://play.google.com/store/apps/details?id=com.ttxapps.drivesync

Um, well...:

Contains ads · Offers in-app purchases

D:

angrybacon commented 7 years ago

@alphapapa you can just read the second link, no need to install. :-)

alphapapa commented 7 years ago

Oh, I thought you were suggesting to use it. :)

timoc commented 7 years ago

Back to git - Why? The thing about it is that it can solve the merge problem without requiring that you to change your sync method. It is the git repository that is the important part, because it is designed from the ground up for distributed content management. With stash, branch and other git state storage, it has autosync etc handled, and has had for some time. it should be easy to help users maintain multiple syncable/mergable versions of the same org-document base with some hand holding and documentation. I would also suggest that you do not need to write the code yourself. You can leverage existing tools (via extents?) or use code from open android ports like https://github.com/maks/MGit or https://github.com/rtyley/agit.

alphapapa commented 7 years ago

@timoc Git can sometimes merge automatically, but in the case of conflicts, I can't imagine resolving them on a smartphone UI being practical.

it should be easy to help users maintain multiple syncable/mergable versions of the same org-document base with some hand holding and documentation.

I don't follow you here. I've become familiar with git via the command line and Magit, and I know how to merge branches, resolve conflicts, diff branches, etc--but it was not "easy" or quick to learn, and that was with the power of the CLI and Magit and copious documentation available to me on a desktop PC. Trying to do all that on a tiny touchscreen will surely be frustrating and error-prone, and it will surely by no means be easy to teach people who aren't familiar with git how to do that, especially on a little smartphone.

Have you actually maintained a set of large Org files in a git repo and tried to merge changes made across different systems? If you have, you know that it's easy to lose data by choosing the wrong hunk when resolving conflicts. You have to be slow and deliberate and careful, and you need to be able to see more than a smartphone can display on its small screen.

I just don't see how merging conflicts with git could be feasible here, other than a simple ours/theirs merge strategy choice--but that would not allow users to truly resolve conflicts and would surely result in data loss at times.

ghost commented 7 years ago

@alphapapa

"Have you actually maintained a set of large Org files in a git repo and tried to merge changes made across different systems?"

I realize this was not a question to me, but for me, Yes.

However, if this feature were to be implemented, I don't think it would be mandatory. If you don't like using git or dealing with merge conflicts in it, I'm sure that could be accommodated by either not using git as your repo type, or by disabling the use of git for automatically handling conflicts. Making that an optional preference would be a good thing.

Git's conflict resolution stuff is basically line-by-line, whereas orgzly's is file-by-file. It is nice to have something handle the easy merges for you, so I like the idea of using git for that especially.

I had an Orgzly conflict last week, and it was in different parts of my org file, but orgzly just refused to deal with it (write/read or mine/theirs were the only choices I was given). However, the external version was already in git (as I am using git and a script to do auto-commits for me), so I just forced the write from orgzly, then went to a shell and diffed the versions and manually applied the proper changes in a different editor. Luckily both versions were at that point already in git, so this was not too difficult. However, if I didn't have this ability (because I wasn't using git) how would I have been able to resolve this issue at all?

Ideally, if I were just adding this feature for myself, I wouldn't want to have to deal with merge conflicts in a separate editor, or another program, or by commandline. I wouldn't want to use mgit at all. Git would be built into Orgzly (by integrating the mgit library), and if enabled for a filesystem repository, would automatically "just work" -- no learning curve. However, the real gem for me would be if the git conflicts were left in the file itself, and the file is committed with them marked. The notebook would also be flagged as conflicted in Orgzly so that I would know I need to fix it. So, both versions are merged together in both Orgzly and git, and all I have to do is edit it in Orgzly to "fix" it and let autosync commit it again.

For example, say there is autosync enabled, and say that there is a git library enabled for my git repo. During a sync there is a conflict when git tries to merge it. If orgzly automatically just "committed" the conflict, as marked by git and mark the notebook as conflicted, I could open the notebook and would see it as (ignoring the fact that this is plain text):

...
* Another heading
<<<<<<< HEAD:something.org
 This is my new line
 =======
 This is a new line from the other me
 >>>>>>> 4e2b407f501b68f8588aa645acafffa0224b9b78:something.org
* My last heading
...

I could then edit the file in orgzly to fix it and the next autosync would commit it again.

I don't see why this would be so hard or painful?

(I realize this doesn't deal with push/pull, but in my case, I don't need a remote repo. At any rate, config for and usage of push/pull during autosync could be by preference, too.)

alphapapa commented 7 years ago

I realize this was not a question to me, but for me, Yes.

Thanks, I appreciate your insightful comments.

I don't see why this would be so hard or painful?

You're right that committing the unresolved conflicts would make it possible to resolve them later. But as far as resolving them in Orgzly, it just seems to me that that would be cumbersome. On the other hand, maybe Orgzly could support the syntax and making resolving conflicts as simple as choosing from A or B, rather than having to manually edit all the lines of text.

mkaito commented 7 years ago

Git aside, most of my conflicts would be resolved by a 'sync' button in the incoming note dialog. You know, the one you get when 'sharing' something to orgzly.

Or (better), an option to sync before and after storing a new note.

Right now, when I want to store a note, I have to manually switch to orgzly, sync, switch back to wherever I was (and hope Android did not decide to clear the app's memory in the interim), store the note via 'share', switch back to orgzly again, and sync again.

I guess I've gotten used to fancy cloud apps where you don't even have to think about sync. Things just somehow stay in sync.

alphapapa commented 7 years ago

Right now, when I want to store a note, I have to manually switch to orgzly, sync, switch back to wherever I was (and hope Android did not decide to clear the app's memory in the interim), store the note via 'share', switch back to orgzly again, and sync again.

Yes, this is the crux of the issue.

mkaito commented 7 years ago

Quoting alphapapa (2017-05-11 19:14:09)

Right now, when I want to store a note, I have to manually switch to orgzly, sync, switch back to wherever I was (and hope Android did not decide to clear the app's memory in the interim), store the note via 'share', switch back to orgzly again, and sync again.

Yes, this is the crux of the issue.

Well, as far as I'm concerned, anything that makes the above less painful would be very, very welcome.

If we could reach sync nirvana, and just forget sync even exists, while everything magically appears on all our devices, hey, that'd be effin great. But that's a pretty complicated proposition without a strict protocol and data schema. Org-mode doesn't really fit very well with strict anything.

psamim commented 7 years ago

For me it would be great and enough if Orgzly had such an option as "Automatically sync after a new note is created".

mkaito commented 7 years ago

It is actually often more problematic when you don't sync before adding a note.

El 13 de mayo de 2017 10:29:29 GMT+01:00, Samim Pezeshki notifications@github.com escribió:

For me it would be great and enough if Orgzly had such an option as "Automatically sync after a new note is created".

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

psamim commented 7 years ago

As I said it works only for me because of the way I organize my files. I have one file which I only add to from my phone.

As Emacs and Org-mode are very customizable and configurable people use it in many different ways. So I think we should keep options open in the app too. For me having an option to sync after a note is created works, we can have this and other similar options in the settings.

So in my opinion we should first implement these easy scenarios (forced one way syncs triggered by real events) as an option and think about more complex scenarios, such as interval sync, merging, diffing and git-style conflicts resolution later.

On Sat, May 13, 2017, 4:53 PM Christian Höppner notifications@github.com wrote:

It is actually often more problematic when you don't sync before adding a note.

El 13 de mayo de 2017 10:29:29 GMT+01:00, Samim Pezeshki < notifications@github.com> escribió:

For me it would be great and enough if Orgzly had such an option as "Automatically sync after a new note is created".

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/orgzly/orgzly-android/issues/8#issuecomment-301244903, or mute the thread https://github.com/notifications/unsubscribe-auth/AByDh0d60Hg7xQ1Z6aAGn8zD0EPV_VtAks5r5aCugaJpZM4MEUH6 .

timoc commented 7 years ago

Have you actually maintained a set of large Org files in a git repo and tried to merge changes made across different systems?

@alphapapa yes i have maintained fleets of org files spanning multiple git repositories, and small coteries of Big Fragging Org files of everything, also under git.

Git can sometimes merge automatically, but in the case of conflicts, I can't imagine resolving them on a smartphone UI being practical.

@alphapapa I agree with you here. I find that:

Org files are hard to diff and merge, much like Yaml or XML - a special tool needs to be created
Phone and Table UI's are impractical for this work, even with hand-holding and documentation. Luckily i can recommend another tool to help with the text editing part :), where documentation may be more effective.

The ;TLDR; is, that i stand by the proposal for using git to manage the orgzly org-files. Its a no-brainer IMHO, because it is a robust, cross platform, out of the box document sync tool, with global support and training for those who need it. Git supports syncing by design. It supports branching and merging for managing conflicts and sharing with others.

The longer explanation is:

Git does not enforce sync mechanism, rather Git enforces consistency and integrity, the golden standard of syncing documents. Git can do its own syncing, but you can use Syncthing, rsync, filesync or any other tool to copy your repo around the place. Used properly, git will maintain the integrity of your org document set across multiple devices.

Using a git as a document repo for org-files is not without some workflow difficulties, and possibly a learning curve, but i think the benefits outweigh the problems. The problems outlined in earlier comments are all workflow related, with the presumption that people's workflow includes maintaining their org-documents on an android device. I think this is an edge-case at best, (and if not, then using git is even more important)

Let me explain my workflow, I'm on the GTD bandwagon, so my general workflow is:

Capture -> Process -> Organize -> Clarify/Review -> Do.

The only tool that GTD recommends is some form of list manager. Org is not a list manager.

Org agenda is a list manager. it flattens out information based on queries of your org document-set/knowledge-base/repository into lists/reports based on context, tags etc. These agenda lists are what i use to Do and make proper choices in the moment, and the equivalent of the GTD managed list. These lists are the equivalent of tasks in a todo.txt or similar.

I used to use todo.txt type tools to share my task lists with other applications, so they were useful when i didn't have Emacs available. Especially for the Do phase, where Android apps like that can use todo.txt like clockwork tomato, are fantastic aids to getting stuff done.

Orgzly is for me, a tool at the beginning of the GTD process (Capture) and at the the end (Do). is not a tool for the middle of the process (i.e. Process, Organise, Clarify/Review). That core part is what i use Emacs and org for.

In orgzly i can capture to an org file, and while _Do_ing i have access to the source org-document as support material for doing. It It would be nice to have an automatic feedback loop, but because i know Emacs+git, i am happy with the manual merge/update as it fits into the Process step in my general approach. Automation would only mean that instead of that task needing re-Process, it would come up in Review.

claiire commented 7 years ago

Could we, for the sake of thorough documentation, have a brief guide on how to use the Tasker service for people who might not have used Tasker before?

EDIT: I figured it out. That wasn't so bad actually. Just a bit of a learning curve. For future googlenauts stumbling accross this thread, if you want sync to run every X minutes/hours,

Open up tasker and go to the Profiles tab.
Click the + at the bottom-center of the screen and select Time
Leave From alone. Uncheck To, check Repeat and put in the update interval you'd like.
Now that you've set up your Context for updating, go back to the Profile screen and there should be an Enter task box hovering over your new profile. Hit New Task, give it a name like "Sync Orgzly" or something, and hit the checkmark. This should open the Task Edit screen.
Add a new Action to the Task with the plus button at the bottom of the screen. Select Code > Run Shell on the popup.
In the Command field, write am startservice --user 0 -a com.orgzly.intent.action.SYNC_START com.orgzly/com.orgzly.android.sync.SyncService Tasker should now sync orgzly every x minutes/hours/whatever you used. Probably. I'm bad at this stuff and new to it.

psamim commented 7 years ago

Calling sync service also can be done using Automate, which is an app similar to Tasker.

carllacan commented 7 years ago

Can we just get an option where Orgzly reads and writes the notes to a specific folder in the phone? (Rather than loading/exporting them to a repository, as it does now). That way we could just use Dropbox to sync.

psamim commented 7 years ago

@carllacan You can choose a directory as a repository, then it reads and writes to a normal directory on syncs.

carllacan commented 7 years ago

@psamim Sure, but you still have to manually tell Orgzly to sync to/from that repository. What I am suggesting is that Orgzly directly stores notes in that repository, which is automatically synced by Dropbox (or whatever).

psamim commented 7 years ago

@carllacan What you want is exactly what we are talking about here. Automatic sync means notes are automatically synced with a repo, want it be a simple directory repo or a Dropbox repo. Maybe you are suggesting that if the repo is a simple directory repo, the app should sync on each read and write.

carllacan commented 7 years ago

@psamim Yeah, I'm basically suggesting that there is a sync after each read and write, but since there's a discussion here as to how exactly conflicts should be handled I was suggesting making it so that Orgzly simply reads and overwrites the files in a certain folder and let the user deal with conflicts however they want.

mkaito commented 7 years ago

Quoting Carles (2017-06-13 12:53:57)

Can we just get an option where Orgzly reads and writes the notes to a specific folder in the phone? (Rather than loading/exporting them to a repository, as it does now). That way we could just use Dropbox to sync.

You can sync to a "folder" repository, but sync still has to be manually triggered. That's the whole problem here. I don't want to have to remember to hit the sync button.

khinsen commented 7 years ago

As a new user, having discovered Orgzly recently, I almost immediately ran into the synchronization issue, which is for the most important obstacle to adopting Orgzly as part of my daily workflow.

My point of view is that the idea of "synchronization with a local directory" makes no sense. Why would anyone want to have two copies of the same information on the same device and sync between them?

There are two models of integrating a mobile device into a multi-device data synchronization scheme:

A mobile app manages data internally and provides synchronization with cloud services (WebDav, git, Dropbox, ...).
One mobile app stores data in local files and leaves synchronization with the external world to other mobile apps (FolderSync, MGit, ...).

The first approach is easiest for users if their preferred cloud storage is supported. Orgzly implements it only for Dropbox, which I suppose Dropbox users are happy with (I am not one of them).

The second approach is the most flexible one, but shifts the burden of synchronization to the user. Power users are probably happiest with it, and in particular those using "exotic" cloud storage systems (e.g. git). In the org-mode world, this user profile is probably much more present that among general Android users. I suspect most people contributing to this thread would like to have this approach implemented (I certainly do), and see Orgzly's "synchronization with local files" as a poor implementation (which it is). Making this on-device synchronization automatic is just a complicated way of getting the same result: data storage in the file system.

tulth commented 7 years ago

I am considering adding short-term quality of life sync improvements.

My use case:

file-based syncing
using syncthing
single user, editing my files on different platforms, (phone, tablet, desktop) but never simultaneous editing.
notes files are relatively small

Approach:

for all features, add checkbox defaulted to not enabled in preferences, option to enable
hook into MainActivity::OnResume() to sync all notes on resume
on starting to edit a note, force a sync of that note
big floating sync button in the bottom left similar to the floating plus for adding a new note

tulth commented 7 years ago

i have not tried to do the note specific sync. It might be worth an architecture discussion. I am thinking each note would be more autonomous... They could maintain file mapping and have an async task to sync. Sync all would iterate over notes and tell each to sync. On starting or finishing a note update a specific note could be synced.

Typed on phone...

colonelpanic8 commented 7 years ago

What is the status of this / #24 ? Has anyone made an attempt to get this working?

tulth commented 7 years ago

See pull request #159 for the experimental auto sync

odinu commented 7 years ago

If I am understanding this correctly, using sync or running those sync commands only saves the changes you made in Orgzly, to the repo. Is there any such command that updates Orgzly if the files were modified outside of it ? I sync my orgfiles using git but when changes are made to the files directly I need to force load each file in order to see changes in Orgzly.

tulth commented 7 years ago

it can read in the changes whenever the app is started (or comes forward) if you enable the resume sync.

It does not yes have file observers to catch changes that happen while the app is open

colonelpanic8 commented 7 years ago

It does not yes have file observers to catch changes that happen while the app is open

Why not add some type of reasonable polling mechanism?

orgzly / orgzly-android

Implement auto synchronization #8