Open alexanderadam opened 3 years ago
stale[bot]
commented
3 years ago Hey there, it looks like there has been no activity on this issue recently. Has the issue been fixed, or does it still require the community's attention? This issue may be closed if no further activity occurs. You may comment on the issue and I will leave it open. Thank you for your contributions.
jaanli
commented
3 years ago Agreed, this is a major issue on iOS!
skylize
commented
3 years ago The pain of missing a background sync feature is greatly accentuated by the way in-app sync works:
Joplin fetches 100% of the available updates in a single transaction. If you switch to another app before sync completes, you risk power management shutting down Joplin's network connection(s) and/or Joplin itself, thus breaking the transaction.
Joplin only tells you anything went wrong if the OS decided not to completely kill Joplin between the failure and reopening the app. In the case Joplin alerts you, it does so with a brief red-italic message underneath the Sync/Cancel button. Despite the coloring, this message demands very little attention because it is normal weight, a smaller font than anything else in the Sidebar, and stuck against the bottom edge of the screen. To even have a chance of spotting it, you need to explicitly open the Sidebar, which is hidden by default.
The problem is self-perpetuating: A larger update list means a transaction is more likely to fail to complete during the time Joplin is both open and in focus. That increases the likelihood the next attempt to sync will include yet even more updates appended, and an even greater chance of failure.
I use Joplin on both my phone and laptop several times per day. Yet, as I type this, I am waiting on exactly this kind of gigantic sync on my phone. At some point (I'm guessing just over a week ago?) an initially-slightly-oversized update spiraled out of control into a monster.
After finally working out why a bunch of my notes were missing, I committed to keep the app open until completion (with Stay Awake activated in Developer Options). Unfortunately, after 1.5 hours and 68,000 updates fetched, I had an urgent need to use my phone for another task. When I came back to Joplin, all that progress had been just dumped in the trash 😠. ...Now waiting again; 36k fetched so far; fingers crossed the job will complete this time. 🤞
To attack this problem directly, sync would need to be changed to chunk update lists to a reasonable size; and failure notifications would need to be made more aggressive.
On the other hand, even if you completely put aside the obvious UX improvement that would come from having notes already-in-sync when you open the app, Background Sync would transform this absurd problem of unbounded sync transactions into a total non-issue.
Consistent periodic syncing naturally leads to smaller update sizes than occasional random syncing. But even if you throw some extra-large updates into the mix, finalizing and retrying would both be way less fragile when freed from coupling to UI uptime.
So with Background Sync, this issue simply up-and-disappears.
jaanli
commented
3 years ago Thank you @skylize - you nailed it, much more eloquently than I could have.
This is the only thing keeping me on 1writerapp.com which also syncs markdown files to dropbox, but changes appear within a second (really!) on either device, so I don't have to worry about losing info like @skylize did.
Would be curious to know what the missing pieces are to get this kind of performance, given that 1writerapp is so similar and nails the UX in this respect.
alexanderadam
commented
3 years ago Would be curious to know what the missing pieces
I wanna know this as well. The previous statements claimed that some things are missing but it wasn't explained what the missing things are. It's certainly not the background functionality, since there are libraries for that, which are much older than the comments. Thus it must be something else. 🤷
roman-r-m
commented
3 years ago It's just not been implemented, that's all.
alexanderadam
commented
3 years ago It's just not been implemented, that's all.
But this comment from September 2018 says:
Cannot currently be fixed due a limitation of the underlying framework
And this comment from October 2019 claims:
we rely on existing libs, which for now don’t exist on React Native
jaanli
commented
3 years ago Fantastic, thanks @alexanderadam!
How many hours of dev time would this take, or is this simply infeasible on React Native?
If so, I’d imagine this is a decisive factor of whether or not someone would use/switch to Joplin.
I donated 20 euros recently and happy to donate more.
Would be good to get a sense of how badly people need this feature (I certainly do), not sure how budgets work in open source but we can help fundraising on social! Open to any ideas/advice.
E.g. is this React Native iOS/Android library for background sync incompatible for some reason?
https://github.com/transistorsoft/react-native-background-fetch
alexanderadam
commented
3 years ago E.g. is this React Native iOS/Android library for background sync incompatible for some reason?
https://github.com/transistorsoft/react-native-background-fetch
I don't know but it's the first example library that's mentioned in the first comment.
jaanli
commented
3 years ago Sounds good, curious to know what libraries don’t exist on React Native then!
jaanli
commented
3 years ago The RescueTime.com app uses push notifications to trigger a background sync.
Could that be a quick and dirty workaround while the library issue is sorted out? The push notification could just trigger a background sync with Dropbox it seems. Maybe I’m missing something! Trying to think of quick fixes.
tangodev
commented
3 years ago This is the main reason why I dislike using Joplin on my phone. I even have a separate folder for taking notes on the phone as to avoid conflicts with desktop instead of editing stuff directly.
I'm supporting Joplin financially and I'd like to know when can I expect this to be implemented? I'm not angry, just disappointed.
"Babysitting" a sync is just sad.
jaanli
commented
3 years ago +1, same issue as @tangodev
danepowell
commented
2 years ago Note that someone implemented background sync in https://github.com/laurent22/joplin/pull/4925 but it was not merged due to perceived reliability issues.
roman-r-m
commented
2 years ago Note that someone implemented background sync in #4925 but it was not merged due to perceived reliability issues.
It's wasn't actually completely implemented, see https://github.com/laurent22/joplin/pull/4925#issuecomment-834292363
tangodev
commented
2 years ago As it was mentioned by @alexanderadam in https://github.com/laurent22/joplin/pull/4925 :
Joplin currently has an unreliable sync in general since it's not very obvious that Joplin stops syncing only because the screen goes off.
Every application I have (mail, calendar, chat) synchronizes as long as there's a working network. I assumed for quite a while that Joplin would be no exception. Then I started troubleshooting, since I thought something was wrong with my phone, configuration etc.
See: https://github.com/laurent22/joplin/pull/4925#issuecomment-1027130083
tangodev
commented
2 years ago Hello,
I'd like to ask, if something new (ex. libraries) got developed that could be used in Joplin's code-base to solve the issue?
I'd like to also reiterate what I said (here: https://github.com/laurent22/joplin/pull/4925#issuecomment-1027130083): without any, even an alpha-grade solution, we won't have a production-ready solution. You could just add a feature toggle and display a warning:
THIS IS AN ALPHA/BETA FEATURE! IT MAY EAT YOUR CAT! USE AT YOUR OWN RISK! Got feedback? Tell us HERE (link to forum topic)
Given how many smart and dedicated people there are in the community some of them are bound to latch on to this and generate valuable feedback. This issue's thread in fact is a great starting place to get the ball rolling.
betamaxdroid
commented
1 year ago I too am having this issue. iPhone 11 Pro. Had it with iOS 15, and still having it with 16. Please, please, please fix the sync issue.
tangodev
commented
1 year ago 
The sad and epic story of Joplin's non-sync on mobile
I assume you've found this issue via a search engine or someone linked to it on the forum, GitHub, reddit, etc.
My objective here is to give you a comprehensive one-stop-shop place to read about the issue of Joplin's mobile synchronization and hopefully help push this towards resolution.
Current, future and prospective Joplin users: look no further!
2.10.8, which is the latest (as of the time of writing) version on Google Play.With that out of the way, let us begin!
Before I go further I have to dispel one notion: that the majority of "normal users", a.k.a Aunt Tillies give a single, solitary damn about The Semantic War whether or not Joplin mobile has sync, working sync, background sync, foreground sync, the kitchen sink or whatever.
The issue, at it's technical core is that Joplin mobile can synchronize only when the app is opened and in the foreground. The screen can NOT get locked.
If this happens, the sync stops. So, from a purely technical perspective Joplin mobile has sync, but it's not background sync.
I'm surrendering the notion that in the end of the day this distinction matters at all. I'm drinking Earl Grey with Aunt Tillie and munching on those chocolate mints.
Dear geeks, nerds and techies. If you're tempted to interject with:
...but it does sync, you just need to --
Stop right there. Stop any and all semantic, philosophical or analytical hair splitting. This is not productive. This will get us nowhere.
Less technical users have the right to open, privacy respecting and friendly computing. Software such as Joplin manages the Crown Jewels: the data. People have the right to expect software to "just work".
Imagine purchasing accounting software and then using workarounds to input the bank account numbers correctly. Imagine using an IDE, but the included terminal is broken, so you need a separate window just to issue test/lint/build/deploy commands.
Some things are expected to just work. If your application needs another one just to do its job, you've failed hard. This is the reason I never recommended Joplin to anyone; I'd sound like an idiot saying:
Joplin is an awesome note-taking program, but the mobile sync sucks (and not in the good way), so you need to install an application to keep your screen from locking itself and keep Joplin in the foreground.
The moment you have to say:
[...] but [...]
It's game over.
For all intents and purposes, mobile sync and background sync are implicitly considered the same thing (===) for a majority of users, both techies and non-techies. This works in telegram-foss, proton, k9-mail and countless others... EXCEPT Joplin mobile.
Which brings me to...
How I use Joplin Mobile falls almost exclusively into one of the two categories.
#1: NOT Getting Things Doneinbox notebook on mobile (inbox as in Getting Things Done). I'm away from my laptop or it's powered down and it's faster to add a note on mobile;#2: running errands lateJoplin has, for me, struck 8/10, 9/10 and 10/10 in a bunch of areas such as security/privacy, clean UI, plugin support etc.
But it scores a 1/10 in mobile sync and that drags the entire experience down. The weakest link becomes the most important one.
This makes me pretty frustrated and I'm far from the only one...
The community is not passive about the problem. There were in fact multiple instances of people trying to raise awareness and do something about the problem.
Some of those have been mentioned already by the OP.
| what | when | result / comments |
|---|---|---|
| Background Synchronization/Sync for iOS | 2020-08-10 | No replies |
| Android (background) sync interval | 2019-03-01 | Has some technical details why the issue is a thing |
| Joplin sync on mobile | 2022-11-18 | "sync'ing to mobile was what pushed me away. Totally painful experience." |
| [Rant] I really like the app, but my goodness is the syncing process garbage on mobile... | 2022-10-31 | "This is the main reason I just can't use this app as my primary notes app, no matter how much I like the rest of it." |
| Android not syncing in background | 2018-08-19 | Closed by Laurent less than a month in |
| Android app sync never completes | 2021-12-14 | User applied workaround, issue closed |
| [mobile] Ability to keep the app running in background #1645 | 2019-06-10 | Laurent: "Most likely not possible", closed |
| [Mobile] Joplin synchronisation doesn't run in background | 2020-08-27 | Closed by Caleb; has links to other issues |
| iOS mobile app background sync | 2018-12-06 | Closed on 2020-08-01 as stale by the bot |
This last one is esp. interesting, since it has a bunch of comments that are hidden:
Those users were not impolite or unprofessional in any way, so... Huh.
This GH issue is noteworthy because it has a direct response from Laurent:
Cannot currently be fixed due a limitation of the underlying framework.
For context: that comment is from 2018-09-05 (almost five years ago!), close Joplin's android-v1.0.132 release around the time Android 9 pie got released.
Note: This version of Android is no longer supported. It's direct successor also.
And according to the entry on Google Play Joplin for Android first appeared in mid 2017:
And currently the second review reads as follows:
Latest update for Patrons about mobile states:
Fixed filesystem sync on mobile [...] So you can now sync again your notes with Syncthing and other file-based synchronisation systems.
This is a different problem altogether: it's about accessing the file-system. You still need to have the application running in the foreground.
It boggles my mind that the mobile client got support for speech recognition but not for proper sync.
So now instead of typing my notes and waiting for sync in the foreground I can talk to Joplin, have it write stuff down for me and wait for sync in the foreground. But only in French, a language I don't speak.
Why?
Here we see our MVP alexanderadam pondering for the n-th time why is this feature missing and one of the contributors answers him with:
It's just not been implemented, that's all.
Let's examine this GH Issue and it's discussion:
I see your point but I'm generally reluctant to add half-working features to the app. Even though it's not fully working, it's something we'll have to maintain permanently afterwards. Additionally it means we'll have forever bug reports about sync not always working in the background and we won't be able to fix it.
[...] the mobile app is and has always been very important, [...]. If what we have is unreliable background sync, then I prefer not to add it. Same for the editor, this is top priority for me [...].
Joplin currently has an unreliable sync in general since it's not very obvious that Joplin stops syncing only because the screen goes off. [...] Would you consider this obvious, reliable and a good user experience? 🤔
Repository owner locked as resolved and limited conversation to collaborators on Nov 13, 2022
This is a different battle in The Semantic War. As alexanderadam is trying to explain, all in vain sadly, is that this feature is already broken, so any step will be a step in the right direction.
What's also happening here is a destructive cycle of no-code:
ALSO: this issue (the one we're commenting in right now) is labeled as enhancement.
It's another semantic doge:
It's not a bug! It's a missing feature!
So: @camgraff and @roman-r-m , I respect your efforts and I'm grateful for them. You did what you could.
While I understand the desire of the team to give users a robust experience and the (justified) fear of maintaining something that may turn out to be kludge'y, I can not wrap my head around the resistance towards at least trying.
What is there to lose by having an alpha/beta feature hidden with a toggle so that us, the power users, can test this and generate feedback?
And before anyone says anything about "marring the experience" or "the risk of adding beta features", allow me to quote the Joplin Patreon updates:
To enable the feature, go to the configuration screen and selected "Opt-in to the editor beta". It is already very stable so we will probably promote it to be the main editor from the next version.
Also: Joplin desktop has had a beta toggle since... I can't even remember. And it's for another pretty basic feature, spell check.
In the YEARS that this problem existed, the developers and community could've tested out ALL of the libraries mentioned by alexanderadam and came up with know-how and proposed resolutions.
Heck, in order to preserve the safety and integrity of the user data it would be possible to release a separate version Joplin Mobile - Consume The Cat Edition that would be explicitly for testing sync and only for early adopters/power users.
Meanwhile, new users, confused why mobile sync can't mobile sync are still encouraged to follow The Sacred Path Of The Kludge.
I'm struggling to find an up-to-date road-map for Joplin. The latest one I managed to find is this one, but it's outdated and it points here.
Older ones:
Take a good look at those road-maps. Giving mobile sync the work that it deserves and needs is nowhere to be seen.
Clearly, fixing this has little to no priority.
Note: the lack of a publicly available road-map when there were some previously is troubling.
And now I need to quote myself from almost a year ago, to avoid writing the same thing again, but poorly:
This is extra problematic, since now, Joplin has its 1st party cloud offering, Joplin Cloud. Imagine this sales pitch: "great note-taking program, does not sync with mobile unless you invent crummy workarounds and bend your workflow to them. So it's not a great program". Laurent, I assume, wants Joplin Cloud to be successful, but this is a dealbraker for anyone using mobile any consistent amount. The weakest link (mobile version) ends up pulling down the rest of an otherwise great product.
I feel trapped in some kind of perverse quantum phenomenon where I keep finding the dead bodies of myself trying to solve the enigma of the space-time anomaly that trapped my notes about bananas that I need to buy.
ALSO: any form of Customer Acquisition / User Acquisition is a nightmare even if your product/service is of excellent quality. And Joplin Mobile is... well... it's not.
I simply can't imagine anyone except for the most hardcore of users actually paying for this. Because when you have a 1st party cloud service that no longer requires the Bring Your Own Sync Back-end approach, you're competing 1:1 with Evernote. And people have expectations.
And before anyone says:
But I use only desktop + laptop sync! I don't need mobile sync!
Good for you! Until you'll forget to buy ramen noodles or bananas at the store and you'll say:
If only I had notes on my... Oooooooh!
As someone who's been supporting Joplin's development for years now via Patreon, has contributed time and effort by reporting bugs and now has spent a bunch of hours writing this summary, I'm entitled to answers to the following questions:
Prototype #1 will have experiment #1 in it and will be available on THIS date. The new sync will be hidden behind a toggle."and what were the results?
I believe that other users of Joplin are equally entitled to said answers.
Please respond @laurent22 .
laurent22
commented
1 year ago There's not much to add than what's in other threads. I also wish there was background sync but as of now there's no satisfying solution, I don't know what else there is to say.
alexanderadam
commented
1 year ago I don't know what else there is to say.
Well @laurent22, @tangodev and I were already mentioning that the information in the aforementioned threads were either misleading or even wrong. Other projects are successfully using some of the mentioned libraries, some of them being available since 2016.
Only Joplin doesn't.
And now @tangodev even provided a list with numbered questions. So for that matter it surely would be a kind thing to respond one by one on these questions. 😉
PS: the nice folks at @streetwriters have are currently working on selfhosting on Notesnook. Thus I have hope that this will finally end the whole sync odyssey for me.
laurent22
commented
1 year ago Sorry I don't want to answer that list of questions, but I want to add that one reason we don't add an half-working solution, as was requested, is because we're playing with user data here. We want such a feature to be rock-solid, as close as possible to 100% reliable. Anything less and we risk losing data, creating sync conflicts, etc. All happening invisibly in the background. So the bar to get that in is higher than other features.
And that would be just the beginning. This would be a feature with system integration, which means it will break now and then when Apple or Google decide to deprecate an API, or lock down the device further for "security", or whatever excuse they come up with. So someone will have to maintain and keep this working forever, because once it's there we can't remove it. And if we don't fix it, we can't release new versions.
Not saying it will never happen, but just want to highlight that there's nothing easy about this feature. It's not just about pushing a "merge" button and moving on.
alexanderadam
commented
1 year ago Sorry I don't want to answer that list of questions
Why is that exactly? 🤔
but I want to add that one reason we don't add an half-working solution
The sync in Joplin is currently a not-working-at-all solution. It's surely not even as close as half-working.
as was requested, is because we're playing with user data here.
Well, I had countless sync issues and conflicts because I forgot to babysit my phone on sync. I had to fully resync every now and then, causing other encryption weirdness. Probably because my phone locked the screen and the operating system was killing the sync when I wasn't paying attention.
What I'm saying: previous user data is already at risk and corrupting right now. That's the state of Joplin since it's first release.
Are you really implying that you would rather keep it totally broken than improving one percent?
We want such a feature to be rock-solid, as close as possible to 100% reliable.
Your users want this too. This is the reason why this issue was opened in the first place.
Right now it's definitely not rock-solid but broken.
Anything less and we risk losing data, creating sync conflicts, etc.
What you're describing is the current state.
it will break now and then when Apple or Google decide to deprecate an API, or lock down the device further for "security", or whatever excuse they come up with.
So why are other projects having working solutions then?
I hate to say it but your argumentation still doesn't add up. By avoiding to answer the questions and repeating what was already disproved earlier, you're not supporting your own argumentation at all.
betamaxdroid
commented
1 year ago Jumping in from a user’s perspective, although idk how I got roped into this thread. Evernote, one note, etc are able to do it. Sync. In which ever way they do it.
Mr. Cozic, have you seen Greyhound races and how they get the dogs to run around the track? If you haven’t, they dangle a piece of meat that the dogs can never get to. It’s kind of like that with this sync issue. You are dangling this, but it’s not attainable. We’re getting some nibbles or are licking the meat sometimes, but we cannot get to the whole piece.
My suggestion is that if you cannot afford to pay developers to constantly fix the api’s that apple and google are breaking, then remove this feature entirely, and say ok Joplin has NO mobile sync. You’re being a perfectionist imo, and more than likely said companies will continue to do this indefinitely, for as long as there’s no standardized api. Expect world peace before this happens.
I really like Joplin, but I went back to Evernote because of this sync issue.
On Sun, May 7, 2023 at 11:16 Alexander ADAM @.***> wrote:
Sorry I don't want to answer that list of questions
Why exactly?
but I want to add that one reason we don't add an half-working solution
The sync in Joplin is currently a not-working-at-all solution.
as was requested, is because we're playing with user data here.
Well, I had countless sync issues and conflicts because I forgot to babysit my phone on sync. I had to fully resync every now and then, causing other encryption weirdness. Probably because my phone locked the screen and the operating system was killing the sync.
What I'm saying: previous user data is already at risk and corrupting. That's the state of Joplin since it's first release.
Are you really implying that you would rather keep it totally broken than improving one percent?
We want such a feature to be rock-solid, as close as possible to 100% reliable.
Your users want this too. This is the reason why this issue was opened in the first place.
Right now it's definitely not rock-solid but broken.
Anything less and we risk losing data, creating sync conflicts, etc.
What you're describing is the current state.
it will break now and then when Apple or Google decide to deprecate an API, or lock down the device further for "security", or whatever excuse they come up with.
So why are other projects having working solutions then?
— Reply to this email directly, view it on GitHub https://github.com/laurent22/joplin/issues/3872#issuecomment-1537465567, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIC3G3PN3U4I7MVUHOGFPTXE64EZANCNFSM4SGBJCVA . You are receiving this because you commented.Message ID: @.***>
YoteZip
commented
1 year ago I also ended up switching away from joplin (to standard notes) after subscribing to this thread for a long time. Using joplin was too cumbersome - whenever I wrote out a note while I was on-the-go I had to spend time navigating back to the main panel and tapping sync multiple times before it would finally imply that it was done syncing. If I ever forgot to do this, when I checked the note on my PC, the note from my phone would frequently show up as an empty note, and would never fix itself even if I went back and synced my phone's joplin again. I would have to manually maneuver the note around to fix the problem, either by copying it into a new note or using different software (or just delete the note and forget about it, as I often ended up doing). Eventually after writing too many important notes that I forgot about due to this issue I decided I had to switch to something more reliable.
These claims about user data being paramount over UX are tonedeaf given my experience, and I'm willing to put up with a lot of clunky UX in order to use open source software. There's just no way that regular people would put up with having to manually sync for as long as I did. Burying your head in the sand and saying features take work is killing this project's viability full-stop - nothing else matters if people hate using your software. You can take that advice or you can leave it. From how old this thread is and your persistent response to this vital feature, the only thing left to do is champion other note-taking software that treats UX with something other than apathy.
laurent22
commented
1 year ago Ok let's stop here because nothing new is being said and it's getting heated. We need technical expertise, which takes time and costs money. Rants are cheap, but we get nothing out of them.
There are various issues (1, 2, 3, 4, 5, …) and threads (1, 2, 3) on it but it's still not clear to me what exactly is technically not possible to implement.
This comment from September 2018 says:
Which might be true but that's what libraries are for, right? Therefore this is a very misleading comment.
And this comment from October 2019 claims:
But it doesn't mention which exact libraries would be needed. The logic for syncing is obviously already there. So in theory only the support for doing exactly this sync in a background task would be needed. And there are libraries for react-native that are doing exactly this since a few years. For example
react-native-background-fetchfor Android & iOS (first version from August 2016)react-native-sync-adapterusing the Android SyncAdapter (first version from April 2017)react-native-background-taskagain for Android & iOS (first version from February 2017)react-native-background-jobfor Android (available since December 2016)As you can see, these libraries are far older than these comments. So again this would be wrong and a false statement in case this comment refers to syncing libraries.
Also there's this comment from June 2019
Well,
So also this comment is false / misleading since the already mentioned website also explicitly mentions workarounds.
I would obviously understand if the issue would only be the lack of people working on this. In that case this issue could be kept open for tracking, avoiding duplicates and having the possibility to show this issue for possible contributors, so that they know what's still open.
But currently the given comment are at least somehow lacking some information (or even not relevant / true).
Thus it would be great if someone could clarify what's really missing here.
PS: Thank you for your work!