search

moneymanagerex / android-money-manager-ex

Manage your finances on-the-go, encrypted for security, and sync via your cloud
http://android.moneymanagerex.org/
GNU General Public License v3.0
477 stars 186 forks source link

Sync issues with Syncthing #1571

Closed dl200010 closed 4 months ago

dl200010 commented 7 months ago

Describe the bug MMEX does not sync the local to the remote, even when the remote is a local file. It keeps separate files.

To Reproduce Steps to reproduce the behavior:

  1. Set up Syncthing and MMEX on desktop
  2. Set up Syncthing and MMEX on Android
  3. Make sure MMB is synced between
  4. Open MMEX on Desktop to make sure it can open the same MMB
  5. Open MMEX on Android and point it at the MMB in Syncthing folder (with or without changing anything)
  6. Close MMEX on Android
  7. Open MMEX on Desktop and add a record
  8. Close MMEX on Desktop (might have to wait for Syncthing to sync)
  9. Open MMEX on Android
  10. An error about a conflict between the local and remote file

Expected behavior No sync error and anything added in either will automatically be in both.

Screenshots

Device Information:

Additional context It seems that MMEX is not syncing the MMB from local to remote unless being directly told to do so. Since Syncthing creates a local MMB (To Android) I would expect that MMEX would not create a new "local" file inside Android/data and just uses the local MMB in the Syncthing folder. I would expect the local file inside Android/data for any syncing that doesn't create a local file. MMEX should always use whatever local file it is pointed at instead of creating a "local" and "remote". It is already local, so why is it creating another "local"? Without doing this Syncthing cannot sync anything from Android to Desktop, or when MMEX does sync it causes sync errors on both desktop and android inside of Syncthing.

guanlisheng commented 7 months ago

hi @dl200010 , can you share some screenshots to help understand the mmex setting, incl. database, sync, and etc. you can refer to https://f-droid.org/packages/com.money.manager.ex/

dl200010 commented 7 months ago

hi @dl200010 , can you share some screenshots to help understand the mmex setting, incl. database, sync, and etc. you can refer to https://f-droid.org/packages/com.money.manager.ex/

I can tomorrow, but I quite literally installed the app and added the local MMB. No other settings changed from default.

renato-mmex commented 7 months ago

could synchronization not be simplified by using multithreading? Here are a few hints: https://www.google.com/search?q=sqlite+multithreading

https://copyprogramming.com/howto/what-about-multithreading-in-android-sqlite

https://www.sqlite.org/threadsafe.html https://www2.sqlite.org/cvstrac/wiki?p=MultiThreading https://stackoverflow.com/questions/65570640/multithreaded-reads-and-writes-to-a-single-sqlite-database-using-the-c-api

guanlisheng commented 7 months ago

in Android ecosystem, only app-specific files are called local. Both other folders (even within your phone) and cloud storage are remote, which can't be accessed directly.

please refer to https://developer.android.com/training/data-storage for details

This precondition explains almost everything, incl. copying to local and introducing SAF.

dl200010 commented 7 months ago

in Android ecosystem, only app-specific files are called local. Both other folders (even within your phone) and cloud storage are remote, which can't be accessed directly.

please refer to https://developer.android.com/training/data-storage for details

This precondition explains almost everything, incl. copying to local and introducing SAF.

There are many programs that access the files on the android system that are not inside of Android/data just fine. Neo Backup and KeePassDX are two that are open source that do just this. Neo Backup uses SAF to pick a local folder to place it's files in. I am not sure how KeePassDX does it, but it allows me to use the exact file that is in the Syncthing folder without issue and no issues syncing. Neo Backup is a root app, but it isn't using root to access the folder.

guanlisheng commented 7 months ago

Following your logic, it works for many other mmex Android users... @dl200010, let's be reasonable instead of repeating them from time to time.

again, please share your details. incl. Syncthing setting.

guanlisheng commented 7 months ago

local mmb <-A-> syncthing folder <-B-> what ever other deivces

as you mentioned conflicts in A, hence the sync takes effect somehow and you just need to dive into the details. e.g. freq of A and B

guanlisheng commented 7 months ago

There are many programs that access the files on the android system that are not inside of Android/data just fine.

it is the same way mmex4android does today... backup scenario is simply on the file level while MMEX needs to perform CRUD via SQLite3. great if you have an example

again, you need to share details to move forward.

dl200010 commented 7 months ago

local mmb <-A-> syncthing folder <-B-> what ever other deivces

as you mentioned conflicts in A, hence the sync takes effect somehow and you just need to dive into the details. e.g. freq of A and B

The problem lies in "<-A->". For some reason it will not automatically sync the two local files on Android and won't do it properly, which causes problems in "<-B->".

"<-B->" happens almost instantly. I use Syncthing for my password DB for KeePassDX (android) and KeePassXC (desktop). I can add one password to KPDX and it appears in KPXC in seconds (usually less than a second). Same thing with KPXC to KPDX.

There are many programs that access the files on the android system that are not inside of Android/data just fine.

it is the same way mmex4android does today... backup scenario is simply on the file level while MMEX needs to perform CRUD via SQLite3. great if you have an example

again, you need to share details to move forward.

It is not the same as MMEX. MMEX keeps a file in Android/data, but the other two open source programs I stated do not do that.

I do not know what more info you are looking for. There is a major issue with how MMEX syncs the two databases. That is why I am saying you shouldn't use the two and just directly access the one not in Android/data. It even creates the two files when you create a local MMB. It doesn't make any sense why it is creating two files, and especially in the case where you create a local MMB.

guanlisheng commented 7 months ago

did you read this? just some screenshots to show how you configure mmex and syncthing, https://github.com/moneymanagerex/android-money-manager-ex/issues/1571#issuecomment-1939870113

dl200010 commented 7 months ago

Syncthing is set up to "watch for changes", which means Android will inform Syncthing when a file changes. It does this quite well for KPXC and KPDX. KPXC and KPDX have no sync functionality built in and uses the exact file I tell it to without copying it to Android/data. I do not want to share screen shots of my Syncthing setup for security reasons.

I literally install MMEX on Android, open the MMB in Syncthing folder, and turn on sync, nothing else. MMEX copies the file to Android/data and uses that. MMEX just isn't keeping the files in sync. Even when I manually tell it to sync it is hit or miss if it will actually sync. It has been this way for a very long time.

guanlisheng commented 7 months ago

well, as you don't share anything detailed to help understand how you setup your mmex and Syncthing so far, it's hard for us to guess the details. maybe you can explore and compare it yourself with https://f-droid.org/packages/com.money.manager.ex/

guanlisheng commented 7 months ago

fyi, there is a live post on facebook to collect the usage on different platforms, maybe you can get inspired. https://www.facebook.com/MoneyManagerEx/posts/pfbid0CLkW93bWVFKwtLWX3iLKBZyQVbHGVjaC4hesy4YmT9TV8Kypsg7gJPzBZ9PH7gm2l

dl200010 commented 7 months ago

Since Syncthing works perfectly fine for KeePassDX and KeePassXC, it isn't a Syncthing issue and the settings for Syncthing are sensitive in the security sense.

I have shared all the settings I changed from default inside of MMEX. There isn't anything else to share. I have pinned it down to being an issue in how MMEX deals with the "local" and "remote" files, even though both are 100% local. Other open source apps can access the SD card files directly without having to copy from the SD to android/data.

I guess I will just stick with using the Windows app only, because it seems that the Android one will not be repaired. Like how I have been for years since the Android app broke. I was just hoping that since development has started again this problem would be addressed.

dl200010 commented 7 months ago

Screenshot_20240213-230114.png

guanlisheng commented 7 months ago

what we need is more facts to dig into the issue.

Default is 3 hrs, and I haven't changed that. we might need to update the default interval.

dl200010 commented 7 months ago

The interval is irrelevant. The issue is it just doesn't properly sync. I have hit synchronize in the menu to manually force a sync, but it is hit or miss if it will sync, even when I have changed an entry or added one on either side.

I have created a database in the app, selecting a local storage point. It sets up the two files, needlessly. It always fails to keep the two files in sync, whether the sync is on or not. It isn't that big of a deal if you are only using the app, but once you want to just copy off the local SD card file it will be a hit or miss if you have the entire set of your changes.

You are just asking for more information without taking into account what I am reporting. The app doesn't keep the files in sync. That is a problem in MMEX, not with anything external. I have played around with the settings and nothing makes it sync properly.

guanlisheng commented 7 months ago

I have hit synchronize in the menu to manually force a sync

this is not a force sync, it is just a regular sync (either pull or push to remote).
you would perform a force pull or push via menu -> settings -> synchronization (download and upload in your screenshot).

but it is hit or miss if it will sync

not sure what this means. any message or database file's modification timestamp

guanlisheng commented 7 months ago

The app doesn't keep the files in sync. That is a problem in MMEX, not with anything external. I have played around with the settings and nothing makes it sync properly.

i belive we both are addressing the problem and wanted to understand more how you played. does this make sense?

dl200010 commented 7 months ago

The app doesn't keep the files in sync. That is a problem in MMEX, not with anything external. I have played around with the settings and nothing makes it sync properly.

i belive we both are addressing the problem and wanted to understand more how you played. does this make sense?

Well, settings only have two that would matter. Sync on start and interval. I have tried even a 5 minute interval while waiting 10 minutes after changing something, and it still doesn't. I have turned on Sync on start, but that still fails just as the app opens and not when I click sync in the menu. I have not turned on "Only on WiFi", because all MMEX should be keeping in sync are the local files. Nothing to transmit over the internet directly.

I have hit synchronize in the menu to manually force a sync

this is not a force sync, it is just a regular sync (either pull or push to remote). you would perform a force pull or push via menu -> settings -> synchronization (download and upload in your screenshot).

Yes, it is a regular sync, and that is what I was expecting to happen. I did not mean it was forcing remote to overwrite local, or vice versa. The force overwrites in the screenshot are the syncs that seem to always work. What I meant by "forced" is that it would be me starting the sync and not the app doing it automatically.

but it is hit or miss if it will sync

not sure what this means. any message or database file's modification timestamp

I have changed an entry in my MMB on Android and synced. Sometimes the entry ends up in the desktop app, sometimes it does not, and many times it ends up with a sync error in MMEX.

guanlisheng commented 7 months ago

it is more clear now. with sync on start checked, you can have a retry with app restart, hopefully it could fix the issue once time...

i reproduced this issue and suspect it was due to event abuse during runtime. https://github.com/moneymanagerex/android-money-manager-ex/pull/1575 to keep event bus dedicated.

dl200010 commented 7 months ago

it is more clear now. with sync on start checked, you can have a retry with app restart, hopefully it could fix the issue once time...

i reproduced this issue and suspect it was due to event abuse during runtime. #1575 to keep event bus dedicated.

Restarting the app after it errors that both "local" and "remote" have changed (because it isn't keeping the changes synced between the two) will not fix the error. It just errors out again and again. Only way to fix that error is to force pull from "remote", but a force push to "remote" probably would have the same effect. "Remote" is the authoritative version in my mind with my files. Maybe 1575 will fix my issues. It is possible. I do not see how it would, but I will not rule out the possibility.

guanlisheng commented 7 months ago

just a guess: it was a runtime issue and seems the message queue was abused or broken somehow. then some sync messages were not processed as planned.

hence, sync code has no change in the PR.

dl200010 commented 7 months ago

Once a new stable version with this PR is in F-Droid, then I will run my testing again. If I find the issue has resolved, then I will comment back.

guanlisheng commented 7 months ago

thank you, for both reporting the issue and the back/forth.

f-droid takes time (still 1017) while this change will be released in 1019,

if ok, here you go a manual built apk , as the data is storage outside of application, i think reinstall will not result in data loss.

dl200010 commented 7 months ago

I am willing to wait or download from GitHub releases, but I will not from Google Drive.

I do not have the app installed, because it isn't working. Every time I test things I install it and remove it after. All that to say I won't have an issue with data loss at all.

guanlisheng commented 7 months ago

ok, GitHub release will be soon this week.

guanlisheng commented 7 months ago

here you go https://github.com/moneymanagerex/android-money-manager-ex/releases/tag/2024.02.15.1019

thomasboehm commented 7 months ago

I had the same problem as @dl200010 and today's update from F-Droid seems to have solved it.

guanlisheng commented 7 months ago

thanks for the feedback. Please write sth on mmex4android + syncthing setup practices.

ggraziotti commented 7 months ago

I have 1019 version. I can't sync file, and I can't upload on my onedrive. I have sync on start, but do not sync.

dl200010 commented 7 months ago

here you go https://github.com/moneymanagerex/android-money-manager-ex/releases/tag/2024.02.15.1019

I will get to this, soon. Sorry it has taken so long. I had a family medical emergency happen between the last time we talked and now.

jeepcook commented 6 months ago

For my part I have the remote file field empty and unable to select a file. Screenshot_2024-02-20-18-14-53-566_com money manager ex-edit

guanlisheng commented 6 months ago

hi @jeepcook , the empty remote file might be from older versions.

you can export this to an remote storage. after verified via MMEX4Destkop, you can reopen it with MMEX4Android. main menu -> settings -> database -> export database

refer to https://forum.moneymanagerex.org/viewtopic.php?p=24753&hilit=export#p24753

Hence, focus download/upload, and re open/export functions are provided to solve the conflicts if any.

jeepcook commented 6 months ago

hi @jeepcook , the empty remote file might be from older versions.

you can export this an remote storage. after verified via MMEX4Destkop, you can reopen it with MMEX4Android. main menu -> settings -> database -> export database

refer to https://forum.moneymanagerex.org/viewtopic.php?p=24753&hilit=export#p24753

Hence, focus download/upload, and re open/export functions are provided to solve the conflicts if any.

Hi, Finally solved this issue after repairing my smartphone os system. I did a too large miui debloat. After that I reinstalled ammex and after the selection of the database file the field are now ok.

The synchronisation seems to be ok between Android/data folder and remote file.

guanlisheng commented 6 months ago

hi @dl200010 , is it ok to closing the issue

dl200010 commented 6 months ago

hi @dl200010 , is it ok to closing the issue

Sorry, I will run some tests tomorrow and let you know. I was dealing with life.

jeepcook commented 6 months ago

Hello, For my part, after few days of use, I can confirm the synchronisation is ok. The app used to select the remote file is com.google.android.documentsui. Thanks to the team. ;-)

dl200010 commented 6 months ago

I just installed 2024.02.15.1019 from F-Droid. I turn on sync and sync when opening. At first I had no issues, but now:

  1. Open on Android. Look at transaction. Press sync in menu. Close app.
  2. Give enough time for Syncthing.
  3. Open on Windows. Look at transaction. Close app.
  4. Give enough time for Syncthing.
  5. Open on Android. Get error that both local and remote have changed

I am also seeing this toast every time I open. Screenshot_20240229-104601

guanlisheng commented 6 months ago

thanks, very like https://github.com/moneymanagerex/android-money-manager-ex/issues/1592

any other logs? this might help to address the issue,

dl200010 commented 6 months ago

thanks, very like #1592

any other logs? this might help to address the issue,

Where would I look for said logs? I do have a rooted device, so I can get at anything.

guanlisheng commented 6 months ago

About -> LOGCAT,

maybe you can try the latest apk via https://drive.google.com/file/d/1adbTNIsGBawEukogULl_Mppo2f2jAhCH/view?usp=sharing

guanlisheng commented 4 months ago

closing it and feel free to re-open it.