AndBible / and-bible

AndBible: Bible Study
https://andbible.org
GNU General Public License v3.0
577 stars 193 forks source link

Synchronize bookmarks and notes to Google Drive #154

Closed tuomas2 closed 1 year ago

tuomas2 commented 5 years ago

(Note: plans here are outdated, implementation more or less ready in #2584)

I'm planning to implement this by writing journal of events in files in sync folder (in Google drive) that can be replayed on synced devices. Periodic cleanup of the folder can be performed and full database dump will be stored.

I.e. sync folder would contain something like this:

- database-20190101010101.db #Full database
- journal-20190101020101.json #Changes since last database dump
- journal-20190201030101.json #Changes since last journal entry
-...
- database-20190201010101.db #Full database will be written at some point as a new starting point, then old journals and database dumps can be removed.
tuomas2 commented 5 years ago

From user:

Missing an account sync ,so you can highlight the verse but it is only for you device, after reinstall, you are loosing data (some manual saving is possible), but not good for user.

MasterOfTheTiger commented 5 years ago

Maybe MEGA or Nextcloud as well?

steelwil commented 5 years ago

Do not forget about reading plan progress.

mattzab commented 5 years ago

An app called Fitnotes syncs to a hidden Google Drive folder in the background approximately one hour after the user has input data. This allows access on multiple devices.

I would specifically recommend against using MEGA simply because of its recent association with the Chinese Government.

tuomas2 commented 5 years ago

Not sure but maybe this can be service transparent (like installing document from zip).

timbze commented 3 years ago

Maybe this idea is overkill, but perhaps this feature could be mostly implemented in a separate Kotlin native library, that way it would be easier to also sync with other sword apps on desktop and IOS.

There would need to be the option to work with different types of data separately, e.g. one app might want to only sync bookmarks.

tuomas2 commented 3 years ago

Developer:

Perhaps easier starting point would be to add uuid's and timestamps to all db rows in all tables and make it first possible to "merge" backup database (instead of replacing one). Then, if same uuid is in both db's, then use more recent one.

agrogers commented 2 years ago

Hi Tuomas

I think this is becoming even more important now that we can use BSA (Bible Study App) for so much.

I think it is worth considering a really simple solution. Just backup the DB to google drive automatically (perhaps when the app loses focus) and check that file when starting BSA gets focus. If the DBs are out of date prompt the user to restore it with some info on when it was last backed up and from what device.

This is how I do it now when switching between devices but it is tedious and I can forget sometimes.

Or if it can't be done manually, add an option to the backup form to allow 'Synching' which basically uploads or downloads based some version or datetime stamp.

tuomas2 commented 2 years ago

simple solution better than no solution at all. this might work in many cases well enough.

agrogers commented 2 years ago

My feeling is:

  1. Implement a manual and easy option ASAP.
  2. Try to automate it a little.
  3. Implement a last change always wins fancier option on a record by record basis.
tuomas2 commented 2 years ago

apparently it is possible to have limited access to google drive like this Selection_018 :

mattzab commented 2 years ago

That's how FitNotes works, as I mentioned above .

There's also an open source project that I use, called rclone, that is designed for cloud storage sync to many (almost 50) different cloud providers. It can easily be configured to access this hidden part of Google drive. It may help get this feature going in the right direction. rclone.org

They have arm binaries, here's a link to an unofficial Android rclone project, https://github.com/x0b/rcx

agrogers commented 2 years ago

Yes, something is better than nothing for sure.

It is one feature that is noticeably absent.

If on opening AndBible it could check if there is a latter DB backup and prompt to restore it automatically that would be enough in most cases.

On Sat, 6 Aug 2022, 11:32 pm mattzab, @.***> wrote:

That's how FitNotes works, as I mentioned above https://github.com/AndBible/and-bible/issues/154#issuecomment-451754613 .

There's also a project that I use, called rclone, that is designed for cloud storage sync to many (almost 50) different cloud providers. It can easily be configured to access this hidden part of Google drive. It may help open the door for better solutions. rclone.org

— Reply to this email directly, view it on GitHub https://github.com/AndBible/and-bible/issues/154#issuecomment-1207247578, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKGTJWUBH5EINWW4LKM3U3VX2LCHANCNFSM4GFOROSQ . You are receiving this because you commented.Message ID: @.***>

tuomas2 commented 1 year ago

Unfortunately all cloud services have their own APIs and no generic approach to support (all) cloud storage services seem to exist. So, I'm planning to implement only Google Drive support.

tuomas2 commented 1 year ago

Hi Tuomas

I think this is becoming even more important now that we can use BSA (Bible Study App) for so much.

I think it is worth considering a really simple solution. Just backup the DB to google drive automatically (perhaps when the app loses focus) and check that file when starting BSA gets focus. If the DBs are out of date prompt the user to restore it with some info on when it was last backed up and from what device.

I have been playing around with this a little in #2584. I made some optimization to the db filesize and gzipped it and my db currently is about 500 kB, gzipped. And I know that some people have much larger databases. This approach that we upload / download whole db when focusing/unfocusing feels quite wasteful for bandwidth. Need to think about different options...

tuomas2 commented 1 year ago

One option would be to use some sort of binary diffing

tuomas2 commented 1 year ago

Do you think there might be some interest for self-hosted solution? It would be possible to implement support for minio (s3 compatible) as an "free alternative". https://github.com/minio/minio I'm just thinking about all the de-googlers out there...

timbze commented 1 year ago

Do you think there might be some interest for self-hosted solution?

How about just a directory? I use Syncthing, or even Nextcloud etc could be used like that. Haven't looked at minio yet

tuomas2 commented 1 year ago

Do you think there might be some interest for self-hosted solution?

How about just a directory? I use Syncthing, or even Nextcloud etc could be used like that. Haven't looked at minio yet

Does not sound a reliable method. Implementation requires that files are uploaded in order, immediately. If AB would write one patch first, then after one minute, another patch, and then Syncthing would do its job in wrong order, that would make things messy.