search

kaiserbh / tachiyomi

Free and open source manga reader for Android.
https://tachiyomi.org/
Apache License 2.0
4 stars 3 forks source link

Syncing on triggers like closing/opening the app in addition to syncing by timeout #6

Closed undefiened closed 8 months ago

undefiened commented 1 year ago

Describe your suggested feature

I was thinking whether it would make sense or would it be possible to sync not only on timeout but also on some meaningful triggers, for example:

It can be possible to make a bunch of checkboxes in the settings allowing the user to select what exactly do they want.

I can try to make an attempt at implementing it if it would be helpful, but I am not 100% sure if this is the right time, or whether I should wait until this implementation gets merged into the upstream?

Other details

No response

Acknowledgements

kaiserbh commented 1 year ago

Describe your suggested feature

I was thinking whether it would make sense or would it be possible to sync not only on timeout but also on some meaningful triggers, for example:

* when a chapter is finished,

* when a chapter is closed (i.e., I was reading and pressed "back" to leave reading),

* when the app is closed,

* when the app is opened,

* when the screen is turned off

It can be possible to make a bunch of checkboxes in the settings allowing the user to select what exactly do they want.

I can try to make an attempt at implementing it if it would be helpful, but I am not 100% sure if this is the right time, or whether I should wait until this implementation gets merged into the upstream?

Other details

No response

Acknowledgements

* [x]  I have searched the existing issues and this is a new ticket, **NOT** a duplicate or related to another open or closed issue.

* [x]  I have written a short but informative title.

* [x]  If this is an issue with an extension, I should be opening an issue in the [extensions repository](https://github.com/tachiyomiorg/tachiyomi-extensions/issues/new/choose).

* [x]  I have updated the app to version **[0.14.6](https://github.com/tachiyomiorg/tachiyomi/releases/latest)**.

* [x]  I will fill out all of the requested information in this form.

Thank you for your feature suggestion regarding enhanced sync triggers. It's an interesting idea, one that was also discussed recently on our Discord server. The concept of allowing users to customize their sync triggers—such as syncing when a chapter finishes, the app opens or closes, etc.—could bring a more dynamic and responsive experience to Tachiyomi users.

As for your implementation offer, I appreciate the enthusiasm. However, I'd recommend waiting until our current features are merged upstream. That way we can get feedback and fix any issues that arises from current implementation.

Now, let's address the elephant in the room—API limitations. Google Drive has an API rate limit of 20,000 queries per 100 seconds, both overall and per user. While the "per user" limit seems generous, the overall limit could pose challenges. Given Tachiyomi's substantial user base, estimated to be around 2+ million, we should be mindful of exhausting these limits. Even if Google approves a higher quota, the sheer volume of sync operations could strain the API, especially during peak usage times.

image

Therefore, in my opinion, this feature could be more readily adopted for self-hosted instances where API limitations are less of a concern. For cloud services like Google Drive, a more phased implementation strategy may be necessary. This would involve rate-limiting, monitoring, and possibly even a user opt-in feature to manage API usage efficiently.

undefiened commented 1 year ago

Thank you very much for your response! I didn't know there was a Discord server, I will try to join it so that I don't double the number of conversations. Just to make sure: you are referring to the https://discord.gg/tachiyomi Discord, right?

Oh, wow, I didn't know about the API limitations. As a workaround, such sync features could be enabled only for users who registered their own Google APIs projects (e.g., require user to enter the secrets json from Google API). However, I would guess that Google wouldn't be very happy about such use cases of their free services either 😅

Or, maybe such features can be enabled only for certain sync services like self-hosted version, I don't think there is much difference in terms of implementation.

An additional reason why I am thinking in this direction is because it would allow to avoid spamming notifications of successful syncing. Right now notifications are needed because sync is done on schedule, so you don't know when it happens. With trigger-based sync the user knows exactly when sync happens, so you only need to create notifications when sync was unsuccessful.

Either way, thank you very much for your feedback, I will then simply wait until the current implementation is merged. So far 30 min frequency has been quite sufficient for me.

kaiserbh commented 1 year ago

The discord server I am usually at for the testing and builds I release is at https://discord.gg/C8NdwjBAwt.

No, I totally agree, and regarding the notification, arkon mentioned something similar and definitely it's a bit spammy, should only notify the user when it's unsuccessful.

Also yea I thought about giving the user an option to add their own secret/token from Google Drive. That's what rclone is doing, and it should be fine. Google doesn't mind, pretty sure.

So maybe with this approach, we can check if the user is using their own secret. If so, allow them to enable syncing on event based instead of timer if the selected service is self host also enabled it with the exception gdrive that uses the default secret then we will just disable the setting.

Something like that should work, but the more I think about it, we can just apply the rate limit. In a way, we don't allow users to sync back to back within a minute or 30 seconds? That will give us leeway of 20k requests per 100 sec, but I think if we ask Google nicely to increase the 20k request to maybe 50k, it should be plenty.

There are many approaches to this. Feel free to chime in what you think is best?

Don't worry too much about doubling the number of conversations. In fact, it's better that you mention it in an issue. That way, we have a public record of it, and people can always be directed to it.

P.S: The more I think about it, the 20k requests are per 100 seconds, if rclone which probably have more users assuming most of them uses the default one it should be fine again assuming they haven't requested Google to increase their quota. So we can probably implement it first and see if there is any issue. If so, we can just swap it after.

undefiened commented 1 year ago

I honestly have no idea how to proceed with the API limits problem and how severe it is. However, some of my thoughts: 1) I am not confident that syncing on some triggers should be generating more API calls than syncing on schedule. I am pretty sure that right now most of the time Tachiyomi syncs nothing and I suspect that syncing on only app open/close would generate way fewer API calls than now. I will maybe sometimes try changing the sync strategy in the code from schedule to on app open/close just to look at API calls statistics: I have only one user in Google apps — myself, so it would be pretty easy to compare. 2) One additional argument for trigger-based syncing is that time-based sync will inevitably work inconsistently due to various battery optimizations and app killings that Android phones do. 3) One thing to remember is that in the case of Tachiyomi, every user should be generating at least x2 number of API calls, since sync is pretty useless unless you have at least 2 devices. 4) Maybe it is possible to just generate several such free API projects. For example, one per region (EU, US, ...), and then simply automatically assign users based on geographical position or whatever. Then 20k easily becomes 60-80k. 5) A possible solution for the initial roll-out is to have no predefined secret. Make a field where users must enter secrets, and make a link to the Tachiyomi site with a FAQ entry containing Tachiyomi secrets. If limits are not enough, then just remove the secret from the FAQ and make instructions on how to register your own Google App instead. Plus that would scare many users, so roll-out will be slower.

kaiserbh commented 1 year ago

Hmm yea I am with you with the inconsistent syncing.

Okay, I think we should implement what you mentioned here. With one exception, instead of manually copying and pasting the secret by default have it filled, it's better for user experience. (most people will not be using their own secret)

Change list would be:

I think with this it will also fix the issue with Google Drive requiring login every 7 days it's random but I think it's inconsistent with the scheduled sync if it doesn't run then the token won't be refreshed.

kaiserbh commented 8 months ago

Added in Branch feat/add-sync-triggers-experimental.