Automatic offline machine translation

badonyx commented 1 year ago

user story

As a Damus user who would like to read notes in foreign languages, I would like my local device to translate notes for me automatically, so that I can seamlessly enjoy reaading notes in foreign languages on my device translated to my preferred language.

acceptance criteria

there is a method to enable offline local device auto translation (i.e. no need to tap "translate note")
there is a way to disable
User is warned if there is a need to download offline language packs
data size download MB/GB is displayed
If not on wifi warn user about mobile network bandwidth

Machine translation of notes could be done on-device or otherwise using the language API provided by Apple (which can do offline translations if you download additional data in system settings). ~~Unfortunately Apple does not seem to allow access to their translation API?~~ (Edit: I have seen some activitypub app do this with a button that calls the system translation UI on the note contents, so it seems that is possible at least.)

A nice alternative I found is Google's ML Kit. It is used in another nostr client nostrmo. The translation quality is good and the downloaded models are not very large -- on nostrmo I have loaded 6 languages for translation to English and the app's total documents & data size is still only ~300 MB.

alltheseas commented 1 year ago

Have you tried the current translation options?

cc @tyiu

tyiu commented 10 months ago

I would love to have offline translations and have spent a significant amount of time investigating. However,

Apple has not opened up their translation APIs.
Google's ML Kit for iOS is poorly maintained and does not work for modern development environments. They require Pods but the industry standard is to use Swift Package Manager. It also doesn't work natively for the new Apple M* processors.

I hope things will develop over the next year or so, but nothing is ready at the moment.

jb55 commented 10 months ago

On Thu, Jan 18, 2024 at 04:33:00AM -0800, Terry Yiu wrote:

It also doesn't work natively for the new Apple M* processors.

surprised to hear this! that's kind of a deal breaker isn't it.

jb55 commented 10 months ago

Let's keep this open, maybe a solution will crop up eventually

badonyx commented 10 months ago

some activitypub app do this with a button that calls the system translation UI on the note contents

I have no idea how this was implemented, but it would be a great alternative.

I will try to find the app again if it helps.

badonyx commented 10 months ago

I will try to find the app again if it helps.

It is Radiant.

alltheseas commented 10 months ago

@badstacker is radiant FOSS?

badonyx commented 10 months ago

I do not think it is FOSS.

alltheseas commented 10 months ago

@alexgleason any idea how Radiant mastodon client does iOS translations?

tyiu commented 10 months ago

Whoa, that's definitely Apple native translations. I wonder if they're somehow hacking together some UIKit functionality to get that to work. I'll dig around a bit.

alltheseas commented 10 months ago

@tyiu just found this on iOS. Have you explored?

tyiu commented 10 months ago

@tyiu just found this on iOS. Have you explored?

I've seen that but I don't think that has anything to do with what Radiant does to expose the Translate sheet.

@bryanmontz do you know how Radiant might be accessing Apple's native translate sheet in-app? Is there some UIKit or accessibility magic going on? I don't think there's an API for it at the moment.

bryanmontz commented 10 months ago

No, I'm not sure how they're triggering Apple's translate sheet, if that's what's happening. I googled around for a minute and didn't see a way to do it.

alltheseas commented 10 months ago

Support@radiant.social

Jake is the dev name. Maybe you could ask him @tyiu

badonyx commented 10 months ago

I wonder if they're somehow hacking together some UIKit functionality to get that to work.

@tyiu I think you are on the right track here. Notice in my screenshot those two big dots at the beginning of the text -- those only appear after translation, and they sort of look like a collapsed form of the text selection cursors. Suggests they are doing some trickery to select the text and then trigger the native translation somehow.

tyiu commented 10 months ago

Support@radiant.social

Jake is the dev name. Maybe you could ask him @tyiu

Seems like @jknlsn might be the developer. I've privately messaged him on Mastodon to see if he'd be willing to offer insights.

alltheseas commented 10 months ago

I wonder if they're somehow hacking together some UIKit functionality to get that to work.

@tyiu I think you are on the right track here. Notice in my screenshot those two big dots at the beginning of the text -- those only appear after translation, and they sort of look like a collapsed form of the text selection cursors. Suggests they are doing some trickery to select the text and then trigger the native translation somehow.

Interesting approach!

Here is the manual flow Screenshot 2024-01-18 at 6 36 23 PM