Translate from Camera/Image

[OkyDooky]

Describe your suggested feature

I'd like to be able to translate text recognized from images, either from gallery or website (maybe via the "Share" funcion) or from a photo taken by the camera. The SimplyTranslate-Mobile app does this using downloadable text recognition libraries. I'm assuming there might be some Tesseract (OCR) integration for this to happen. I don't know how big that would make the app, but if it could be done easily and in a size-efficient way, then I'd love to see this feature (especially since the ST-M dev seems to be having personal issues preventing him from working on the app, which means no updates, and it's the only free/libre app that supports that feature).

Other details

No response

Acknowledgements

• [X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
• [X] I have written a short but informative title.
• [X] I will fill out all of the requested information in this form.

[Bnyro]

It looks like there is a library for Java, however I'll need to test how much it increases the size, in the worst case we might need a build version to make including the library optional.

[OkyDooky]

Ooh, cool. That sounds promising. It might be okay to have yet another build, since there are two flavors, already. But that would be nice if it doesn't bloat the app that much. Thanks for looking into it.

[Bnyro]

Just found that library, looks quite promising. No idea of the actual site yet though!

[OkyDooky]

That looks like a decent place to start. Have you looked at SimplyTranslate-mobile's code? I'm not sure what libraries it uses, since it doesn't say on the main page. But it uses a system where you can download the "trained daya files" for each language from within the app, so it doesn't have to ship with all that bloat but can provide a convenient way for users to select and acquire which one(s) they want. That may be the way you might want to go, if this feature works out.

[Bnyro]

SimplyTranslate uses Flutter with Dart while Translate You uses native Android with Kotlin and Jetpack Compose, hence there aren't the same libraries available as those are different languages :)

[Bnyro]

When using the library, each language has to be installed on its own, one language (for example English) already requires about 25MB of storage, hence we can't include any language packages by default.

I've had a look at SimplyTranslate, it uses https://pub.dev/packages/flutter_tesseract_ocr which internally uses the library I mentioned a few comments before as well. What's really interesting is that the library downloads the data files including the needed language specs dynamically from GitHub and hence is able to keep its size smaller.

So in the end we'll need to add an "option" to the app to download the language packages manually as required by the user and then use that within the library.

However, I'm currently a bit too lazy to work on that, I'll keep it in mind for the future and start working on it once I find motivation :)

[OkyDooky]

Yeah, that's what I was thinking of when I mentioned "looking at how they did it." I knew they downloaded the files separately, since I did tgat for the Japanese file, but wasn't sure at all where they were pulling it from and if it would be practical for your project to do something similar (ie, does it use an expensive hosting option or something anyone could use for free).

In their app, you go under Settings -> Text Recognition and then tap the download/save icon next to each language you would like to have trained data files for. So, it's essentially what you were thinking.

Haha, okay, cool. At least we got the basic outline and list of options straightened out beforehand. Looking firward to whatever you do with this app. :)

[OkyDooky]

Just checked it out and I can't get it working. I select an image and it goes back to the main screen and nothing happens. Also, three things:

1. The language list (for trained files) only displays plain abbreviations. They should show the full name, both for practicality and enjoyability.
2. There is no option for me to use my gallery app (Simple Gallery Pro, in this case) to pick files from. When I tap the "image" button in the Translate You app, it only shows the system-included Files manager app and, oddly, Aria2App from what I have installed. I also cannot share images from the gallery app to this one. And, if I do pick the Files app, I cannot then pick any other apps from there (see first pic). This is a problem when picking files/images from both Jami and Twidere X, as well. But other apps, like Element and Bromite, give me a bunch of options in the drawer for the file manager (second pic) and I'm not sure why or what the deal is. QKSMS also gives Simple Gallery Pro as an option for picking images.
3. Are they any plans to have it trigger a camera app?

Thanks for adding these features. Even though it increased the size just a bit ;), this is beciming a really great option for a translator app.

[Bnyro]

1. There's no list of them, they contain more than 100 languages and I'm not really motivated to search the language name for each. So no.
2. Doing it differently would probably require to add storage permissions to the app, hence I don't really want to. Sharing images to the app would be possible though.
3. Yes, that's planned to be added.

However, I wonder why it won't work for you, I don't have any issues with it (except the one I opened), but when using it properly it always should work. You made sure to select a language data file after downloading it by simply clicking it? (There will be a checkmark then). The process is asynchronously, which means that the detected text should appear after some seconds when it finished the process.

[OkyDooky]

There's no list of them

Seriously? That's unfortunate. Would you say it's more difficult or just tedious to do? If it's the latter, I may be motivated to do that, at some point. It definitely would be an upgrade, but it isn't essential for functionality.

Storage permission

That's what would be required to get those other options? Or to directly offer a gallery app as a picker? Hmm.

Sharing images to the app would be possible though.

That would be fine. It would be nice if it were a bit easier to access images from within the app, but this would both be a nice work around and fit the common use case of sharing text or whatever to a translator app, rather than opening it directly, every time.

That's planned

Sweet.

You made sure to select a language data file after downloading it by simply clicking it?

...No? >_>

So, I just tried that and I got the check mark. Good, so far. Tried my image again and waited for about a minute and nothing still. It shouldn't be a hard image to work with. I also cleared the cache and tried again. No dice. Here's what I tested with: It's a screenshot of a random tweet in Japanese (the only one I could find without hash tags or links in it). When I copy the text data and place it in the text box of the app, it translates fine.

[OkyDooky]

If you want me to try anything, let me know.

[Bnyro]

Seriously? That's unfortunate. Would you say it's more difficult or just tedious to do? If it's the latter, I may be motivated to do that, at some point. It definitely would be an upgrade, but it isn't essential for functionality.

The data is taken from https://github.com/tesseract-ocr/tessdata, creating a list of the languages will not be hard but take a while when done manually. That's what would be required to get those other options? Or to directly offer a gallery app as a picker? Hmm.

It's possible differently as well, but it's a bit complicated, not sure whether I want to spend efforts into that. So, I just tried that and I got the check mark. Good, so far. Tried my image again and waited for about a minute and nothing still. It shouldn't be a hard image to work with. I also cleared the cache and tried again. No dice. Here's what I tested with: It's a screenshot of a random tweet in Japanese (the only one I could find without hash tags or links in it). When I copy the text data and place it in the text box of the app, it translates fine.

That's interesting, I'll attempt that later as well, so far I only tried Latin languages which just worked fine.

[Bnyro]

Tested your example, just works fine for me: . You downloaded the "jpn" language data in the settings and selected it?

[Bnyro]

(The translation is only so bad because I used LibreTranslate in the screenshot, other engines provide better results)

[OkyDooky]

Take a while

Okay. I'll look at it and see if I'd be okay with putting in the effort.

not sure whether I want to spend efforts into that.

Well, as I said just above, I can understand that. Lol No worries.

You downloaded the "jpn" language data in the settings and selected it?

Yes. I also downloaded the jpn vert (vertical?) and the eng packages, but did not select that. I don't know if that could be interfering, somehow. I even tried selecting the eng one and using a plain text on white background image to translate to Japanese, but still nothing. Just now, I uninstalled and reinstalled the app on one of my devices, only downloaded the jpn file and tried that image. Couple minutes later: nothing still. Update: I tried it (using the screenshot I sent you) on another device and it worked! The one that worked is a Galaxy Note 3 with LineageOS 14.1 (yes, I know, it's old) or Android 7.1.2, while the one that didn't is stock Android 9 on a Moto G7 Supra (Power rebrand).

[Bnyro]

Oh okay, that's very weird as I'm using an Android 13 device with LineageOS, so Android 7.1.2 and Android 13 work while Android 9 doesn't? That's very strange :/

No idea either what vrt means for the language packages, might be written somewhere in the Tesseract docs. Language packs shouldn't conflict with each other as long as you don't rename them manually (I doubt you did that).

[OkyDooky]

Maybe it also has something to do with it being stock? I assume vrt means vertical, because Japanese can be written like that and the model may need to be trained differently to properly detect it. And, no, I didn't touch any code or strings.