Open the-eclectic-dyslexic opened 5 months ago
I read all of what I wrote again. There were some mistakes. I'll probably continue to update the original comment any time it becomes clear to me that I made a mistake, or something was unclear.
Wow, thanks for this huge amount of information :heart: Definitely take your time for this... I also realized that currently I need to work more slowly on HeliBoard in favor of some other stuff.
no documentation in the code
Even in the java part this is happening too often, and the sometimes very convoluted style also doesn't help figuring out what is going on...
Maintaining that compatibility is important, at least until the open implementation is as good or better than the google library one
Would it be possible to have a separate interface only for your approach? Making sure it's not called (from java side) when the Google library isn't loaded should be simple.
That is a pretty big ordeal though, as I would need a bunch of simulated user data, and I would need to build mock environments around those tests
Maybe we coud just record user data using a special build? Only drawback I see here is that for longer words I may change my swiping a bit when I see the preview words... which would likely differ between libraries.
buckets
In the CSV I see a lot of (Patricia)Trie functions, that's maybe sort of related to dictionary structure, and maybe also with the buckets you're speaking of?
using that other library from wordmage's fork
The issue is some different naming. I remember quite a while ago I compiled wordmage's version with some changes, so I could load the resulting libary like it's done now with the google one. Will dig around in my backups...
The issue is some different naming. I remember quite a while ago I compiled wordmage's version with some changes, so I could load the resulting libary like it's done now with the google one. Will dig around in my backups...
I think when building wordmage's version you need to revert https://github.com/wordmage/openboard/commit/70e59e88ec3d4ac5a50190ceb199765a5ad0078c so the native library refers to com_android_inputmethod
instead of org_dslul_openboard_inputmethod
Wow, thanks for this huge amount of information :heart:
Hopefully some of if it's useful information!
Definitely take your time for this... I also realized that currently I need to work more slowly on HeliBoard in favor of some other stuff.
Definitely do that. Heliboard, at least for me, is in a good place for my daily use. Let it ride for a bit is what I think. You have to have a life outside of open source work, especially if you aren't paid a livable wage for it.
Even in the java part this is happening too often, and the sometimes very convoluted style also doesn't help figuring out what is going on...
Ya, I think this is likely Heliboard biggest hurdle going into the long term. It's hard to contribute to something so opaque. I honestly think the best way for people to contribute right now is to read, document, and in sometimes refactor poorly understood code. That's just my opinion though, maybe others feel differently. I'm hoping to do this for the data structures I need to interact with...
Would it be possible to have a separate interface only for your approach?
Absolutely, the biggest downside is breaking the interface makes it harder for other AOSP forks to share their changes with Heliboard, but it can be done. On the other hand, breaking the interface is likely to be necessary at some point, both for maintainability when removing unused undocumented code, if and when it gets written around or replaced.
Maybe we coud just record user data using a special build? Only drawback I see here is that for longer words I may change my swiping a bit when I see the preview words... which would likely differ between libraries.
We could! I think I still would need to build the mock environments for the tests though, and that seems very daunting to me right now with how little I understand the code base. It would definitely be extremely valuable, but I'm honestly not sure where to start at the moment. I'll ponder this as I read code and try to think of the lowest barrier to entry for this. Building tests, even if only for the code I write, would definitely improve the speed of iteration. It would be so stress free to just have a couple hundred gestures with expected suggestions, and an expected top suggestion, and be able to check how well we are doing without even having to run heliboard. It'll likely involve feeding each test a dictionary, a layout, and a gesture. A single test structure could then be repeated over a massive set of input combinations.
This wouldn't yet cover how to test the natural language model ranks the words the gesture recognition returns though. I'll have to think about how to test that without obfuscating the results the gesture recognition returns. So far the language model has been at the bottom of my list, as I just have assumed we could jerry rig something up to the other suggestion methods in heliboard. I'm thinking there has to be a way to hook up to the next word prediction and feed it a list of possible next words to rank, then create a combined score from the shape & location scores and the next word scores. This would likely be similar to the bigram model used in the paper, but I'm willing to bet the next word prediction in Heliboard goes more than two words deep. I've had it replicate entire second halves of sentences I've said before, just by tapping the middle suggestion.
In the CSV I see a lot of (Patricia)Trie functions, that's maybe sort of related to dictionary structure, and maybe also with the buckets you're speaking of?
They could be, I'll see what I dig up. They might also be where the n-grams are stored for next word predictions. They could also be related to partially typed word completion suggestions.
The issue is some different naming. I remember quite a while ago I compiled wordmage's version with some changes, so I could load the resulting libary like it's done now with the google one. Will dig around in my backups...
Oh! I vaguely remember reading some commits where the JNI code kept having namespaces changed, both in the wordmage fork and in other forks. That's helpful thanks! I forgot about that completely when writing. I'll endeavour to update that later.
Just as an FYI though you all probably know this, FlorisBoard had a very functional gesture typing but took it out to redo with other nlp stuff.
Maybe it would be of help to look at 🤷♂️
Just as an FYI though you all probably know this, FlorisBoard had a very functional gesture typing but took it out to redo with other nlp stuff.
Maybe it would be of help to look at 🤷♂️
Thanks @Charles7z, I'm already in contact with them! Their gesture typing code is also an undocumented mystery, and I'm looking to help them reimplement it to work with their NLP. (Ya, I'm a silly person. I actually approached them before I even discovered heliboard...)
Which reminds me @Helium314, that's one of the things that is simultaneously slowing down my work here, but in the long run should be a good thing. I'm going to be writing the same algorithm twice, and getting the benefit of testing there before I even really start here. I got to dictate the interface there exactly, so it shouldn't be as big an ordeal as this. This means, if we don't find a good way to test here, then at least if I use the same weights we should get the benefits of the testing I'll do in florisboard, if I implement it the same way. We have a solid plan over there, so don't worry about collecting gesture data yet. The maintainer of florisboard has a plan for how I can generate some, at least with a mouse or tablet on desktop.
I'm looking to help them reimplement it to work with their NLP.
@the-eclectic-dyslexic
That's great to hear the code will be used by more than one project.
@the-eclectic-dyslexic
I forgot to mention, though again it's probably something you've already read, a study done to combine T9 prediction or Suretype(T9+) keyboard with gesture typing
Just food for thought, not that you don't have enough, but always good to have ideas beforehand than after.
Swype was doing this as well but apparently shelved the whole keyboard thing.
That's great to hear the code will be used by more than one project.
It won't be the exact same code likely. The data structures are different, and florisboard wants to go rusty. It'll be the same algorithm as far as I can make it though... if we want to make things interesting we could try and add the crate I'll be making for florisboard, as what heliboard's jni code delegates to. That would complicate building heliboard, but it would mean every time one project improves gesture typing both projects get to use it. That would be very cool, but I know nothing about the build headaches that might be involved. It also locks in breaking with the google library. But that might be okay! We can cross these bridges when we get to them. First order of business is for me to actually implement this for florisboard, then we can decide what to do.
but it would mean every time one project improves gesture typing both projects get to use it.
That's certainly a big issue right now with open source keyboards. There's a lot of them but nothing really being pooled together to help the foundation of autocorrect, next word suggestion, prediction, and gesture typing, and therefore the big tech companies keep users tied to their keyboards cause they're the only ones able to put in the man power and pull in the data. So, IMHO, anything that helps at the base level for the whole ecosystem is a big PLUS! Because we all know how open source goes — many forks & many projects.
That would complicate building heliboard
I think it would still be worth trying. But the biggest potential issue I see is that a different dictionary format might be necessary for that glide typing library.
That's certainly a big issue right now with open source keyboards
TBH I wouldn't be surprised if it was an issue with closed source keyboards... As far as I know some code has found the way into multiple AOSP keyboard forks (LineageOS, OpenBoard, Indic Keyboard, maybe more) in the past.
@the-eclectic-dyslexic
just thought i would mention, just in case you didn't notice, that Futo keyboard has glide typing working.
@Charles7z thanks I'll check it out! I haven't had much time to work on this, between life getting hectic and trying to manage a flair up in my chronic pain. I'm happy to have a lead for when I tackle this though!
edit: on a quick glance, it looks like they implemented the holes in the JNI library. That might be pretty easy to port into heliboard if it can be determined that the code actually originated in futo instead of one of the places aforementioned where it wasn't licenced. I will take a look at the code, and compare the interfaces to check. At the absolute minimum, if the futo implementation checks out, it'll create the means to close this issue (though maybe not achieve exactly what I was setting out to do, but that's okay!)
That might be pretty easy to port into heliboard if it can be determined that the code actually originated in futo instead of one of the places aforementioned where it wasn't licenced
I don't think this is correct: the licence basically says you're allowed to look and compile, but I don't think you're allowed to modify it. Probably it's not compatible with GPL 3.0 of this project.
At the absolute minimum, if the futo implementation checks out, it'll create the means to close this issue
It would certainly allow for more flexibility when there is no need to stay compatible to the google library.
Ah yep, I went and assumed before I checked the licence. Thanks!
I haven't read this thread in full, but a fun simple idea I have that could be done in the meantime is "spider typing." Basically, every character key would work as sliding input. Dragging from one to another would type those two characters. You could chain more together by hovering keys for a configurable duration—either tied to long-press timeout or its own config.
This would be nice to have alongside glide typing even, as you'd be able to gesture out things like numbers, spaces, and punctuations with perfect precision. Maybe you could have a functionality to temporarily fall back to spider-typing if you wanted to gesture non-dictionary gibberish or something. Something like a toolbar key.
I'll add that another FOSS keyboard with glide typing (and not just English) is AnySoftKeyboard, which uses the Apache license.
I finally have a little bit of time to write down what I have been doing on this front, so I wanted to open an issue here to keep track of the progress I have made, just in case anyone wants to give me a hand, as my time is super limited at the moment and I blow chunks at juggling tasks. Also, it should mean my work here doesn't go to waste if I fall off the edge of the world or something, haha!
So, first off, I will put a list of things I have yet to figure out, so I can get moving on this. Unfortunately, most of these have nothing to do with the algorithm that needs to be implemented, but rather my unfamiliarity with the Heliboard code base and the JNI overall. If I already knew about these things, I might have already made a working implementation to build on.
Stuff I know I still need to do before I could bring gesture typing to Heliboard without the google library as a hack:
static const JNINativeMethod sMethods[] = { { const_cast<char >("setDicTraverseSessionNative"), const_cast<char >("(Ljava/lang/String;J)J"), reinterpret_cast<void >(latinime_setDicTraverseSession) }, { const_cast<char >("initDicTraverseSessionNative"), const_cast<char >("(JJ[II)V"), reinterpret_cast<void >(latinime_initDicTraverseSession) }, { const_cast<char >("releaseDicTraverseSessionNative"), const_cast<char >("(J)V"), reinterpret_cast<void *>(latinime_releaseDicTraverseSession) } };