[BUG] On StackOverflow, spaces are not recognized after shortcut runs

[psads-git]

When writing a question on StackOverflow, after a shortcut is run, the spaces are considered as non-spaces. Consequently, using backspace to delete superfluous spaces causes the deletion of entire words, Mike.

A video demonstrating the issue is included below.

https://github.com/mike-fabian/ibus-typing-booster/assets/75945439/6cf18ead-1ec5-4ee7-ae6a-b77de1e210d3

[mike-fabian]

I cannot reproduce that yet. Do I need to do something special? Can you give more hints how to reproduce this?

Is this in firefox or google-chrome or maybe both?

[psads-git]

Thanks, Mike. The issue does not occur on Chrome: It only occurs on Firefox.

Steps to reproduce:

1. Write several words.
2. Run a shortcut with the following contents:

[`pandas.concat`](https://pandas.pydata.org/docs/reference/api/pandas.concat.html)

3. Press backspace, which delete entire words instead of single spaces.

[mike-fabian]

Your shortcut does neither contain spaces nor newlines, right?

Does it depend on the contents of the shortcut?

The StackOverflow user interface you see looks a t different what I see:

I wonder where that difference comes from...

[mike-fabian]

Here is a video how it looks for me when I type what you describe:

https://github.com/mike-fabian/ibus-typing-booster/assets/2330175/e39e9377-16e8-41ed-b83b-ca2b6b54a1af

[psads-git]

Are you inserting the shortcut with tab and then enter, Mike?

Please, see this new video, which is more detailed.

https://github.com/mike-fabian/ibus-typing-booster/assets/75945439/85959f55-075a-4b52-aae2-717c68b3b401

[mike-fabian]

Are you inserting the shortcut with tab and then enter, Mike?

I inserted using taband then space. Is that an import difference?

[psads-git]

I inserted using taband then space. Is that an import difference?

I was Just speculating that inserting the shortcut otherwise might influence the outcome.

[mike-fabian]

I still cannot reproduce that at all.

Is there anything interesting in the debug log?

~/.local/share/ibus-typing-booster/debug.log

If you set the debug level in the typing booster setup tool to 3 and then do the stuff you need to do to reproduce the problem, does anything interesting appear in the debug log then? You could reproduce the problem and then attach the log here.

Attention: the log shows what you have been typing recently, so don't attach it here if it might contain anything private!. You could delete the log file first, then restart typing booster and just reproduce the problem without typing anything secret, then attach the log file.

[psads-git]

You can find attached my log file, Mike. debug.log

[mike-fabian]

Does the problem also happen if you insert the shortcut using tab space instead of tab Return?

[psads-git]

The problem occurs using both tab+space and tab+return, Mike.

[psads-git]

There is a much shorter path to reproduce the issue, Mike:

1. Write the word saturday+space.
2. Press backspace.
3. The entire word saturday is deleted.

https://github.com/mike-fabian/ibus-typing-booster/assets/75945439/484580b3-9fa3-4629-9144-4414015adbfa

[mike-fabian]

2023-12-16 11:26:05,140 hunspell_table.py line 6290 do_process_key_event DEBUG: KeyEvent object: 'val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True' 2023-12-16 11:26:05,141 hunspell_table.py line 6097 _handle_compose DEBUG: KeyEvent object: 'val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True' 2023-12-16 11:26:05,141 hunspell_table.py line 6100 _handle_compose DEBUG: Ignoring release event. 2023-12-16 11:26:05,141 hunspell_table.py line 5864 _handle_hotkeys DEBUG: KeyEvent object: 'val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True'

2023-12-16 11:26:05,141 hunspell_table.py line 5864 _handle_hotkeys DEBUG: KeyEvent object: 'val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True'

2023-12-16 11:26:05,141 hunspell_table.py line 3110 _maybe_reopen_preedit DEBUG: KeyEvent object: 'val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True' 2023-12-16 11:26:05,141 hunspell_table.py line 3111 _maybe_reopen_preedit DEBUG: self._arrow_keys_reopen_preedit=True 2023-12-16 11:26:05,142 hunspell_table.py line 3199 _maybe_reopen_preedit DEBUG: Old surrounding_text = ["tomorrow, tomorrow, tomorrow pandas.concat ", 112, 112] 2023-12-16 11:26:05,142 hunspell_table.py line 3228 _maybe_reopen_preedit DEBUG: New surrounding_text = ["tomorrow, tomorrow, tomorrow pandas.concat", 111, 111] 2023-12-16 11:26:05,143 hunspell_table.py line 3338 get_context DEBUG: Getting context: surrounding_text = [text = “tomorrow, tomorrow, tomorrow ”, cursor_pos = 29, anchor_pos = 111] 2023-12-16 11:26:05,143 hunspell_table.py line 3354 get_context DEBUG: Found from surrounding text: tokens=['tomorrow', 'tomorrow', 'tomorrow'] 2023-12-16 11:26:05,143 hunspell_table.py line 3366 get_context DEBUG: Updated context from surrounding text=“tomorrow” “tomorrow” “tomorrow” 2023-12-16 11:26:05,143 hunspell_table.py line 859 _insert_string_at_cursor DEBUG: string_to_insert=['[', '', 'p', 'a', 'n', 'd', 'a', 's', '.', 'c', 'o', 'n', 'c', 'a', 't', '', ']', '(', 'h', 't', 't', 'p', 's', ':', '/', '/', 'p', 'a', 'n', 'd', 'a', 's', '.', 'p', 'y', 'd', 'a', 't', 'a', '.', 'o', 'r', 'g', '/', 'd', 'o', 'c', 's', '/', 'r', 'e', 'f', 'e', 'r', 'e', 'n', 'c', 'e', '/', 'a', 'p', 'i', '/', 'p', 'a', 'n', 'd', 'a', 's', '.', 'c', 'o', 'n', 'c', 'a', 't', '.', 'h', 't', 'm', 'l', ')'] 2023-12-16 11:26:05,144 hunspell_table.py line 860 _insert_string_at_cursor DEBUG: self._typed_string=[] 2023-12-16 11:26:05,144 hunspell_table.py line 861 _insert_string_at_cursor DEBUG: self._typed_string_cursor=0 2023-12-16 11:26:05,144 hunspell_table.py line 1532 _update_transliterated_strings DEBUG: self._typed_string=['[', '', 'p', 'a', 'n', 'd', 'a', 's', '.', 'c', 'o', 'n', 'c', 'a', 't', '', ']', '(', 'h', 't', 't', 'p', 's', ':', '/', '/', 'p', 'a', 'n', 'd', 'a', 's', '.', 'p', 'y', 'd', 'a', 't', 'a', '.', 'o', 'r', 'g', '/', 'd', 'o', 'c', 's', '/', 'r', 'e', 'f', 'e', 'r', 'e', 'n', 'c', 'e', '/', 'a', 'p', 'i', '/', 'p', 'a', 'n', 'd', 'a', 's', '.', 'c', 'o', 'n', 'c', 'a', 't', '.', 'h', 't', 'm', 'l', ')'] 2023-12-16 11:26:05,144 hunspell_table.py line 1533 _update_transliterated_strings DEBUG: self._transliterated_strings={'NoIME': 'pandas.concat'} 2023-12-16 11:26:05,144 hunspell_table.py line 2630 _update_ui DEBUG: entering function 2023-12-16 11:26:05,144 hunspell_table.py line 2283 _update_preedit DEBUG: entering function 2023-12-16 11:26:05,144 hunspell_table.py line 2290 _update_preedit DEBUG: _str=“pandas.concat” 2023-12-16 11:26:05,145 hunspell_table.py line 6358 _process_key_event DEBUG: Preedit reopened successfully. 2023-12-16 11:26:05,145 hunspell_table.py line 6009 _return_false INFO: key='val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True' 2023-12-16 11:26:05,146 hunspell_table.py line 6052 _return_false INFO: Forwarding key event 2023-12-16 11:26:05,146 hunspell_table.py line 1115 _update_candidates DEBUG: self._typed_string=['[', '', 'p', 'a', 'n', 'd', 'a', 's', '.', 'c', 'o', 'n', 'c', 'a', 't', '', ']', '(', 'h', 't', 't', 'p', 's', ':', '/', '/', 'p', 'a', 'n', 'd', 'a', 's', '.', 'p', 'y', 'd', 'a', 't', 'a', '.', 'o', 'r', 'g', '/', 'd', 'o', 'c', 's', '/', 'r', 'e', 'f', 'e', 'r', 'e', 'n', 'c', 'e', '/', 'a', 'p', 'i', '/', 'p', 'a', 'n', 'd', 'a', 's', '.', 'c', 'o', 'n', 'c', 'a', 't', '.', 'h', 't', 'm', 'l', ')'] 2023-12-16 11:26:05,148 hunspell_table.py line 2283 _update_preedit DEBUG: entering function 2023-12-16 11:26:05,148 hunspell_table.py line 2290 _update_preedit DEBUG: _str=“pandas.concat” 2023-12-16 11:26:05,150 hunspell_table.py line 7071 do_reset DEBUG: do_reset() self._prev_key='val=65288 code=14 state=0x40000010 name=“BackSpace” unicode=“\x08” msymbol=“\x08” shift=False lock=False control=False super=False hyper=False meta=False mod1=False mod2=True mod3=False mod4=False mod5=False button1=False button2=False button3=False button4=False button5=False release=True modifier=True'

[mike-fabian]

In the last comment I quoted the relevant part from your debug.log.

I marked 3 important lines in bold face.

When the backspace key is released, the preedit to the left of the cursor is reopened. The string [pandas.concat](https://pandas.pydata.org/docs/reference/api/pandas.concat.html) is put back into preedit and then _update_preedit() is called (marked in bold face). Then _update_candiates() is called and after the candidates have been calculated, _update_preedit() is called again.

As we have learned in one of the other bugs you reported recently, calling _update_preedit() twice in quick succession can trigger a call to the callback function do_reset() which we see in the last line in bold in the last comment.

When do_reset() is called, the preedit has already been deleted (not by ibus-typing-booster but by ibus-core. So this call to do_reset() here is a sign that something bad already happened.

I still cannot reproduce this problem, calling _update_preedit() twice in quick succession does not always cause this problem of trigging do_reset() only sometimes. If this happens on a certain system, then it seems to be quite easy to reproduce. Unfortunately I cannot reproduce this on any of my systems.

But as we know that calling _update_preedit() twice is likely to cause this problem on some systems, I am working now on avoiding one of these _update_preedit() calls. When I have done that you can test whether it fixes the problem for you.

[mike-fabian]

I have uploaded ibus-typing-booster-2.24.7 builds to https://copr.fedorainfracloud.org/coprs/mfabian/ibus-typing-booster/builds/

Please test whether the problem is fixed with this update.

[psads-git]

No, Mike, it is not right yet (backspace inserts a new expansion of the shortcut):

https://github.com/mike-fabian/ibus-typing-booster/assets/75945439/9498654d-886a-44f5-a263-48beee3990ba

[mike-fabian]

I cannot reproduce that either.

Actually this does not look like a new expansion of the shortcut being inserted, it looks like the deletion of surrounding text failed.

When backspace is typed and the cursor moves to the right side of an already committed word/token (In this case the token is [pandas.concat](https://pandas.pydata.org/docs/reference/api/pandas.concat.html)), than that token is deleted from the application, the new context without that token is determined, then the token is inserted into the typed string (i.e. it is back to the state of being typed and not yet committed). Then the UI is updated based on the new situation. This is the code doing this:

https://github.com/mike-fabian/ibus-typing-booster/blob/main/engine/hunspell_table.py#L3209

        if key.val in (IBus.KEY_BackSpace, IBus.KEY_Left, IBus.KEY_KP_Left):
            if cursor_pos != cursor_pos_old - 1:
                LOGGER.debug('Cursor has not moved one column left, '
                             'cannot reopen preedit.')
                return False
            pattern = re.compile(r'^($|[\s]+.*)')
            match = pattern.match(text[cursor_pos:])
            if not match:
                if DEBUG_LEVEL > 1:
                    LOGGER.debug(
                        'No whitespace or end of line or buffer '
                        'to the right of cursor.')
                return False
            pattern = re.compile(r'(^|.*[\s]+)(?P<token>[\S]+)$')
            match = pattern.match(text[:cursor_pos])
            if not match:
                if DEBUG_LEVEL > 1:
                    LOGGER.debug('Could not match token left of cursor.')
                return False
            token = match.group('token')
            # Delete the token, get new context and put it into preedit again:
            self.delete_surrounding_text(-len(token), len(token))
            self.get_context()
            self._insert_string_at_cursor(list(token))
            self._update_ui()
            return True

What you are seeing looks like the token was parsed correctly from the buffer but then the

           self.delete_surrounding_text(-len(token), len(token))

failed.

I already know that these things unfortunately depend very much on timing, therefore I introduced some sleeps a long time ago to work around such problems:

https://github.com/mike-fabian/ibus-typing-booster/blob/main/engine/hunspell_table.py#L177

I have now added two such sleeps after the two self.delete_surrounding_text() calls in the function which is reopening preedits.

New builds of ibus-typing-booster-2.24.8 are at the https://copr.fedorainfracloud.org/coprs/mfabian/ibus-typing-booster/builds/

On my system the sleeps make no difference (except for a short delay of course) but I cannot reproduce the problem.

It would be interesting to know whether these sleeps make a difference on your system.

Does ibus-typing-booster-2.24.8 behave differently for you?

https://github.com/mike-fabian/ibus-typing-booster/blob/main/engine/hunspell_table.py#L177

[psads-git]

Thanks, Mike. Unfortunately, that does not fix the problem: By doing the same test I did in my previous video, the result is the same.

[mike-fabian]

Then I am afraid I cannot do anything at the moment.

If I could reproduce the problem on my machine, I might find a trick to influence the timing, but as it happens on your system only I can just guess.

You can avoid that problem of course by disabling the option

☑️ Enable reopening preedits

as this is a failure produced by the attempt to get a string out of the application and back into preedit.

If that works, it is a nice feature, but it is not essential.

In many applications it does not work anyway. You wrote above that the issue does not occur in google-chrome. That is only because google-chrome does not support surrounding text at all. When trying this in google-chrome one sees this message in the log file:

2023-12-19 00:09:04,217 hunspell_table.py line 3174 _maybe_reopen_preedit DEBUG: Surrounding text is not supported. No way to repopen preedit.

[psads-git]

In the attached screenshot, the second tomorrow is inserted when, while backspacing, I stop backspacing.

[psads-git]

Then I am afraid I cannot do anything at the moment.

That is fine, Mike! And thanks for all your effort!

[mike-fabian]

It is true all these problems go away if you uncheck the

☑️ Enable reopening preedits

option, is that correct?

[psads-git]

It is true all these problems go away if you uncheck the

☑️ Enable reopening preedits

option, is that correct?

Yes, that seems to fix the problem, Mike!

[mike-fabian]

It is true all these problems go away if you uncheck the

☑️ Enable reopening preedits

option, is that correct?

Yes, that seems to fix the problem, Mike!

It doesn't really “fix” it, it just stops trying to reopen preedits so it cannot run into that problem.

Reopening preedits means, if you use Backspace or the arrow keys or the delete key and after using such a key, the cursor happens to be at the beginning or the end of a word (here “word” means some text without whitespace) then that word is taken out of the buffer and into preedit again and a lookup table of candidates will be calculated again as if you had just typed that word and not yet committed it. If that works, that is a great feature, but currently it rarely works right.

It used to work well in all almost all Gtk programs (gedit for example).

Then it stopped working in most gtk programs except in firefox. I have no idea why it stopped working there, I didn't change the code in typing-booster at all when this stopped working so it must have been because of changes in ibus and/or these gtk applications.

I usually keep that option on because it still works reasonably well for me in firefox.

I hope I can find out what broke this in the other applications and make it work again as it used to.

[psads-git]

I imagine it is quite challenging to maintain a piece of software that depends on a myriad of third-party software, Mike!

We can survive without reopening preedits! Do not worry, Mike! :-)

[mike-fabian]

Which version of ibus do you have?

[psads-git]

This one, Mike:

$ rpm -qa ibus
ibus-1.5.29~rc2-5.fc39.x86_64
$

[mike-fabian]

I have exactly the same.

[mike-fabian]

In the attached screenshot, the second tomorrow is inserted when, while backspacing, I stop backspacing.

What is in the log file when this happens?

(By the way, the two F39 systems where I am trying to reproduce this, are updated to the lastest versions of all packages including from the testing repositories (sudo dnf --enablerepo=updates-testing update) Is that the case for your machine as well? I just wonder what the difference could be as you can reproduce a problem when preedits are reopened but I cannot).

[psads-git]

The log file is attached: debug.log

I am using Fedora 39 with XFCE. And I do not use testing repositories.

Some ideas offered by chatGPT:

---

The difference in behavior you're experiencing—where you can reproduce a problem with reopening preedits but someone else cannot—could be due to several factors:

1. Different System Configurations: Variations in operating system versions, ibus-typing-booster versions, or other software on your system compared to the other person's system could result in different behaviors.

2. Language or Input Settings: If you're using different languages or have different settings configured in ibus-typing-booster, this might impact how preedits behave.

3. Hardware Differences: Sometimes, issues can be related to specific hardware, although this is less likely with software like ibus-typing-booster.

4. User Interaction Patterns: The way each user interacts with the software—such as typing speed, the particular combinations of keys pressed, or the timing of actions—can sometimes trigger bugs or different behaviors in software.

[mike-fabian]

Can you please try again with the ibus-typing-booster-2.24.9 from https://copr.fedorainfracloud.org/coprs/mfabian/ibus-typing-booster/builds/ ?

I made some small improvements which definitely should fix the original problem reported here, which was that a just reopened preedit sometimes disappears immediately. In the last few days I found a procedure how I could reproduce this about 1 out of 3 times and I think I could really fix that.

But I still could not reproduce the problem you describe in:

https://github.com/mike-fabian/ibus-typing-booster/issues/474#issuecomment-1861837176

I never saw that at all.

So if you can still reproduce this with ibus-typing-booster-2.24.9 I would be very interested in the debug.log as I have added some extra debug output.

[psads-git]

Thanks, Mike, for having been trying to fix the problem!

With the new version, I do not see

#474 (comment)

anymore. However, it is still buggy, as your can see on the attached video. The log file is also attached.

https://github.com/mike-fabian/ibus-typing-booster/assets/75945439/638a1b39-e80a-4842-ad65-0367d9cbfc55

debug.log

[mike-fabian]

To me this looks like you are still seeing the problem in https://github.com/mike-fabian/ibus-typing-booster/issues/474#issuecomment-1861837176

Strange that I cannot reproduce that at all.

[psads-git]

I hope the new log file will offer new clues, Mike!

[mike-fabian]

I hope the new log file will offer new clues, Mike!

Unfortunately I cannot spot anything unusual in the new log file.

[mike-fabian]

I wonder why the UI to type the question in Stackoverflow looks different for you and for me. If you compare

https://github.com/mike-fabian/ibus-typing-booster/issues/474#issuecomment-1855890201 and https://github.com/mike-fabian/ibus-typing-booster/issues/474#issuecomment-1855905194

you see that I don’t have the bar

Links  Images  Styling/Headers  Lists  Blockquotes  Code HTML  Tables More 

seen in your screenshot.

Your screenshot is white on black and mine is black on white, so I changed to the dark theme in my Stackoverflow settings, but this made no difference whatsoever, it didn’t give me that bar and it didn’t help me to reproduce the problem, so I switched back to the light theme.

Is there any other place except Stackoverflow where you can reproduce the problem?

[psads-git]

If I turn on the button indicated in the attached screenshot, the keywords you mention disappear.

Up to now, I have not met with the reported problem anywhere but StackOverflow.

[mike-fabian]

I don’t see that slider button to enable "The Ask Wizard" either.

[psads-git]

OK, Mike. Please, follow the following steps:

1. Login and authenticate on StackOverflow.

2. Go to pandas forum:

https://stackoverflow.com/questions/tagged/pandas

3. Click on the icon Ask Question.

[mike-fabian]

OK, Mike. Please, follow the following steps:

1. Login and authenticate on StackOverflow.

2. Go to `pandas` forum:

https://stackoverflow.com/questions/tagged/pandas

3. Click on the icon `Ask Question`.

No that does not give me the interface you are seeing.

Maybe this "Ask Wizard" is a feature that new users like me cannot disable anymore? I created my Stackoverflow only to investigate this bug here. Probably yours is older. Maybe that new way of asking questions is opt-in for you but apparently I cannot choose the same way to ask questions you can.

Maybe these are different editors which behave differently.

[psads-git]

Yes, my StackOverflow account is years old, Mike.

[mike-fabian]

If we cannot find another place where this problem https://github.com/mike-fabian/ibus-typing-booster/issues/474#issuecomment-1861837176 happens, then it is maybe only a problem in the old editor of StackOverflow. And it looks like I cannot switch back to that editor, I am stuck with using only new "Stacks" Editor (https://stackoverflow.design/product/components/editor/). And in this editor, the problem does not occur anymore so I cannot reproduce it which makes it very hard for me to work on this.

Can you maybe switch to the new editor? Is there an option somewhere in your Stackoverflow account to switch to the new editor? Maybe if you turn on that slider to enable the “Ask Wizard”?

If switching to the new fixes the problem and we cannot reproduce the problem anywhere else, then I should maybe close this issue here and make a new release of ibus-typing-booster with the things we could fix so far.

[psads-git]

I guess I found a way for you to reproduce the issue: Go to

https://meta.stackexchange.com/questions/ask

There is only the traditional "Ask a question" form!

[mike-fabian]

https://meta.stackexchange.com/questions/ask

Great, there I can resproduce it easily. So this (old?) version of the StackOverflow editor really seems to be quite special.

[psads-git]

That is fantastic that you can now reproduce the issue, Mike!

I am not sure that is an old "Ask a question" form: Maybe, over time, that "old" form is offered as default form to experienced users, who do not need to be guided during the procedure of asking a question.

[mike-fabian]

https://copr.fedorainfracloud.org/coprs/mfabian/ibus-typing-booster/builds/ has ibus-typing-booster-2.24.10 builds now. I think they fix the problem in https://github.com/mike-fabian/ibus-typing-booster/issues/474#issuecomment-1861837176 as well. At least they seem to fix it for me.

[psads-git]

Thanks, Mike: It appears to have fixed the issue!

[mike-fabian]

Great! That was quite hard to fix/workaround. Everything related to surrounding text seems to be quite buggy.

I'll make an offiial release tomorrow unless you find problems with ibus-typing-booster-2.24.10.

[psads-git]

I believe it was quite hard, Mike. But it appears to be working fine!

[mike-fabian]

I released https://github.com/mike-fabian/ibus-typing-booster/releases/tag/2.24.10