carnager commented 8 years ago

teapot earnest %i earnest ~ print "Spaß" | xdotool type --clearmodifiers --file -
Spas
teapot earnest %i earnest ~ Spas

carnager@caprica ~ > printf "Spaß" | xdotool type --clearmodifiers --file -
Spa-%

Same thing without xdotool piped to another application:

print "Spaß" | cat    
Spaß

xdotool version:

xdotool -v
xdotool version 3.20150503.1

Earnestly commented 8 years ago

Just to add:

% xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'
àáâãäçèéêëìíîïñòóôõöùúûüÿàáâãäçèéêëìíîïñòóôõöùúûüý

% xdotool type 'øøøø'
oooo

This is likely how gconv works with the locale but I have no idea personally.

carnager commented 8 years ago

seems xdotool reacts differently depending on keymap used.

carnager@caprica: ~ > setxkbmap us
carnager@caprica: ~ > echo "Spaß" | xdotool type --clearmodifiers --file -
Spaß

Earnestly commented 8 years ago

But still fails with àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ as it outputs:

àáâãäçèéêëìíîïñòóôõöùúûüýÿàáâãäçèéêëìíîïñòóôõöùúûüý

Moilleadoir commented 8 years ago

I just noticed this too. It makes xdotool useless since I’m using it to type the name of a directory which of course does not exist in the lower case form.

Is there any way around this?

fpiccinali commented 8 years ago

use : setxkbmap before your call to xdotool

Earnestly commented 8 years ago

That doesn't help.

fidergo-stephane-gourichon commented 7 years ago

Result is different and surprising here on Ubuntu 16.04 AMD64, keyboard layout "fr".

printf "Spaß" | xdotool type --clearmodifiers --file -
Spqs

setxkbmap us
printf "Spaß" | xdotool type --clearmodifiers --file -
Spaß

setxkbmap fr
printf "Spaß" | xdotool type --clearmodifiers --file -
Spaß

So, setxkbmap changes something, even when setting the same keymap (i.e. supposedly no-op) since things start to work ok after it is used. Very strange.

Anachron commented 6 years ago

Just to add to this, even just running setxkbmap before lets xdotool type work correctly.

Earnestly commented 6 years ago

@Anachron No it really doesn't. This has even been suggested in this thread before, see https://github.com/jordansissel/xdotool/issues/97#issuecomment-206951879

The test case of àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ still fails after doing that.

drzraf commented 6 years ago

Confirming that xdotool result depends upon X keyboard layout as set by setxkbmap. In some case, at least, it's an annoying behavior.

fidergo-stephane-gourichon commented 6 years ago

Indeed `setxkbmap` changes something

Just to add to this, even just running setxkbmap before lets xdotool type work correctly.

Indeed.

Tested again, Ubuntu 16.04 here, french keyboard layout set at X level. For the sake of completeness, running XFCE desktop environment where xfce4-keyboard-settings is set to "Use default parameters". AFAIU this means that xfce doesn't try to interfere in any way, it would just be the same with a plain X session.

printf "Spaß" | xdotool type --clearmodifiers --file -
Spqs

$ setxkbmap
$ printf "Spaß" | xdotool type --clearmodifiers --file -
Spaß

Indeed, calling setxkbmap alone is enough to change future behavior.

What does `setxkbmap` do?

For what it's worth, here's a glimpse at what setxkbmap does:

ltrace setxkbmap

__libc_start_main(0x400f90, 1, 0x7ffc3fe26458, 0x403180 
calloc(4, 8)                                                                    = 0x1b9e010
__strdup(0x4034e5, 0, 32, 0x7f063a3e5b20)                                       = 0x1b9e040
strcmp(".", "/usr/share/X11/xkb")                                               = -1
__strdup(0x4032be, 0x4032be, 47, 0)                                             = 0x1b9e060
XkbOpenDisplay(0, 0, 0, 0x7ffc3fe2633c)                                         = 0x1b9e080
setlocale(LC_ALL, nil)                                                          = "C"
XkbRF_GetNamesProp(0x1b9e080, 0x7ffc3fe26308, 0x7ffc3fe26310, 0)                = 1
__snprintf_chk(0x7ffc3fe25330, 4096, 1, 4096)                                   = 13
XkbRF_Load(0x7ffc3fe25330, 0x7f063a1af997, 1, 1)                                = 0
__snprintf_chk(0x7ffc3fe25330, 4096, 1, 4096)                                   = 30
XkbRF_Load(0x7ffc3fe25330, 0x7f063a1af997, 1, 1)                                = 0x1ba04b0
XkbRF_GetComponents(0x1ba04b0, 0x605380, 0x7ffc3fe25300, 0)                     = 1
XkbGetKeyboardByName(0x1b9e080, 256, 0x7ffc3fe26310, 255)                       = 0x1bc21d0
XkbRF_SetNamesProp(0x1b9e080, 0x1ba0200, 0x605380, 0)                           = 1
XCloseDisplay(0x1b9e080)                                                        = 
exit(0 
+++ exited (status 0) +++

carnager commented 6 years ago

sigh. No, running just setxkbmap is not enough, as @Earnestly proved above. It will fix some cases, but not all. The only way to properly fix this, is by setting your actual keyboard layout with setxkbmap.

fidergo-stephane-gourichon commented 6 years ago

@carnager What you write suggest (I might be wrong) that you have a clear understanding of what happens, which I clearly do not have.

If anyone can write a summary, like:

"root cause is ...",
"position of root cause in code is ...",
"proper fix is ...",
"some cases look like fixes but aren't because actually they do ..." ... that would be great !

PS: I'm not affected by the bug, just trying to put two cents of method. :-)

Earnestly commented 6 years ago

@carnager Sadly no, not even setting your actual layout will fix it. The test case I've posted multiple times seems to be ignored.

jordansissel commented 6 years ago

This is a long standing issue in xdotool, and I don’t know if anyone has a solution yet. Sorry. I hope we find a solution — as the maintainer I would love to see this bug resolved if we can find a way!

On Sun, Mar 25, 2018 at 4:09 AM Earnestly notifications@github.com wrote:

@carnager https://github.com/carnager Sadly no, not even setting your actual layout will fix it. The test case I've posted multiple times seems to be ignored.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jordansissel/xdotool/issues/97#issuecomment-375962251, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIC6ghJZRAWj2Vm6ipQ1OD8nR9xHYReks5th3rWgaJpZM4HJEJO .

carnager commented 6 years ago

actually I believe this to be a XTEST bug. I tested all available alternatives back then and each had the same issue, so I don't think xdotool can do much about that.

jordansissel commented 6 years ago

actually I believe this to be a XTEST bug

@carnager My hunch was more around how a client (xdotool) can query xkbcommon and the older XGetKeyboardMapping.

Given users are typing correctly with their keyboard in their preferred locale, and given this problem is often resolved for xdotool by using setxkbmap, my assumption is that something is broken about the API xdotool uses for querying the keyboard map. I haven't dug much in recent years, but I did spend significant time frustrated trying to fix this problem in xdotool. I hope someone finds a solution some day :(

Earnestly commented 6 years ago

The problem isn't resolved by setting setxkbmap, at best it appears to be partial.

bdantas commented 5 years ago

@Earnestly - This problem of xdotool changing upper case Unicode characters to lower case has been a pain point for me for years, but I finally found a workaround.

First I tried xvkbd and pynput, but both of them disappointed (xvkbd sends a different character altogether, while pynput works with lower codepoint characters only).

I found xclip to be an acceptable workaround:

printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip
xdotool click 2

Running the above puts the string in X's primary clipboard (clarification here), then pastes the clipboard contents at the cursor. It results in the correct string: àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ

Limitation of this workaround is that the mouse pointer needs to be inside the window (and, if in a GUI text editor, below the line) where user is typing.

If someone has a cleaner workaround for entering Unicode characters (with correct case) into an X application via a script, please let me know.

carnager commented 5 years ago

@Earnestly - This problem of xdotool changing upper case Unicode characters to lower case has been a pain point for me for years, but I finally found a workaround.

First I tried xvkbd and pynput, but both of them disappointed (xvkbd sends a different character altogether, while pynput works with lower codepoint characters only).

I found xclip to be an acceptable workaround:
printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip
xdotool click 2
Running the above puts the string in X's primary clipboard (clarification here), then pastes the clipboard contents at the cursor. It results in the correct string: àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ

Limitation of this workaround is that the mouse pointer needs to be inside the window (and, if in a GUI text editor, below the line) where user is typing.

The even bigger limitation is that the application needs to support middle mouse click (think terminals with ncurses input dialog (which often needs the Shift key in addition)

bdantas commented 5 years ago

True, but this is the most "universal" solution I've been able to cook up with my limited knowledge of X.

If I go with the xclip approach, we have primary (selection/middle mouse click), "clipboard" (MS-style), and secondary clipboards to work with. Does one of them have a truly universal way of pasting--preferably one that does so at cursor location instead of pointer location? "Clipboard" requires Control+v in some places and Shift+Control+v in others, so no-go. Secondary clipboard seems very hard to paste from unless I'm missing something.

carnager commented 5 years ago

Nope, there is no universal solution. It all depends on the toolkit and implementation of the individual applications.

bdantas commented 5 years ago

Pity, but good to know. At least I won't chase my tail looking for a better clipboard-based workaround.

bdantas commented 5 years ago

I couldn't help myself and improved my workaround, anyway. All but one GUI application I tested accepted Control+v to paste from clipboard (mate-terminal is the one that didn't, but could be configured to accept it). So while this new version of the workaround may not be actually universal, it is nearly so--and the user does not have to worry about mouse pointer location because paste occurs at the cursor position:

xclip -o -selection clipboard | xclip -selection secondary # save clipboard contents
printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip -selection clipboard # put unicode characters in clipboard
xdotool key Control+v # paste from clipboard
xclip -o -selection secondary | xclip -selection clipboard # restore clipboard contents

jordansissel commented 5 years ago

Shift-insert is also what I use for pasting — https://github.com/jordansissel/keynav/blob/master/keynavrc#L96

On Fri, Mar 22, 2019 at 11:16 AM bdantas notifications@github.com wrote:

I couldn't help myself and improved it, anyway. All GUI apps I tested accepted Shift+Insert to paste from clipboard. This workaround may not be any more universal, but user does not need to worry about cursor location:

xclip -o -selection clipboard | xclip -selection secondary # save clipboard contents

printf 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ' | xclip -selection clipboard # put unicode characters in clipboard

xdotool key Shift+Insert # paste from clipboard

xclip -o -selection secondary | xclip -selection clipboard # restore clipboard contents

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jordansissel/xdotool/issues/97#issuecomment-475727095, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIC6hh8hp19ObwtcSDjZOF24zb0Wwtcks5vZR37gaJpZM4HJEJO .

TitouanT commented 4 years ago

Hi, I don't really know if it's the same issue but mine lead me here, and reading the discussion got me to a workaround (and maybe a solution ?).

First observation is that the example from @Earnestly will behave differently if you hold the shift key:

# don't hold shift
$ xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'
àáâãäçèéêëìíîïñòóôõöùúûüýÿàáâãäçèéêëìíîïñòóôõöùúûüý

#hold shift
$ xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'
ÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ

so it seems that xdotool print accentuated letters based on the state of the shift key.

Since I needed to make xdotool print only one letter in my script, I checked before hand the case I needed and set the shift key state accordingly:

islower=$(python -c "print('$letter'.islower())")
if [ "$islower" = "True" ]
then
    xdotool keyup Shift # because it might be down 
else
    xdotool keydown Shift
fi
xdotool type "$letter"
xdotool keyup Shift

TitouanT commented 4 years ago

also these four tests are interesting:

$ xdotool type 'aàÀ'
aàà # no shift
AÀÀ # shift

$ xdotool type 'AàÀ'
Aàà # no shift
Aàà # shift

$ xdotool type 'àÀa'
ààa # no shift
ÀÀA # shift

$ xdotool type 'àÀA'
ààA # no shift
ÀÀA # shift

From those, I get that capitalized ascii letter 'protect' letters to their right from the effect of the shift key. See the next example to show that point:

$ xdotool type 'aAaàÀ'
aAaàà # no shift
AAaàà # shift

I don't know if the behavior for simple ascii characters is intended to be like that with the shift key but to get a similar behavior with accentuated characters, xdotool would only need to correctly detect the case of such characters.

jordansissel commented 4 years ago

Not sure if this helps, Titouan, but you can try the --clearmodifiers flag which will tell xdotool to clear any currently-held modifier keys like shift/control/alt while it types.

On Thu, May 21, 2020 at 10:45 AM Titouan Teyssier notifications@github.com wrote:

also these four tests are interesting:

$ xdotool type 'aàÀ'

aàà # no shift

AÀÀ # shift

$ xdotool type 'AàÀ'

Aàà # no shift

Aàà # shift

$ xdotool type 'àÀa'

ààa # no shift

ÀÀA # shift

$ xdotool type 'àÀA'

ààA # no shift

ÀÀA # shift

From those, I get that capitalized ascii letter 'protect' letters to their right from the effect of the shift key. See the next example to show that point:

$ xdotool type 'aAaàÀ'

aAaàà # no shift

AAaàà # shift

I don't know if the behavior for simple ascci characters is intended to be like that with the shift key but to get a similar behavior with accentuated characters, xdotool would only need to correctly detect the case of such characters.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jordansissel/xdotool/issues/97#issuecomment-632248231, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAF2W75FJLXVGV6OET7LDRSVSCBANCNFSM4BZEIJHA .

TitouanT commented 4 years ago

Thanks @jordansissel but in this case it doesn't help to output capitalized accentuated characters.

user202729 commented 4 years ago

The bug is caused by the behavior of X:

Within each group, if the second element of the group is NoSymbol, then the group should be treated as if the second element were the same as the first element, except when the first element is an alphabetic KeySym "K" for which both lowercase and uppercase forms are defined. In that case, the group should be treated as if the first element were the lowercase form of "K" and the second element were the uppercase form of "K."

Because xdotool always find a completely-empty key code for the scratch space, the bug will always happen for single characters not already present in the keyboard mapping with "both lowercase and uppercase forms".

However, because xdotool only uses keysyms either in 0-0xff or those >= 0x01000000 (for Unicode characters), only the uppercase characters in ASCII range will cause the error.

The ranges in the patch covers all the uppercase characters in 0-0xff, so it should be good, except in the corner case that some uppercase character is already defined but in the lowercase range.

$ xmodmap -e "keycode 8 = A a"
$ xdotool type A

outputs a.

fidergo-stephane-gourichon commented 4 years ago

Trying to make sense of all this in layman's terms.

Experiments after JuanPotato's patch #283.

Tried current xdotool master.

TL;DR: Ran cases mentioned above, they work here after running setxkbmap (tested on XFCE and icewm sessions). Things fail when messing with the keyboard while xdotool runs.

Before `setxkbmap`: fail

xdotool type 'øøøø'

oooo
#Fail

echo "Spaß" | xdotool type --file -

Spqs
#Fail

sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

0áâãä972êë'úãöùúûüý
#Fail

sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

àáâãçèéêñóùúûýÿáâãäçéêìíòôö
#Fail

Running setxkbmap fixes cases in both environment.

After `setxkbmap`: success

echo -n "Spaß" | xdotool type --clearmodifiers --file -

Spaß
# Success

xdotool type 'øøøø'

øøøø
# Success

xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

àâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ
# Success

Messing with 'shift' key while xdotool runs: fail

# Hold shift while xdotool works
sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

0ÁÂÃÄ972ÊËÌÍÎÏÑÒÓÔÕÖ%ÚÛÜÝÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ
# Fail

# Press and release "shift" many times while xdotool works
sleep 1 ; xdotool type 'àáâãäçèéêëìíîïñòóôõöùúûüýÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ'

àÁÂãäçèéêëÌÍîïñòÓÔõöùúûüÝÿÀÁÂÃÄÇÈÉÊËÌÍÎÏÑÒÓÔÕÖÙÚÛÜÝ
# Fail

Trying to explain simply

Regarding lowercase/uppercase mess, I understand @user202729 explanation like this: X include some kind of "convenience shortcut" to "simplify" (avoid repeating oneself) creation of keymaps, and this caused xdotool to work in simplest cases and fail in others.
Regarding setxkbmap I have no real clue why things work after running it. Perhaps it processes something (the in-memory keymap?) so that it meets xdotool assumptions.

Overall, I feel like anyway what xdotool does is ask X to simulate keypresses, not character inputs, in a non-atomic way. Perhaps it has no choice, that's all X protocol actually allows. Anyway this makes the whole xdotool business susceptible to race conditions between inputs from actual keyboard and simulated inputs.

More fun messing with 'Control' key

# Hold "Control" after pressing Enter, but release it as soon as you see the ^C
sleep 1 ; xdotool type 'c'

^C
$ ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

"c" continues to appear until you "type something". Also, the next actual keypress on the 'c' key would seem ignored.

I guess from X point of view, your c key just got stuck because it received a KeyDown event but xdotool was interrupted before providing the corresponding KeyUp event.

Try again and while 'c's appear click into another window where things can be types: the streams of 'c's continues there.

Conclusions

(My conclusions, anyway)

One thing that could be needed (or not) is a kind of transactional API where X clients would send a batch of commands to be executed atomically, but I'm not even sure this would be foolproof or even desirable.
One thing that seems better is a higher level API where you can directly inject characters or symbols, not individual keypresses, irrespective of current keymap and keyboard stats.
X being phased out those will probably never happen.
Writing xdotool is a tricky business. Kudos for the maintainers!

Hoping this clarifies somehow to at least one person, as the explanation by @user202729 seemed cryptic for me. Please anyone correct me if I'm wrong.

rampaq commented 3 years ago

There is also a weird issue with two keyboard layouts in 3.20210804.1/2, which was not present in previous versions.

I set a primary Czech (cz) keyboard and add a russian phonetic layout, toggled between by Alt+Shift. setxkbmap -layout cz,cz -variant ,rus -option grp:alt_shift_toggle

Then a capital 'Y' types as russian 'Ы' for some reason.

echo "Y" | xdotool type --clearmodifiers --file -                                             
# output: Ы

echo "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" | xdotool type --clearmodifiers --file -                                             
# output: ABCDEFGHIJKLMNOPQRSTUVWXЫZabcdefghijklmnopqrstuvwxyz

What is interesting that this happens only with capital 'Y'. I thought that xdotool pushes the toggle switch command but that does not seem to be the case as setxkbmap -layout cz,cz -variant ,rus -option grp:alt_space_toggle (changed toggle key) behaves exactly the same.

However, when I change the primary layout to us then everything is ok:

setxkbmap -layout us,us -variant ,rus -option grp:alt_space_toggle
echo "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" | xdotool type --clearmodifiers --file -
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

And yes, the 'Y' is still ascii hex 59 on both keyboards.

Note: setxkbmap after setting the layout does not solve the problem, as was suggested by some here

jordansissel / xdotool

xdotool is changing letters #97

Indeed `setxkbmap` changes something

What does `setxkbmap` do?

Experiments after JuanPotato's patch #283.

Before `setxkbmap`: fail

After `setxkbmap`: success

Messing with 'shift' key while xdotool runs: fail

Trying to explain simply

More fun messing with 'Control' key

Conclusions

jordansissel / xdotool

xdotool is changing letters #97

Indeed setxkbmap changes something

What does setxkbmap do?

Experiments after JuanPotato's patch #283.

Before setxkbmap: fail

After setxkbmap: success

Messing with 'shift' key while xdotool runs: fail

Trying to explain simply

More fun messing with 'Control' key

Conclusions

Indeed `setxkbmap` changes something

What does `setxkbmap` do?

Before `setxkbmap`: fail

After `setxkbmap`: success