scaled / scaled

My own private IDE-aho.
Other
148 stars 9 forks source link

Character/Keyboard Mixup for Non-Qwerty Layouts #4

Open xpe opened 9 years ago

xpe commented 9 years ago

I use the Colemak layout. When I type normal characters into Scaled, they appear correctly. However, commands such as "C-e" are not interpreted correctly. Perhaps there are some low-level options you can check on?

(For example, the "k" key in Qwerty corresponds to the "e" key in Colemak, for example. So when I press "C-e" Scaled deletes the line.)

samskivert commented 9 years ago

Interesting! JavaFX (which is the UI toolkit on which Scaled is built) is woefully limited in how it allows me to interact with the keyboard, but hopefully I can figure out some way to extract the right information from its key events.

I'm guessing that when it reports a key pressed event (which has the logical keycodes), it reports K (the Qwerty key) even though you pressed E (in your Colemak layout), and only when the key typed event comes in, does it report a character code of 'e'.

Or it might have to do with the fact that the CTRL key is held down. JavaFX seems to arbitrarily decide whether to provide "keymap translated" information depending on which modifier keys are down. It does different things if SHIFT is down, or if CTRL is down, or if META is down. And there are weird bugs where it generates spurious keyboard events in some circumstances. I have an outstanding bug that's been bouncing between JDK/JavaFX engineers for six months now: https://javafx-jira.kenai.com/browse/RT-37399

Anyway, that bug report has a program in it:

import java.util.Arrays;
import java.util.Set;
import java.util.HashSet;
import javafx.application.Application;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.Scene;
import javafx.scene.input.KeyEvent;
import javafx.scene.control.Label;
import javafx.scene.layout.StackPane;
import javafx.stage.Stage;

public class MetaDot extends Application {

  public static void main(String[] args) {
    launch(args);
  }

  @Override
  public void start(Stage primaryStage) {
    primaryStage.setTitle("Meta Dot!");
    StackPane root = new StackPane();
    root.getChildren().add(new Label("Press Meta Dot."));
    primaryStage.addEventHandler(KeyEvent.ANY, ev -> System.out.println(toDebugString(ev)));
    primaryStage.setScene(new Scene(root, 300, 250));
    primaryStage.show();
  }

  private String toDebugString (KeyEvent ev) {
    StringBuilder sb = new StringBuilder();
    sb.append("type=").append(ev.getEventType());
    sb.append(" code=").append(ev.getCode());
    sb.append(" mods=").append(getModifiers(ev));
    sb.append(" char='").append(ev.getCharacter());
    sb.append("' (");
    for (char c : ev.getCharacter().toCharArray()) sb.append(" ").append((int)c);
    sb.append(")");
    return sb.toString();
  }

  private Set<String> getModifiers (KeyEvent ev) {
    Set<String> set = new HashSet<>();
    if (ev.isAltDown()) set.add("alt");
    if (ev.isControlDown()) set.add("ctrl");
    if (ev.isMetaDown()) set.add("meta");
    if (ev.isShiftDown()) set.add("shift");
    return set;
  }
}

If you could compile that program, and press CTRL-e and send me the output, that'd give me a better idea of what events JavaFX is generating and what I might need to do to work around its foolishness.

Eventually, I'm probably going to have to write native code to handle key input because the JavaFX team seems not to know what they're doing much in that regard. But I'd like to avoid that substantial complication to Scaled's build and deployment as long as I can.

samskivert commented 9 years ago

Actually, if you could type just plain e, then CTRL-e, then SHIFT-e, then META-e, then ALT-e, that would be useful. Because JavaFX does weird things with all of them and I'd better find out now how all that weirdness is further weirded by having a non-standard keyboard layout.

samskivert commented 9 years ago

Also, just for reference, what OS are you using?

xpe commented 9 years ago

I'm using Mac OS X 10.10.1.

Thanks for the debugging program! That's a nice way to debug keystrokes -- valuable to any JavaFX developer for sure! The results are (note the ; comments for clarity -- I typed them in):

; e
type=KEY_PRESSED code=K mods=[] char='' ( 0)
type=KEY_TYPED code=UNDEFINED mods=[] char='e' ( 101)
type=KEY_RELEASED code=K mods=[] char='' ( 0)

; C-e
type=KEY_PRESSED code=CONTROL mods=[ctrl] char='' ( 0)
type=KEY_PRESSED code=K mods=[ctrl] char='' ( 0)
type=KEY_TYPED code=UNDEFINED mods=[ctrl] char='' ( 5)
type=KEY_RELEASED code=K mods=[ctrl] char='' ( 0)
type=KEY_RELEASED code=CONTROL mods=[] char='' ( 0)

; S-e
type=KEY_PRESSED code=SHIFT mods=[shift] char='' ( 0)
type=KEY_PRESSED code=K mods=[shift] char='' ( 0)
type=KEY_TYPED code=UNDEFINED mods=[shift] char='E' ( 69)
type=KEY_RELEASED code=K mods=[shift] char='' ( 0)
type=KEY_RELEASED code=SHIFT mods=[] char='' ( 0)

; M-e
type=KEY_PRESSED code=ALT mods=[alt] char='' ( 0)
type=KEY_PRESSED code=K mods=[alt] char='' ( 0)
type=KEY_TYPED code=UNDEFINED mods=[alt] char='' ()
type=KEY_RELEASED code=K mods=[alt] char='' ( 0)
type=KEY_RELEASED code=ALT mods=[] char='' ( 0)

; s-e (e.g. Super-e / Command-e)
type=KEY_PRESSED code=COMMAND mods=[meta] char='' ( 0)
type=KEY_PRESSED code=K mods=[meta] char='' ( 0)
type=KEY_TYPED code=UNDEFINED mods=[meta] char='e' ( 101)
type=KEY_RELEASED code=K mods=[meta] char='' ( 0)
type=KEY_RELEASED code=COMMAND mods=[] char='' ( 0)
type=KEY_PRESSED code=COMMAND mods=[meta] char='' ( 0)
type=KEY_RELEASED code=COMMAND mods=[] char='' ( 0)
type=KEY_PRESSED code=COMMAND mods=[meta] char='' ( 0)
type=KEY_RELEASED code=COMMAND mods=[] char='' ( 0)
type=KEY_PRESSED code=COMMAND mods=[meta] char='' ( 0)
type=KEY_RELEASED code=COMMAND mods=[] char='' ( 0)
type=KEY_PRESSED code=COMMAND mods=[meta] char='' ( 0)
type=KEY_PRESSED code=CONTROL mods=[ctrl, meta] char='' ( 0)
type=KEY_RELEASED code=COMMAND mods=[ctrl] char='' ( 0)
type=KEY_RELEASED code=CONTROL mods=[] char='' ( 0)
type=KEY_PRESSED code=COMMAND mods=[meta] char='' ( 0)

WAT? You weren't kidding about the non-uniformity. :)

samskivert commented 9 years ago

Blah. That's really unfortunate. You'll notice that in most situations, 'e' is nowhere to be seen. How am I to discover that you've actually typed a CTRL-e when JavaFX is cleverly keeping that information from me completely?

Looks like I'm going to have to dig into the JavaFX source and see if I can extract what I need, otherwise I'm going to have to write a native library for mapping raw key codes based on the current keymap. Whee!

xpe commented 9 years ago

I created an issue here: https://javafx-jira.kenai.com/browse/RT-39765 (login required just to view the ticket, shamefully!)

As you know, we can't trust the .getCode method on KEY_PRESSED events.

Still, the .getCharacter method in the follow-on KEY_TYPED event seems to offer hope. I don't know if this is cross-platform compatible though.

As I mention over there:

(Last edited at 5:38 pm EST. I should use preview more.)

samskivert commented 9 years ago

This also seems crazy, because it means that menu shortcuts in a JavaFX program must not work properly for people who have non-standard keymaps. I did some Googling and found some people complaining about apparently related issues, but nothing talking about that limitation directly.

Here's another JavaFX program. Can you compile it and see if Ctrl-X "clears" as it claims it will?

import java.util.AbstractMap.SimpleEntry;
import java.util.Map.Entry;
import javafx.application.Application;
import javafx.event.ActionEvent;
import javafx.geometry.Insets;
import javafx.geometry.Pos;
import javafx.scene.Scene;
import javafx.scene.control.*;
import javafx.scene.effect.DropShadow;
import javafx.scene.effect.Effect;
import javafx.scene.effect.Glow;
import javafx.scene.effect.SepiaTone;
import javafx.scene.image.Image;
import javafx.scene.image.ImageView;
import javafx.scene.input.KeyCombination;
import javafx.scene.layout.VBox;
import javafx.scene.paint.Color;
import javafx.scene.text.Font;
import javafx.scene.text.TextAlignment;
import javafx.stage.Stage;

public class MenuSample extends Application {

  final PageData[] pages = new PageData[] {
    new PageData("Apple", "http://upload.wikimedia.org/wikipedia/commons/thumb/1/15/Red_Apple.jpg/265px-Red_Apple.jpg",
                 "The apple is the pomaceous fruit of the apple tree, species Malus "
                 +"domestica in the rose family (Rosaceae). It is one of the most "
                 +"widely cultivated tree fruits, and the most widely known of "
                 +"the many members of genus Malus that are used by humans. "
                 +"The tree originated in Western Asia, where its wild ancestor, "
                 +"the Alma, is still found today.",
                 "Malus domestica"),
    new PageData("Hawthorn", "http://www.2020site.org/trees/images/Maturetree%20(2).jpg",
                 "The hawthorn is a large genus of shrubs and trees in the rose "
                 + "family, Rosaceae, native to temperate regions of the Northern "
                 + "Hemisphere in Europe, Asia and North America. "
                 + "The name hawthorn was "
                 + "originally applied to the species native to northern Europe, "
                 + "especially the Common Hawthorn C. monogyna, and the unmodified "
                 + "name is often so used in Britain and Ireland.",
                 "Crataegus monogyna"),
    new PageData("Ivy", "http://upload.wikimedia.org/wikipedia/commons/thumb/8/87/Ivy_44uf.jpg/320px-Ivy_44uf.jpg",
                 "The ivy is a flowering plant in the grape family (Vitaceae) native"
                 +" to eastern Asia in Japan, Korea, and northern and eastern China."
                 +" It is a deciduous woody vine growing to 30 m tall or more given "
                 +"suitable support,  attaching itself by means of numerous small "
                 +"branched tendrils tipped with sticky disks.",
                 "Parthenocissus tricuspidata"),
    new PageData("Quince", "http://images.wisegeek.com/quince.jpg",
                 "The quince is the sole member of the genus Cydonia and is native"
                 +" to warm-temperate southwest Asia in the Caucasus region. The "
                 +"immature fruit is green with dense grey-white pubescence, most "
                 +"of which rubs off before maturity in late autumn when the fruit "
                 +"changes color to yellow with hard, strongly perfumed flesh.",
                 "Cydonia oblonga")
  };

  final String[] viewOptions = new String[] {
    "Title",
    "Binomial name",
    "Picture",
    "Description"
  };

  final Entry<String, Effect>[] effects = new Entry[] {
    new SimpleEntry<>("Sepia Tone", new SepiaTone()),
    new SimpleEntry<>("Glow", new Glow()),
    new SimpleEntry<>("Shadow", new DropShadow())
  };

  final ImageView pic = new ImageView();
  final Label name = new Label();
  final Label binName = new Label();
  final Label description = new Label();
  private int currentIndex = -1;

  public static void main(String[] args) {
    launch(args);
  }

  @Override
  public void start(Stage stage) {
    stage.setTitle("Menu Sample");
    Scene scene = new Scene(new VBox(), 400, 350);
    scene.setFill(Color.OLDLACE);

    name.setFont(new Font("Verdana Bold", 22));
    binName.setFont(new Font("Arial Italic", 10));
    pic.setFitHeight(150);
    pic.setPreserveRatio(true);
    description.setWrapText(true);
    description.setTextAlignment(TextAlignment.JUSTIFY);

    shuffle();

    MenuBar menuBar = new MenuBar();

    final VBox vbox = new VBox();
    vbox.setAlignment(Pos.CENTER);
    vbox.setSpacing(10);
    vbox.setPadding(new Insets(0, 10, 0, 10));
    vbox.getChildren().addAll(name, binName, pic, description);

    // --- Menu File
    Menu menuFile = new Menu("File");
    MenuItem add = new MenuItem("Shuffle"/*,
                                new ImageView(new Image("menusample/new.png"))*/);
    add.setOnAction((ActionEvent t) -> {
      shuffle();
      vbox.setVisible(true);
    });

    MenuItem clear = new MenuItem("Clear");
    clear.setAccelerator(KeyCombination.keyCombination("Ctrl+X"));
    clear.setOnAction((ActionEvent t) -> {
      vbox.setVisible(false);
    });

    MenuItem exit = new MenuItem("Exit");
    exit.setOnAction((ActionEvent t) -> {
      System.exit(0);
    });

    menuFile.getItems().addAll(add, clear, new SeparatorMenuItem(), exit);

    // --- Menu Edit
    Menu menuEdit = new Menu("Edit");

    // --- Menu View
    Menu menuView = new Menu("View");

    menuBar.getMenus().addAll(menuFile, menuEdit, menuView);
    ((VBox) scene.getRoot()).getChildren().addAll(menuBar, vbox);
    stage.setScene(scene);
    stage.show();
  }

  private void shuffle() {
    int i = currentIndex;
    while (i == currentIndex) {
      i = (int) (Math.random() * pages.length);
    }
    pic.setImage(pages[i].image);
    name.setText(pages[i].name);
    binName.setText("(" + pages[i].binNames + ")");
    description.setText(pages[i].description);
    currentIndex = i;
  }

  private class PageData {
    public String name;
    public String description;
    public String binNames;
    public Image image;
    public PageData(String name, String url, String description, String binNames) {
      this.name = name;
      this.description = description;
      this.binNames = binNames;
      image = new Image(url, 400, 300, true, true);
    }
  }
}
samskivert commented 9 years ago

I'm assuming that Ctrl-X will not work, and Ctrl-whatever-is-X-on-qwerty will be the only thing that activates the clear menu item.

But if that's not the case, then JavaFX must be doing something internally which perhaps I can leverage.

samskivert commented 9 years ago

Re: the bug report, thanks! Given that my other JavaFX bug report is still languishing after six months, I'm not wildly hopeful that they'll ever fix it, but it's clearly good to bring it to their attention.

Re: the ASCII key code, yeah! I was thinking about that, but for some reason was thinking that you were pressing CTRL-e (Colemak) and getting the ASCII code for CTRL-k. But no, you're pressing CTRL-e (Colemak) and getting the right ASCII code.

Obviously there are not ASCII codes for everything, but I think there are "de facto" ASCII codes for a bunch of the standard CTRL combinations (i.e. CTRL-g is ACSII "BEL" / 0x07 , etc.). So I may be able to reconstruct the key combination from ASCII in cases where I can.

On the other hand, that may be a pointless exercise, because Emacs (and hence Scaled) uses so many other combinations that it would almost be more annoying to have it "work" for the control combinations but then fail every time you tried to use the Meta key. Then I'd be temping you to use Scaled because it mostly works, except it would beat you over the head with a hammer when you occasionally used some Meta binding.

At least I can detect when I'm in that situation and maybe I can just pop up an error saying "SORRY! Meta and Alt keys don't work with alternate keyboard layouts. That's probably better than doing something totally random. And at least one can use ESC then key to emulate M-key.

None of that is great, but it may bridge the gap until the JavaFX team fixes the bug... hope hope hope.

xpe commented 9 years ago

In the MenuSample application, the Control-X is detected. However, "x" is the same in both Colemak and Qwerty. (By design, the lower left keys, e.g. undo, cut, copy, paste are not changed.)

I changed the program to have this line instead: clear.setAccelerator(KeyCombination.keyCombination("Ctrl+J")); and observed that my Colemak Ctrl+J is not detected. I have to use the physical Qwerty key.

samskivert commented 9 years ago

Well, that confirms my suspicions. Now I'm slightly regretting building on top of JavaFX, given the shockingly immature nature of the platform. You'd think two major revisions would have ironed out things like this. Alas.

Hopefully they'll address this issue, but in the meanwhile, I'll implement the ASCII control code workaround, and look into getting access to the OS keymap. Maybe the AWT or Swing provides a way to translate raw key codes into mapped key codes, which I can sneakily leverage.

samskivert commented 9 years ago

This also looks promising in the absence of a non-native-library workaround: https://code.google.com/p/jnativehook/

xpe commented 9 years ago

@samskivert Thanks for your help.

iphydf commented 6 years ago

I just ran into this issue when trying to create a new UI on JavaFX. Did you end up figuring something out?