Exposing the lookup of (x,y) pixel coordinates to corresponding activeTextEditor (line,col)

[shbonita]

Problem Description (all platforms, all versions) We are in the process of developing a plugin for our eye tracking infrastructure (iTrace) for vscode. This plugin we call iTrace-vscode will let us track a developer's eyes 👀 on specific code elements while they are working on a software task. iTrace is a client server architecture where the server sends eye tracking data to the client (in this case is vscode). The client in turn takes the raw gaze which is usually a (x,y) pixel coordinate and maps it to the line and col that pixel falls on. Please see www.i-trace.org for more details. We have plugins currently for Eclipse, Visual Studio, and Atom that are already implemented in this case.

When we tried to do this for vscode. We hit a roadblock. Here is how I can best describe it:

vscode provides us with a vscode.window.activeTextEditor?.visibleRanges[0]; call, which gives us the lines of text that are currently on screen. However, we have no way to then get info from pixel measurements. vscode seems to have no API features that give information about pixel locations or measurements. If you are able to add a function call to vscode.window.activeTextEditor or something in the visibleRanges for us to give (x,y) info and get back the (line,column) data, we would be golden.

Visual Studio exposes this functionality so we do have a plugin for that at https://github.com/iTrace-Dev/iTrace-VisualStudio. However, we have had numerous requests from the software engineering research and industry community to enable eye tracking in vscode because it is considered to be the primary IDE for developers. iTrace-Atom supports this functionality however since it is being sunsetted, vscode becomes our only option moving forward. Since both are based on Electron, is there a reason this API call is unavailable? Can we somehow use a API call to get the line and col info we need. Thanks so much. :pray: 👁️

[VSCodeTriageBot]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[VSCodeTriageBot]

:slightly_smiling_face: This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[AZHenley]

@isidorn

[isidorn]

Thanks for the ping. As I mentioned in our email discussion:

My initial feeling is that this does not align with VS Code’s API philosophy since we do not expose direct UI elements like pixels anywhere in the API. However I still think opening a request makes sense so we can discuss it in the open and see if there is some workaround.

[shbonita]

@isidorn Thanks for responding to this request. I want to clarify that we are not looking to manipulate any pixels on screen, nor do we need direct access to UI elements. We would just like to receive the line and col as output if we provide as input the (x,y) coordinate such as that of the mouse for instance (in our case it comes from an eye tracker).

Input: (x,y) location on screen Output: (line,col)

If there is a workaround to getting this information, please let us know.

[shbonita]

@isidorn checking back on what the status of this might be. Please see my clarification comment from Oct 27. Thanks

[isidorn]

@shbonita thanks for the clarification, but there is no API to achieve this, and neither is there a workaround. I understand you want the API to be read-only, but there are no plans to add this functionality for now.

If I remember correctly, I think you want to do more precise user studies. For those user studies you could create a fork of VS Code which exposes this API for example.