Open dbalatero opened 3 years ago
@yoricfr This is an example of a built .so
library that bridges Obj-C and Lua together:
This is how to get a focused UI element, which might have a UITextView
pointer under the hood:
https://www.hammerspoon.org/docs/hs.uielement.html#focusedElement
What's remarkable with XVim2: the block adjust with the font and char's width.
Nova (constant block width)
Terminal (constant block width)
Xcode + XVim (variable block width!)
Edit: SublimeText handles it pretty well too, with a sleek transparency effect:
@yoricfr This is an example of a built .so library that bridges Obj-C and Lua together: https://github.com/asmagill/hs._asm.axuielement This is how to get a focused UI element, which might have a
UITextView
pointer under the hood: https://www.hammerspoon.org/docs/hs.uielement.html#focusedElement
@dbalatero Thanks for the links. I feel like trying to build the simplest Mac application with a NSTextView
in it. Then I'll do some experimenting with overriding the drawInsertionPoint()
function and see what it does.
Like your link to JugglerShu or this link to christianTietze.
If I get a block caret, then I'll try to figure out the Lua integration in Hammerspoon.
That sounds great, thanks for your energy on this!
Yeah, the order of operations seems like:
.so
module from LuaUITextView
and swaps its insertion pointOne big thing to figure out:
It's one thing for an application to manage its own UITextView caret style.
What we want is to reach into other applications and mess around with its caret style from the Hammerspoon process.
Is this considered OK by the APIs? Or is it securely isolated?
Here's some dev notes on it, which is great. It looks like it's patching NSTextView though to achieve this.
This Stackoverflow post seems to indicate that you can do it per-application, but not system wide: https://superuser.com/questions/429464/change-the-width-and-color-of-mac-os-x-text-caret-cursor
@yoricfr I did an experiment, and was able to draw a box over the next character.
Downsides:
To try it, bind this code to a hot key, put your cursor somewhere, and fire the hotkey. You should see a box.
local ax = require("hs.axuielement")
currentAxElement = function()
local systemElement = ax.systemWideElement()
return systemElement:attributeValue("AXFocusedUIElement")
end
hs.hotkey.bind(super, 'e', function()
local currentElement = currentAxElement()
-- Get the current selection
local range = currentElement:attributeValue("AXSelectedTextRange")
-- Get the range for the next character after the blinking cursor
local caretRange = {
location = range.location,
length = 1,
}
-- get the { h, w, x, y } bounding box for the next character's range
local bounds = currentElement:parameterizedAttributeValue("AXBoundsForRange", caretRange)
-- draw a black rectangle in the bounding box with 20% opacity
local canvas = hs.canvas.new(bounds)
canvas:insertElement(
{
type = 'rectangle',
action = 'fill',
fillColor = { red = 0, green = 0, blue = 0, alpha = 0.2 },
frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
withShadow = false
},
1
)
canvas:level('overlay')
canvas:show()
end)
I did another pass at this. This time, we redraw the cursor at a 60fps rate, so it captures any movement.
intervalTimer = nil
hs.hotkey.bind(super, 'e', function()
-- draw a black rectangle in the bounding box with 20% opacity
local canvas = hs.canvas.new({ x = 0, y = 0, h = 1, w = 1 })
canvas:level('overlay')
canvas:insertElement(
{
type = 'rectangle',
action = 'fill',
fillColor = { red = 0, green = 0, blue = 0, alpha = 0.2 },
frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
withShadow = false
},
1
)
local repositionCursor = function()
local currentElement = currentAxElement()
-- Get the current selection
local range = currentElement:attributeValue("AXSelectedTextRange")
-- Last visible char
local visibleRange = currentElement:attributeValue("AXVisibleCharacterRange")
local lastVisibleIndex = visibleRange.length + visibleRange.location
if range.location == lastVisibleIndex then
-- hide the caret if we're at the end of the text box
canvas:hide()
else
-- Get the range for the next character after the blinking cursor
local caretRange = {
location = range.location,
length = 1,
}
-- get the { h, w, x, y } bounding box for the next character's range
local bounds = currentElement:parameterizedAttributeValue("AXBoundsForRange", caretRange)
-- move the position and resize
canvas:topLeft({ x = bounds.x, y = bounds.y })
canvas:size({ h = bounds.h, w = bounds.w })
-- show if not shown
canvas:show()
end
end
repositionCursor()
refresh = 1 / 60 -- 60fps
intervalTimer = hs.timer.doEvery(refresh, repositionCursor)
end)
Ok I had another insane idea - what if we covered the blinking cursor with the background color of the text field?
Here's what drawing a "cursor cover" looks like, when I make it obvious and red:
To get the background color of the UITextInput, you can take a screenshot and get the color at a pixel:
local screenshotOfTextInput = hs.screen.mainScreen():snapshot(currentElement:attributeValue("AXFrame"))
local color = screenshotOfTextInput:colorAt({ x = 10, y = 10 })
If we set the color to that instead, you get:
A GIF of it in action:
This is probably not perfect:
2px
to cover sub-pixel offsets of the cursorFinal lua code (for now I just paste it in ~/.hammerspoon/init.lua
as a proof of concept):
currentAxElement = function()
local systemElement = ax.systemWideElement()
return systemElement:attributeValue("AXFocusedUIElement")
end
intervalTimer = nil
hs.hotkey.bind(super, 'e', function()
-- draw a black rectangle in the bounding box with 20% opacity
local canvas = hs.canvas.new({ x = 0, y = 0, h = 1, w = 1 })
canvas:level('overlay')
-- block caret
canvas:insertElement(
{
type = 'rectangle',
action = 'fill',
fillColor = { red = 0, green = 0, blue = 0, alpha = 0.2 },
frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
withShadow = false
},
1
)
-- cursor disabler
local cursorDisableCanvas = hs.canvas.new({ x = 0, y = 0, h = 1, w = 1 })
cursorDisableCanvas:insertElement(
{
type = 'rectangle',
action = 'fill',
fillColor = { red = 255, green = 0, blue = 0, alpha = 1 },
frame = { x = "0%", y = "0%", h = "100%", w = "100%", },
withShadow = false
},
1
)
local repositionCursor = function()
local currentElement = currentAxElement()
-- Get the background color
local screenshotOfTextInput = hs.screen.mainScreen():snapshot(currentElement:attributeValue("AXFrame"))
local color = screenshotOfTextInput:colorAt({ x = 10, y = 10 })
-- Get the current selection
local range = currentElement:attributeValue("AXSelectedTextRange")
-- Last visible char
local visibleRange = currentElement:attributeValue("AXVisibleCharacterRange")
local lastVisibleIndex = visibleRange.length + visibleRange.location
if range.location == lastVisibleIndex then
-- hide the caret if we're at the end of the text box
canvas:hide()
cursorDisableCanvas:hide()
else
-- Get the range for the next character after the blinking cursor
local caretRange = {
location = range.location,
length = 1,
}
-- get the { h, w, x, y } bounding box for the next character's range
local bounds = currentElement:parameterizedAttributeValue("AXBoundsForRange", caretRange)
-- move the position and resize
canvas:topLeft({ x = bounds.x, y = bounds.y })
canvas:size({ h = bounds.h, w = bounds.w })
-- show if not shown
canvas:show()
-- disable the cursor
cursorDisableCanvas:topLeft({ x = bounds.x - 1, y = bounds.y })
cursorDisableCanvas:size({ h = bounds.h, w = 2 })
cursorDisableCanvas:elementAttribute(1, 'fillColor', color)
cursorDisableCanvas:show()
end
end
repositionCursor()
refresh = 1 / 60 -- 60fps
intervalTimer = hs.timer.doEvery(refresh, repositionCursor)
end)
@dbalatero How on earth do you come up with these ideas?
Really, you're taking a snapshot to determine the background colour, then color the cursor to make the caret disappear?!
And this is being done 60 times per second with a bunch of other stuff in the repositionCursor()
function, how can it be that fast?!
I've copied your proof of concept in my init.lua file and it looks like magic. No matter the font and text size, the block's width is adjusting accordingly.
I don't know how you came up with calls like attributeValue("AXFrame")
and parameterizedAttributeValue("AXBoundsForRange", caretRange)
. I didn't even know it was possible.
About the simplest Mac application, I simply added a NSTextView
programmatically after the code automatically generated by Xcode for the window. MyTextView
is a sub-class of NSTextView
so we can override the drawInsertionPoint
function:
import Cocoa
import SwiftUI
@main
class AppDelegate: NSObject, NSApplicationDelegate {
var window: NSWindow!
func applicationDidFinishLaunching(_ aNotification: Notification) {
// Create the window and set the content view.
window = NSWindow(
contentRect: NSRect(x: 0, y: 0, width: 640, height: 480),
styleMask: [.titled, .closable, .miniaturizable, .resizable, .fullSizeContentView],
backing: .buffered, defer: false)
window.center()
window.makeKeyAndOrderFront(nil)
// Code added to add a NSTextView
// Instead of using NSTextView, we use our own class that is a sub-class of NSTextView
let ed = MyTextView(frame: NSMakeRect(20, 30, 360, 280))
ed.font = NSFont(name:"Chalkduster", size:20)
ed.string = "Chalkduster"
ed.isEditable = true
ed.isSelectable = true
window.contentView!.addSubview(ed)
}
func applicationWillTerminate(_ aNotification: Notification) {
// Insert code here to tear down your application
}
}
// Our customised NSTextView
class MyTextView: NSTextView {
var caretSize: CGFloat = 10
open override func drawInsertionPoint(in rect: NSRect, color: NSColor, turnedOn flag: Bool) {
var rect = rect
let customColor = NSColor(red: 1.0, green: 0.3, blue: 0.1, alpha: 0.5)
rect.size.width = caretSize
super.drawInsertionPoint(in: rect, color: customColor, turnedOn: flag)
}
open override func setNeedsDisplay(_ rect: NSRect, avoidAdditionalLayout flag: Bool) {
var rect = rect
rect.size.width += caretSize
super.setNeedsDisplay(rect, avoidAdditionalLayout: flag)
}
}
I tried to play a little with boundingRect
to get the exact box of the next letter, without success so far.
But anyway, as you said, customising the caret from a test application is the easy part. The question is wether it is possible or not to override the drawInsertionPoint
of any frontmost application.
I have no clue. For now, I am in awe with your solution.
How on earth do you come up with these ideas?
For one, the AX documentation is really bad so I made a debug helper:
function debugElement(currentElement)
local role = currentElement:attributeValue("AXRole")
if role == "AXTextField" or role == "AXTextArea" or role == "AXComboBox" then
logger.i("Currently in text field")
logger.i(inspect(currentElement:parameterizedAttributeNames()))
logger.i("attributes:")
logger.i("-----------")
local attributes = currentElement:allAttributeValues()
local names = {}
for name in pairs(attributes) do table.insert(names, name) end
table.sort(names)
for _, name in ipairs(names) do
logger.i(" " .. name .. ": " .. inspect(attributes[name]))
end
logger.i("action names:")
local names = currentElement:actionNames()
logger.i(inspect(names))
logger.i("action descriptions:")
logger.i("--------------------")
for _, name in ipairs(names) do
logger.i(" " .. name .. ": " .. currentElement:actionDescription(name))
end
else
logger.i("Role = " .. role)
end
end
hs.hotkey.bind(super, 'd', function()
local systemElement = ax.systemWideElement()
local currentElement = systemElement:attributeValue("AXFocusedUIElement")
debugElement(currentElement)
end
This dumps out all the parameterized attributes and simple attributes, then I just stare at it and look for interesting combinations.
As far as the screenshot thing, I think I just googled for "hammerspoon color at pixel" and ended up on this issue: https://github.com/Hammerspoon/hammerspoon/issues/1559
which linked to here: https://github.com/Hammerspoon/hammerspoon/pull/1868
and ended up with that idea…
And this is being done 60 times per second with a bunch of other stuff in the repositionCursor() function, how can it be that fast?!
I'm not sure actually. I guess all I know is that I intend for it to run every 1/60th of a second, but it could actually be taking longer than 1 / 60 = 16.66ms
to run the function. I guess I'd actually need to time it.
To get the solution I have actually beyond a proof of concept, some extra things might need to be done:
AXIdentifier
we could use for uniqueness? not sure.)the AX documentation is really bad so I made a debug helper:
That is brilliant, thanks for sharing. It's like being given a new pair of gleamy eyes in the darkness.
Up to now I was exploring with :
for k,v in pairs(currentElement) do
hs.printf("%s - %s", k, v)
end
which obviously don't dig into the table pointers:
2020-11-02 13:23:17: AXParent - hs.axuielement: AXScrollArea (0x6000031fe638)
2020-11-02 13:23:17: AXRole - AXTextArea
2020-11-02 13:23:17: AXWindow - hs.axuielement: AXWindow (0x6000031da7f8)
2020-11-02 13:23:17: AXSharedCharacterRange - table: 0x6000031db5c0
2020-11-02 13:23:17: AXFrame - table: 0x6000031ea600
2020-11-02 13:23:17: AXSelectedTextRanges - table: 0x6000031d8900
2020-11-02 13:23:17: AXChildrenInNavigationOrder - table: 0x6000031ea7c0
2020-11-02 13:23:17: AXSelectedText -
2020-11-02 13:23:17: AXTopLevelUIElement - hs.axuielement: AXWindow (0x6000031d93b8)
2020-11-02 13:23:17: AXIdentifier - First Text View
2020-11-02 13:23:17: AXTextInputMarkedRange - table: 0x6000031eb680
2020-11-02 13:23:17: AXSize - table: 0x6000031e8c80
2020-11-02 13:23:17: AXInsertionPointLineNumber - 2
2020-11-02 13:23:17: AXChildren - table: 0x6000031eb7c0
2020-11-02 13:23:17: AXHelp - nil
2020-11-02 13:23:17: AXValue - Chalkduster
2020-11-02 13:23:17: AXRoleDescription - text entry area
2020-11-02 13:23:17: AXFocused - true
2020-11-02 13:23:17: AXSharedTextUIElements - table: 0x6000031dad40
2020-11-02 13:23:17: AXVisibleCharacterRange - table: 0x6000031e9300
2020-11-02 13:23:17: AXPosition - table: 0x6000031dbc40
2020-11-02 13:23:17: AXSelectedTextRange - table: 0x6000031eb400
2020-11-02 13:23:17: AXNumberOfCharacters - 27
To get the solution I have actually beyond a proof of concept, some extra things might need to be done:
- cache the pixel color as long as we stay inside the field, so we don't take a screenshot 60x/second (I think fields have an
AXIdentifier
we could use for uniqueness? not sure.)- disable the block cursor follow when we're in a field that doesn't have accessibility
- only enable it when the various Vim modes are enabled
All good points.
I'm not sure actually. I guess all I know is that I intend for it to run every 1/60th of a second, but it could actually be taking longer than 1 / 60 = 16.66ms to run the function. I guess I'd actually need to time it.
I ran a few tests, and it obviously depends on how big the FocusedUIElement is: from 300 snap/sec (small text area) to 1/sec (large text area)
hs.hotkey.bind({}, 'd', function()
local systemElement = ax.systemWideElement()
local currentElement = systemElement:attributeValue("AXFocusedUIElement")
start = hs.timer.absoluteTime()
nbSnapShot = 0
-- how many loops in 1 sec?
while hs.timer.absoluteTime() - start < 1E9 do
local screenshotOfTextInput = hs.screen.mainScreen():snapshot(currentElement:attributeValue("AXFrame"))
local color = screenshotOfTextInput:colorAt({ x = 10, y = 10 })
nbSnapShot++
end
hs.printf("%s snapshots", nbSnapShot)
end)
I also noticed snapshots are 4 times larger (supposedly through the retina effect). For example my console's width is 550px but its snapshot is 2250px.
screenshotOfTextInput:saveToFile("~/Desktop/snapshot.jpg")
If we take a smaller portion of the UIElement, it get faster:
local screenshotOfTextInput = hs.screen.mainScreen():snapshot({ x = 0, y = 0, w = 10, h = 10})
but caching the color as you said is even better.
Ah thanks so much for doing the timing! That's super helpful.
I think taking a smaller portion screenshot is the way to go then!
edit: The other possibility is to just leave the cursor blinking for now.
I think the next step I'll take is to get this cursor functionality behind a beta config flag and merge it to master
.
First priority probably should be knocking down the issues you found in the UTF8 thread though, and wrapping that up – that work I think is higher impact than the cosmetic cursor in here (as cool as it is!)
Ack, if we do this screen capture thing, Hammerspoon needs to ask the user for screen recording permissions. This like it would generate a lot of FUD from the user around security, and might be a bridge too far.
First priority probably should be knocking down the issues you found in the UTF8 thread though, and wrapping that up – that work I think is higher impact than the cosmetic cursor in here (as cool as it is!)
I totally rally your point.
The cursor thing was a detour while discussing the cosmetic frustration when reaching a character via F
or T
.
I would never have imagined it would bring us this far.
Ack, if we do this screen capture thing, Hammerspoon needs to ask the user for screen recording permissions. This like it would generate a lot of FUD from the user around security, and might be a bridge too far.
Yes I got this same permission request. After seeing your trick in action, about taking screenshots silently in the background, and being that fast, I was thinking about spyware: imagine how easy it would be to spy someone's screen without him knowing. So I guess we can't blame the OS for requesting such permission. It's not shocking to me. But clearly, taking screenshot to hide the cursor is a mind-blowing, "out-of-the-normal" trick!
@yoricfr I added the cursor overlay behind a beta flag in master
(also this is the first feature to get a beta flag).
If you get the latest master
, you can add vim:enableBetaFeature('block_cursor_overlay')
to your init.lua
to turn it on.
A few things I notice so far:
<textarea>
(like this one)edit: Chrome doesn't seem to support AXBoundsForRange
very well. https://groups.google.com/a/chromium.org/g/chromium-accessibility/c/eB34iqVFAu8
Using the Accessibility API reminds me of writing cross-browser JS, but it's even less consistent somehow.
@yoricfr How is it feeling with the beta for you?
@yoricfr ping again?
have you considered selecting one character for the block cursor vibe?
https://user-images.githubusercontent.com/121373/119135525-1f0dc900-ba71-11eb-8847-401d1cf312a7.mp4
works pretty well in my case. some issues depending on the weather, but probably related to the hardcore Accessibility API.
btw the video is a bit laggy but in the real world it's flawless, on a 10 year old iMac.
(P.S.: and yeah the up is not implemented yet :D AX Strategy is much harder than expected if you want to do it flawlessly. many edge cases. and not even talking about handling smileys.)
😬️
Oh sorry, what did you need @godbout ?
ah, it's me. sorry! i was just showcasing what i was able to achieve, regarding my earlier comment about selecting one character as a block cursor. apologies if that ended up being some noise to you.
have you considered selecting one character for the block cursor vibe?
Screen.Recording.2021-05-21.at.20.12.17.mp4
works pretty well in my case. some issues depending on the weather, but probably related to the hardcore Accessibility API.
From #62, there is a fair amount of discussion from @yoricfr about cursor blocks.
This might be possible to do, but we'd need:
UITextView
https://stackoverflow.com/questions/36311582/caret-cursor-color-and-size http://programming.jugglershu.net/wp/?p=765