Unable to rotate/transform masks in certain situations (hitTest fail)

[chadrolfs]

create a group of two shapes create a shape outside the group select the group and the shape outside the group Layer > Create Vector Mask from Layer Rotate the masked group repeatedly

Result: eventually it will fail

[chadrolfs]

[chadrolfs]

https://www.dropbox.com/s/36u3nlgf2lqnczu/MaskGroup.psd?dl=0

[baaygun]

Just so the problem is clear:

[chadrolfs]

Almost totally, there. There is one case where it still fails; exiting Mask Edit Mode. Open existing file select group that has mask, hit M to enter Mask Mode, double-click to drill into the mask and move anchors hit Esc x2 to exit Mask Mode then try to rotate

Result: rotate fails, no policy on group on Mask Mode exit

[baaygun]

@pineapplespatula Please take a look at the new problem. It has to do with the fact that pressing Enter calls tools.enterPathModalState and escape calls mask.handleEscapeVectorMask, but resetBorderPolicies is not called when that happens. You know that code better.