search

Clozure / ccl

Clozure Common Lisp
http://ccl.clozure.com
Apache License 2.0
853 stars 103 forks source link

%set-macptr-type doesn't work on some MACPTRs #299

Open GOFAI opened 4 years ago

GOFAI commented 4 years ago

Trying to get the new ffigen5 headers to work under CCL 1.12 I discovered the reason for the failure when adding a modeline to the listener window: an ObjC object is not getting tagged for its correct class. 

1 > (slot-value gui::*last-document-created* 'gui::textstorage)
#<A Foreign Pointer #x149E830>

I initially thought that the hemlock-text-storage object that should be in this slot wasn't getting initialized properly, but apparently it is in fact a pointer to the ObjC object it should be:

1 > (#_object_getClass (slot-value gui::*last-document-created* 'gui::textstorage))
#<OBJC:OBJC-CLASS GUI::HEMLOCK-TEXT-STORAGE (#x1403690)>

And the functions from the bridge correctly identify it as an instance of this class:

1 > (ccl::objc-instance-p (slot-value gui::*last-document-created* 'gui::textstorage))
#<OBJC:OBJC-CLASS GUI::HEMLOCK-TEXT-STORAGE (#x1403690)>
1 > (ccl::recognize-objc-object (slot-value gui::*last-document-created* 'gui::textstorage))
432

But the MACPTR isn't tagged correctly for its ObjC class:

1 > (ccl::%macptr-type (slot-value gui::*last-document-created* 'gui::textstorage))
0

And this seems to be because for whatever reason %set-macptr-type simply doesn't work on this particular MACPTR:

1 > (ccl::%set-macptr-type (slot-value gui::*last-document-created* 'gui::textstorage) 432)
432
1 > (ccl::%macptr-type (slot-value gui::*last-document-created* 'gui::textstorage))
0
svspire commented 4 years ago

I'm pretty sure #'tagged-objc-instance-p is incorrect because it's not paying attention to the extended tag bits. See for reference https://blog.timac.org/2016/1124-testing-if-an-arbitrary-pointer-is-a-valid-objective-c-object/ and https://opensource.apple.com/source/objc4/objc4-781/runtime/objc-internal.h.auto.html

Here's my rewrite of #'tagged-objc-instance-p to take into account the extended bits.

(defconstant _OBJC_TAG_EXT_INDEX_MASK #xFF)
(defconstant _OBJC_TAG_EXT_INDEX_SHIFT 4)
(defconstant OBJC_TAG_First52BitPayload  8)
(defun tagged-objc-instance-p (p)
  (when ccl::*objc-runtime-uses-tags*
    (let* ((rawaddr (%ptr-to-int p))
           (basic-tag (logand rawaddr #xf)))
      (declare (ccl::natural rawaddr)
               (fixnum basic-tag))
      (when (logbitp 0 basic-tag) ; we know there is a tag. Now to determine what kind..
        (if (= basic-tag #xF)
            (let ((ext-tag (logand _OBJC_TAG_EXT_INDEX_MASK (ash rawaddr (- _OBJC_TAG_EXT_INDEX_SHIFT)))))
              (declare (fixnum ext-tag))
              (+ ext-tag OBJC_TAG_First52BitPayload))
            basic-tag)))))

The above definition seems to work for rebuilding the CCL IDE and running it on the same machine (i.e. it doesn't make the situation worse). I've tried using that new definition to rebuild CCL IDE on both El Capitan and Catalina. (I don't have a Mojave machine.) But when I build on El Capitan and run on Catalina I get AltConsole errors like this:


> Error: Objective-C runtime exception: 
>        -[__NSSingleEntryDictionaryI _setViewsNeedUpdateConstraints:]: unrecognized selector sent to instance 0x60000028d480
> While executing: #<Anonymous Function #x3000011EE06F>, in process Initial(0).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
1 > :b
 (647CA0) : 0 (FUNCALL #'#<Anonymous Function #x3000011EE06F> #<MODELINE-VIEW <ModelineView: 0x5609370> (#x5609370)> #S(CCL::OBJC-SELECTOR :NAME "setAutoresizingMask:" :%SEL #<A Foreign Pointer #x7FFF726E2F3E>) 2) 277
 (647CC8) : 1 (FUNCALL #'#<(:OBJC-DISPATCH NEXTSTEP-FUNCTIONS:|setAutoresizingMask:|)> ???) 941
 (647D00) : 2 (MAKE-SCROLLING-TEXTVIEW-FOR-PANE #<TEXT-PANE <TextPane: 0x109e30> (#x109E30)> #<HEMLOCK-TEXT-STORAGE HemlockTextStorage : string <HemlockBufferString for #<Hemlock Buffer "Listener">> (#x600002931E30)> T #<NS-COLOR [uninitialized] (#xFFFFFFFF0000008F)> 0) 3173
 (647D58) : 3 (TEXTPANE-FOR-TEXTSTORAGE #<OBJC:OBJC-CLASS GUI::HEMLOCK-LISTENER-FRAME (#x600000C08A80)> #<HEMLOCK-TEXT-STORAGE HemlockTextStorage : string <HemlockBufferString for #<Hemlock Buffer "Listener">> (#x600002931E30)> 80 16 T #<NS-COLOR [uninitialized] (#xFFFFFFFF0000008F)> 0) 261
 (647DB8) : 4 (%HEMLOCK-FRAME-FOR-TEXTSTORAGE #<OBJC:OBJC-CLASS GUI::HEMLOCK-LISTENER-FRAME (#x600000C08A80)> #<HEMLOCK-TEXT-STORAGE HemlockTextStorage : string <HemlockBufferString for #<Hemlock Buffer "Listener">> (#x600002931E30)> 80 16 T #<NS-COLOR [uninitialized] (#xFFFFFFFF0000008F)> 0) 101
 (647E00) : 5 (FUNCALL #'#<GUI::|-[HemlockListenerDocument makeWindowControllers]|> 17591615093644) 709
 (647E60) : 6 (%PASCAL-FUNCTIONS% 142 17591615093644) 373
 (647F18) : 7 (FUNCALL #'#<Anonymous Function #x3000011358CF> #<IDE-APPLICATION <IDEApplication: 0x600003514000> (#x600003514000)> #S(CCL::OBJC-SELECTOR :NAME "run" :%SEL #<A Foreign Pointer #x7FFF7275144E>)) 173
 (647F38) : 8 (FUNCALL #'#<(:OBJC-DISPATCH NEXTSTEP-FUNCTIONS:|run|)> ???) 557
 (647F68) : 9 (EVENT-LOOP NIL) 429
 (647FB8) : 10 (FUNCALL #'#<(:INTERNAL GUI::COCOA-STARTUP GUI::START-COCOA-IDE)>) 1541
1 >

When I build on Catalina and run on El Capitan it actually starts up the IDE about half the time. The other half of the time I get AltConsole errors like this or stack overflows where it's trying to write something to a Hemlock window:

Unhandled exception 11 at 0x2680d140, context->regs at #x7fff5fbfcd20
Exception occurred while executing foreign code
 at CFBasicHashFindBucket + 2640
received signal 11; faulting address: 0x26db7830
? for help
[50444] Clozure CL kernel debugger: :b
[50444] Clozure CL kernel debugger: current thread: tcr = 0xd006d0, native thread ID = 0x160b, interrupts enabled

(#x000000002194BF18) #x00003000011307CC : #<Anonymous Function #x000030000113071F> + 173
(#x000000002194BF38) #x000030000156C00C : #<Function (:OBJC-DISPATCH run) #x000030000156BDDF> + 557
(#x000000002194BF68) #x000030000156B8DC : #<Function EVENT-LOOP #x000030000156B72F> + 429
(#x000000002194BFB8) #x000030000157ACA4 : #<Function (:INTERNAL COCOA-STARTUP START-COCOA-IDE) #x000030000157A69F> + 1541
[50444] Clozure CL kernel debugger:

So if the above is part of the solution, it's not the complete solution. One thing that occurs to me is that if we've not been paying proper attention to all the tag bits, we might be extracting the address portion of the pointer wrong too. However, in looking at #'%safe-get-objc-class, it looks like we're calling #_object_getClass to follow the pointer, so I'd assume that's safe since it's an ObjC method. Are there other places in CCL where we're making assumptions about the address field of a tagged pointer?

GOFAI commented 4 years ago

Issues arising from tagged ObjC instances were my first guess too, and I concur that the parts of the bridge that deal with them need to be updated to address the new stuff that Apple has put into the ObjC runtime. (I'm kind of amazed that the ObjC bridge still mostly works, considering that Apple has done things like change the prototype of objc_msgSend.) But after a bit of experiment I concluded that they weren't the cause of my particular issue. In particular, objc-instance-p returns the correct class for the instance, which I believe wouldn't happen if the issue resulted from failure to recognize a tagged ObjC instance correctly. Also, (%inc-ptr (slot-value gui::*last-document-created* 'gui::textstorage) 0) returns a correctly-tagged MACPTR for the ObjC instance, which shouldn't happen if the existing code can't identify the instance's class.

In any case, %set-macptr-type should be changing the return value of %macptr-type for the object, even if that value isn't the correct one for the ObjC class. I attempted changing the value to incorrect, but definitely legal type identifiers like 1 and 2, but %macptr-type still returned 0. I'm mystified as to what could be causing this--my read of the source for %set-macptr-type and %macptr-type is that they should always change the MACPTR's type. It's really peculiar--clearly they work correctly on nearly all MACPTRs or CCL wouldn't run at all, but this issue arises specifically and consistently for me on one particular MACPTR as the IDE is starting.

svspire commented 4 years ago

Can you reduce this to a test case so I can try to reproduce the failure of %set-macptr-type?

GOFAI commented 4 years ago

The issues you're reporting seem to be somewhat different from mine--probably because I was using experimental Cocoa headers generated from the 10.14 SDK using ffigen5. If you're so inclined, I'd be very interested in discovering if the ffigen5 headers do the same thing I describe above under El Capitan. Unfortunately it doesn't seem to reduce to a straightforward test case, as I haven't come up with another way to get at a MACPTR that doesn't tag correctly. It's admittedly a bit involved, but I can provide detailed instructions on how to reproduce my experimental setup as closely as possible.