Quick Look in Xcode Debugger doesn't work

[vojtamolda]

Clicking the :eye: icon in Xcode Debugger displays very useful previews for built-in types. For example it can show a preview of an image, color, cursor, bezier path and many more types.

Here's an example:

Since about 2014 it is possible to implement custom quick look behavior for user defined data types. Here is Apple's official documentation on how to do it.

---

Unfortunately, as of Xcode 12 custom quick look isn't possible for structs. To the extent of my knowledge the debugQuickLookObject() method is called only for classes. It requires @objc decorator responsible for Objective-C bridging in order to work properly. Since TensorFlow.Tensor is a struct there's no way to implement this feature unless Apple decides to allow it and changes the implementation in Xcode.

Below is an example main.swift file that demonstrates the problem. When debugging the application QuickLookClass shows a nice rectangle view after clicking the quick look icon. Whereas QuickLookStruct doesn't show anything even tough the implementation is identical

import SwiftUI

/// A simple example view.
struct RedRectangleView: View {
    var body: some View {
       Rectangle()
        .fill()
        .foregroundColor(.red)
        .cornerRadius(5)
    }
}

/// A class with working Quick Look in Xcode Debugger.
class QuickLookClass {
    init() { }

    @objc
    func debugQuickLookObject() -> NSView {
        let window = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 20, height: 20),
                              styleMask: [.borderless], backing: .buffered, defer: false)
        window.contentView = NSHostingView(rootView: RedRectangleView())
        return window.contentView!
    }
}

/// A struct with broken Quick Look in Xcode Debugger.
struct QuickLookStruct {
    init() { }

    func debugQuickLookObject() -> NSView {
        let window = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 20, height: 20),
                              styleMask: [.borderless], backing: .buffered, defer: false)
        window.contentView = NSHostingView(rootView: RedRectangleView())
        return window.contentView!
    }
}

let qlClass = QuickLookClass()
let qlStruct = QuickLookStruct()

print(qlStruct, qlClass)

[philipturner]

If all that's required is adding another method to Tensor, I can do that pretty easily. I have been building S4TF exclusively through SPM and soon will even build it on release toolchains (this should be impossible, but I have a knack for bypassing restrictions). If you could give me an outline of precisely what I need to do (or just make a PR), I should be able to merge it into S4TF.

For reference, the new head branch of S4TF is the s4tf/s4tf repository. tensorflow/swift-apis is no longer accepting PRs, and tensorflow/swift has finally been archived. The build script is https://gist.github.com/philipturner/7aa063af04277d463c14168275878511 - only tested on macOS so far.

Warning: You will need to test this out using a development snapshot + Swift Playgrounds. I don't know if Swift Playgrounds supports such a custom build configuration as my build script. Perhaps you could temporarily bypass such a hurdle by copying the libx10.dylib binary into some build products folder. When I make the Metal and OpenCL backends in the future, you will no longer need any special build script; it will build exclusively through SwiftPM. Maybe you should wait until then to try using Swift Playgrounds.

Also, I will acknowledge that I was a bit pushy in the past; please forgive me for my past comments. If you don't want to help me out because it's not the "official" Swift for TensorFlow sponsored by Google, I completely understand that.

[vojtamolda]

Hello @philipturner ,

Adding a method to the Tensor type isn't the issue here. The code in this repo actually contains an extension to do that. The previews are not working because of how Xcode internally works. There's not way to trigger the debugQuickLook() method unless the object in question is a class. Unless we want to switch Tensor from struct to class (which I think is a bad idea), we need to change Xcode. And changing Xcode is kinda hard because it isn't open source.

But that being sad, I have never tried to open an official Radar issue with Apple. Especially, if you know somebody from the Xcode team, there maybe a chance. So let me know if you find a way, I'll be happy to submit a ticket.

---

Thanks for the acknowledgment and no worries. It takes a lot of effort to get things off the ground. I'm actually rooting for you and will help where I can.

[philipturner]

The Xcode quick look isn't what I was referencing in my comment. I was talking about the part of your README concerning a mockup in Swift Playgrounds. You made a library called "TensorFraud" that uses CustomPlaygroundDisplayConvertible. More specifically, this text gave me a lot of hope:

While the previews themselves are working well, as of 0.11 S4TF toolchain doesn't support Xcode Playgrounds. This should eventually fix itself. My understanding is that the ultimate end goal is to have TensorFlow work as a regular SPM library. The differentiation features should be available in release builds of the Swift compiler. Once this is completed, the user experience there should be great.

My near-term goal of compiling S4TF on the Swift 5.7 release toolchain would check most of those boxes. The only problem I foresee is the complicated build process with libx10.dylib. But once I finish the Metal backend, we won't need X10 anymore. Also, S4TF will run on iOS/iPadOS when the Metal backend is finished. So we need to prepare for iPad Swift Playground support.

Preferrably, I would merge all the code in this repository into s4tf/s4tf. That would make the user experience seamless, because many people might not know that the tensor-thumbnail repo exists. And, they must add a Swift package dependency to it in order to see mockups - unless tensor-thumbnail became part of TensorFlow.

[philipturner]

After playing around with Swift Playgrounds, I learned that it's impossible to add a Swift package dependency with custom build flags. I need the unsafe -parse-stdlib flag for Differentiation and ReflectionMirror. I still have one more workaround to try: pre-compile the Swift package as an Xcode framework and insert that as a binary dependency in an encapsulating Swift package.

• *It should technically possible to remove that requirement on ReflectionMirror. I would just emulate the behavior of tail allocation, which is extremely difficult to get right.
• *I could also remove the requirement on Differentiation by getting AutoDiff approved for Swift Evolution. But that's a disproportionate amount of time and effort for iPad Swift Playgrounds. Maybe I'll change my mind in a few years.

Even if I do manage to get packages with unsafe flags working on iPad, there are still caveats. First of all, packages are only supported when you're building an app target. So the script-like REPL mode won't work for native package support. However, you could try adding a package unofficially through the playground's generated Xcode project. That's how it's been done in the past anyway. That would bring support for S4TF in REPL mode, but only on macOS.

There is also one more idea. At bow-swift/nef, they have something that generates a Swift playground file, which can be run on iPad. Hopefully, I could sneak in a dependency to packages with unsafe flags, using some pre-compiled playground file. That playground would have to be downloaded from the internet before you use S4TF on the iPad.

So there are definitely challenges, but it's still a good idea to work on tensor-thumbnail. Maybe one of my proposed workarounds succeeds. Or, Apple will add more flexibility to Swift Playgrounds in the future. I'm thinking of filing a Radar and accompanying Apple Developer Forums post if nothing works for iPad Swift Playgrounds. In the meantime, iPads with M1 processors have a GPU with roughly equal ML processing power to those on Google Colab. There wouldn't be any disadvantage to using Swift-Colab instead of Playgrounds on the iPad for accessing S4TF.

[philipturner]

I just tried all the workarounds mentioned above, and no luck. It seems that I'll have to get AutoDiff officially as part of release toolchains and make an unsafe workaround for the ReflectionMirror tail-allocation builtin. And it's likely going to take years before AutoDiff becomes part of mainline. I can't integrate tensor-thumbnail into s4tf/s4tf anytime soon because there's no way to test it.

But all hope is not yet lost. I see myself being the one to push AutoDiff into Swift Evolution in the far future. That will require a massive refocusing of effort, so I must be done with the Metal and OpenCL backends. S4TF should be "finished" in my mind, if it ever can be. I don't know when that will happen, but likely years in the future.

Or perhaps someone new will join the Swift Numerical/ML Working Group and want to push AutoDiff forward.