Link against / use SourceKit (SKAgent) instead of sourcekitd on macOS

[galli-leo]

It seems like a lot of the "cool" source kit commands (such as finding subclasses, documentation for a certain foundation type, etc.) are only implemented in /Applications/Xcode.app/Contents/SharedFrameworks/SourceKit.framework not in sourcekitd.

So we could start linking against that instead (maybe conditionally with a define, so that the basic requests always work), allowing for more requests. I have started to work on this, but haven't succeeded yet. We probably need to rewrite sourcekitd.h to use sourcekit* functions instead of sourcekitd* functions. (And the swift wrapper file as well, though that already seems to work).

If you are interested in this, I can continue to try and get this to work. Tools such as Jazzy, would benefit from this, imo. For example https://github.com/realm/jazzy/issues/190 could be integrated using Xcode only requests such as source.request.workspace.symbol-protocols.

Example request:

2018-06-03 22:27:27.654 Xcode[19165:307061] SourceKit-client: [2:request:153347: 2.0229] [55] {
  key.request: source.request.workspace.symbol-protocols,
  key.symbol: {
    key.indexer.arg.symbol.name: "Test",
    key.indexer.arg.symbol.kind: "Xcode.SourceCodeSymbolKind.Class",
    key.indexer.arg.symbol.language: "Xcode.SourceCodeLanguage.Swift",
    key.indexer.arg.symbol.resolution: "s:11testingtest4TestC"
  },
  key.indexer.arg.indexer-token: 1
}
2018-06-03 22:27:27.656 Xcode[19165:307061] SourceKit-client: [2:response:153347: 2.0248] [55] {
  key.symbols: [
    {
      key.symbol: {
        key.indexer.arg.symbol.name: "CustomStringConvertible",
        key.indexer.arg.symbol.kind: "Xcode.SourceCodeSymbolKind.Protocol",
        key.indexer.arg.symbol.language: "Xcode.SourceCodeLanguage.Swift",
        key.indexer.arg.symbol.resolution: "s:s23CustomStringConvertibleP"
      },
      key.indexer.arg.occurrence.role: 2,
      key.indexer.arg.occurrence.location: {
        key.indexer.arg.doc-loc.url: "file:///Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphonesimulator/x86_64/Swift.swiftmodule",
        key.indexer.arg.doc-loc.start-line: 9223372036854775807,
        key.indexer.arg.doc-loc.start-col: 9223372036854775807,
        key.indexer.arg.doc-loc.end-line: 9223372036854775807,
        key.indexer.arg.doc-loc.end-col: 9223372036854775807,
        key.indexer.arg.doc-loc.range-loc: 9223372036854775807,
        key.indexer.arg.doc-loc.range-count: 0,
        key.indexer.arg.doc-loc.time-stamp: 5.497426e+08,
        key.indexer.arg.doc-loc.encoding: 0
      },
      key.indexer.arg.occurrence.line: 0,
      key.indexer.arg.occurrence.col: 0,
      key.indexer.arg.occurrence.file: "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphonesimulator/x86_64/Swift.swiftmodule",
      key.indexer.arg.symbol.display-name: "CustomStringConvertible",
      key.indexer.arg.symbol.is-in-project: false,
      key.indexer.arg.symbol.is-virtual: false,
      key.indexer.arg.symbol.is-system: true
    }
  ]
}

[jpsim]

If you wanted to start contributing something along these lines I'd love to see it! I'm very much in favor of exploring this. Although we might want to make new commands for stuff only available on macOS rather than differing behavior of the same command depending on the platform. I'm not 100% sure about that yet though.

Looking forward to seeing what you share!

[galli-leo]

Sounds good. I will try to get it working and throw up a wip pr if I do.

Although we might want to make new commands for stuff only available on macOS rather than differing behavior of the same command depending on the platform

With new commands, do mean commands for the command line? If yes, that's exactly what I had in mind, e.g. doc_expanded.

[galli-leo]

So, little update: I haven't yet got any code working (i.e. it crashes on sourcekit_uid_get_from_cstr). However, after disassembling the Framework itself and SourceKitSupport (uses SourceKit), it seems that sourcekit_initialize is not defined for SourceKit. Instead they use sourcekit_connection_create and sourcekit_connection_start. Additionally, all functions (e.g. sourcekit_uid_get_from_cstr) have an additional parameter with the xpc connection object. Hopefully, I can get something working tomorrow.

[galli-leo]

Hmm a question, why are we not linking directly against sourcekitd and instead are loading the methods dynamically? (Doesn't make that much of a difference, but IMO would improve readability and stability).

Actually, just looked through SourceKit.framework some more, it already provides a nice swift interface / class for interacting with the C methods. So, if I get it to link against SourceKit, we don't have to dynamically load the library and don't have implement all the methods ourselves.

Mangled names of the symbols:

_SourceKitVersionNumber, _SourceKitVersionString, __T09SourceKit10ConnectionC11sendRequestyAA0E10DictionaryC_So13DispatchQueueCSg5queueyAA8ResponseCc8receivertF, 
                       __T09SourceKit10ConnectionC11sendRequestyAA0E10DictionaryC_So13DispatchQueueCSg5queueyAA8ResponseCc8receivertFfA0_, 
                       __T09SourceKit10ConnectionC15sendRequestSyncAA17VariantDictionaryVAA0eH0CKF, 
                       __T09SourceKit10ConnectionC15sendRequestSyncAA8ResponseCAA0E10DictionaryCF, 
                       __T09SourceKit10ConnectionC16getUIDFromStrings13OpaquePointerVSSF, 
                       __T09SourceKit10ConnectionC17sendCallbackReplyyAA17RequestDictionaryCF, 
                       __T09SourceKit10ConnectionC18setCallbackHandlerySo13DispatchQueueC5queue_yAA8ResponseC_AA17RequestDictionaryCtc8receivertF, 
                       __T09SourceKit10ConnectionC22setNotificationHandlerySo13DispatchQueueC5queue_yAA8ResponseCc8receivertF, 
                       __T09SourceKit10ConnectionC5startyyF, __T09SourceKit10ConnectionC6cancelyyF, 
                       __T09SourceKit10ConnectionCACycfC, __T09SourceKit10ConnectionCACycfc, 
                       __T09SourceKit10ConnectionCMa, __T09SourceKit10ConnectionCMm, 
                       __T09SourceKit10ConnectionCMn, __T09SourceKit10ConnectionCN, 
                       __T09SourceKit10ConnectionCfD, __T09SourceKit10ConnectionCfd, 
                       __T09SourceKit12RequestArrayC6appendy10Foundation4DataVF, 
                       __T09SourceKit12RequestArrayC6appendyAA0C10DictionaryCF, __T09SourceKit12RequestArrayC6appendyACF, 
                       __T09SourceKit12RequestArrayC6appendySSF, __T09SourceKit12RequestArrayC6appendySiF, 
                       __T09SourceKit12RequestArrayC6appendys13OpaquePointerVF, __T09SourceKit12RequestArrayCACycfC, 
                       __T09SourceKit12RequestArrayCACycfc, __T09SourceKit12RequestArrayCMa, 
                       __T09SourceKit12RequestArrayCMm, __T09SourceKit12RequestArrayCMn, 
                       __T09SourceKit12RequestArrayCN, __T09SourceKit12RequestArrayCfD, 
                       __T09SourceKit12RequestArrayCfd, __T09SourceKit12VariantArrayV03getD0ACSgSi5index_tF, 
                       __T09SourceKit12VariantArrayV10startIndexSivg, __T09SourceKit12VariantArrayV11descriptionSSvg, 
                       __T09SourceKit12VariantArrayV13getDictionaryAA0cF0VSgSi5index_tF, 
                       __T09SourceKit12VariantArrayV5countSivg, __T09SourceKit12VariantArrayV5indexS2i5after_tF, 
                       __T09SourceKit12VariantArrayV5indexS2i6before_tF, __T09SourceKit12VariantArrayV6getIntS2i5index_tF, 
                       __T09SourceKit12VariantArrayV6getUIDs13OpaquePointerVSgSi5index_tF, 
                       __T09SourceKit12VariantArrayV7getBoolSbSi5index_tF, __T09SourceKit12VariantArrayV8endIndexSivg, 
                       __T09SourceKit12VariantArrayV9getDoubleSdSi5index_tF, __T09SourceKit12VariantArrayV9getStringSSSgSi5index_tF, 
                       __T09SourceKit12VariantArrayVAA0C0OSgSicig, __T09SourceKit12VariantArrayVMa, 
                       __T09SourceKit12VariantArrayVMn, __T09SourceKit12VariantArrayVN, 
                       __T09SourceKit12VariantArrayVs10CollectionAAWP, __T09SourceKit12VariantArrayVs10CollectionAAWa, 
                       __T09SourceKit12VariantArrayVs22RandomAccessCollectionAAWP, 
                       __T09SourceKit12VariantArrayVs22RandomAccessCollectionAAWa, 
                       __T09SourceKit12VariantArrayVs23BidirectionalCollectionAAWP, 
                       __T09SourceKit12VariantArrayVs23BidirectionalCollectionAAWa, 
                       __T09SourceKit12VariantArrayVs23CustomStringConvertibleAAWP, 
                       __T09SourceKit12VariantArrayVs23CustomStringConvertibleAAWa, 
                       __T09SourceKit12VariantArrayVs8SequenceAAWP, __T09SourceKit12VariantArrayVs8SequenceAAWa, 
                       __T09SourceKit13ResponseErrorO2eeoiSbAC_ACtFZ, __T09SourceKit13ResponseErrorOMa, 
                       __T09SourceKit13ResponseErrorOMn, __T09SourceKit13ResponseErrorON, 
                       __T09SourceKit13ResponseErrorOs0D0AAWP, __T09SourceKit13ResponseErrorOs0D0AAWa, 
                       __T09SourceKit13ResponseErrorOs9EquatableAAWP, __T09SourceKit13ResponseErrorOs9EquatableAAWa, 
                       __T09SourceKit16YAMLParsingErrorOMa, __T09SourceKit16YAMLParsingErrorOMn, 
                       __T09SourceKit16YAMLParsingErrorON, __T09SourceKit16YAMLParsingErrorOs0D0AAWP, 
                       __T09SourceKit16YAMLParsingErrorOs0D0AAWa, __T09SourceKit17RequestDictionaryC10Foundation4DataVs13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryC10Foundation4DataVs13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryC10Foundation4DataVs13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryC11descriptionSSvg, __T09SourceKit17RequestDictionaryC14createFromYAMLACSS_AA10ConnectionC4conntKFZ, 
                       __T09SourceKit17RequestDictionaryCAA0C5ArrayCs13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryCAA0C5ArrayCs13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryCAA0C5ArrayCs13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryCACs13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryCACs13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryCACs13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryCACycfC, __T09SourceKit17RequestDictionaryCACycfc, 
                       __T09SourceKit17RequestDictionaryCMa, __T09SourceKit17RequestDictionaryCMm, 
                       __T09SourceKit17RequestDictionaryCMn, __T09SourceKit17RequestDictionaryCN, 
                       __T09SourceKit17RequestDictionaryCSSs13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryCSSs13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryCSSs13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryCSbs13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryCSbs13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryCSbs13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryCSds13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryCSds13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryCSds13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryCSis13OpaquePointerVcig, 
                       __T09SourceKit17RequestDictionaryCSis13OpaquePointerVcim, 
                       __T09SourceKit17RequestDictionaryCSis13OpaquePointerVcis, 
                       __T09SourceKit17RequestDictionaryCfD, __T09SourceKit17RequestDictionaryCfd, 
                       __T09SourceKit17RequestDictionaryCs13OpaquePointerVAEcig, 
                       __T09SourceKit17RequestDictionaryCs13OpaquePointerVAEcim, 
                       __T09SourceKit17RequestDictionaryCs13OpaquePointerVAEcis, 
                       __T09SourceKit17RequestDictionaryCs23CustomStringConvertibleAAWP, 
                       __T09SourceKit17RequestDictionaryCs23CustomStringConvertibleAAWa, 
                       __T09SourceKit17VariantDictionaryV03getD0ACSgs13OpaquePointerV3key_tF, 
                       __T09SourceKit17VariantDictionaryV11descriptionSSvg, __T09SourceKit17VariantDictionaryV6asJSONSSvg, 
                       __T09SourceKit17VariantDictionaryV6getIntSiSgs13OpaquePointerV3key_tF, 
                       __T09SourceKit17VariantDictionaryV6getUIDs13OpaquePointerVSgAF3key_tF, 
                       __T09SourceKit17VariantDictionaryV7getBoolSbSgs13OpaquePointerV3key_tF, 
                       __T09SourceKit17VariantDictionaryV8getArrayAA0cF0VSgs13OpaquePointerV3key_tF, 
                       __T09SourceKit17VariantDictionaryV9getDoubleSdSgs13OpaquePointerV3key_tF, 
                       __T09SourceKit17VariantDictionaryV9getStringSSSgs13OpaquePointerV3key_tF, 
                       __T09SourceKit17VariantDictionaryVAA0C0OSgs13OpaquePointerVcig, 
                       __T09SourceKit17VariantDictionaryVMa, __T09SourceKit17VariantDictionaryVMn, 
                       __T09SourceKit17VariantDictionaryVN, __T09SourceKit17VariantDictionaryVs23CustomStringConvertibleAAWP, 
                       __T09SourceKit17VariantDictionaryVs23CustomStringConvertibleAAWa, 
                       __T09SourceKit7VariantOMa, __T09SourceKit7VariantOMn, __T09SourceKit7VariantON, 
                       __T09SourceKit8ResponseC11descriptionSSvg, __T09SourceKit8ResponseC7isErrorSbvg, 
                       __T09SourceKit8ResponseC8getErrorAA0cE0OSgyF, __T09SourceKit8ResponseC8getValueAA17VariantDictionaryVSgyF, 
                       __T09SourceKit8ResponseCMa, __T09SourceKit8ResponseCMm, __T09SourceKit8ResponseCMn, 
                       __T09SourceKit8ResponseCN, __T09SourceKit8ResponseCfD, __T09SourceKit8ResponseCfd, 
                       __T09SourceKit8ResponseCs23CustomStringConvertibleAAWP, __T09SourceKit8ResponseCs23CustomStringConvertibleAAWa

[norio-nomura]

why are we not linking directly against sourcekitd and instead are loading the methods dynamically?

In macOS, SourceKitten uses xcrun to find the sourcekitd to load. This behavior was introduced to use the toolchain distributed on Xcode beta or swift.org. Dynamic loading is used to realize this behavior.

[galli-leo]

@norio-nomura But that could still be achieved by using rpath, right? Put a bunch of likely paths in there and give the user the option to set LD_LIBRARY_PATH if needed. IMO it could be more flexible compared to the current option, as you can always force a certain sourcekit framework using LD_LIBRARY_PATH

[norio-nomura]

xcrun does not manipulateDYLD_LIBRARY_PATH at all. we thought that it was not realistic for each user to write scripts that set DYLD_LIBRARY_PATH based on the results ofxcrun, and I tried https://github.com/jpsim/SourceKitten/pull/172 to let SourceKitten do it. However, to manipulate DYLD_*, we had to use execv() and so on and it was an unacceptable method such as SwiftLint using SourceKittenFramework. As a way to avoid them, we adopted a method that uses dlopen() and dlsym().

I believe that controlling dlopen() directly, rather than indirectly controlling the library loading by setting DYLD_*, is better because it reduces factors to consider.

[galli-leo]

Got linking working as well as the swift wrapper classes! Xcode specific requests work now in a small test application. The swift wrapper classes however, are very unstable. They only work if you compile the test program with optimizations :/. So I am gonna throw up a pr using the c methods in a few days.

[jpsim]

I'm glad to hear you got it working (even if under limited conditions)! Hopefully with that as a starting point we can all poke at it to figure out ways to get it more robust.

[Nadohs]

@galli-leo did you ever get PR up for this? I would really like to see what you are doing to communicate with SKAgent. Thanks!