Use of SourceKitten for generating Swift interfaces of frameworks

Kacper20 commented 7 years ago

Hi @jpsim

Based on our discussion on twitter I want to tell you a bit more about my use case. I want to receive public interfaces from compiled frameworks, both Swift and Objective-C.

Doing this for frameworks Objective-C works really well when using source.request.editor.open.interface - both for the system and non-system ones. Unfortunately, does not work at all with Swift frameworks - it returns empty results. I want to recreate behavior from swift-ide-test which is described here.

It's about parsing .swiftmodule files and getting public interfaces from them. Using swift-ide-test is not the perfect solution because it's not bundled Swift toolchain. I want to create a tool based on that and shipping binary of swift-ide-test for every Swift toolchain available is not a perfect solution.

If you have some ideas - I would really appreciate hearing them :)

Best Regards, Kacper

jpsim commented 7 years ago

I'll take swift-ide-test entirely out of the equation since it doesn't use SourceKit under the hood, and instead use SOURCEKIT_LOGGING=3 while requesting the interface for a module in Xcode.

When I create a new macOS command line Xcode project with this logging enabled, and I command-click on the 'Foundation' in import Foundation, Xcode requests that module's interface and presents it in the source editor.

Looking at the logs, I found the following which seemed pretty relevant:

2017-07-19 10:36:03.335 Xcode[1474:30582] SourceKit-client: [2:request:775:22.7866] [87] {
  key.request: source.request.editor.open.interface,
  key.name: "F6DA8E2C-95B6-40FB-ABC3-FE66F519B552",
  key.compilerargs: [
    "-target",
    "x86_64-apple-macosx10.12",
    "-sdk",
    "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk",
    "-I",
    "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/local/include",
    "-F",
    "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/PrivateFrameworks",
    ""
  ],
  key.modulename: "Foundation",
  key.toolchains: [
    "com.apple.dt.toolchain.XcodeDefault"
  ],
  key.synthesizedextensions: 1
}

So I reformatted it into YAML and saved it to a file named request.yml:

key.request: source.request.editor.open.interface
key.name: "F6DA8E2C-95B6-40FB-ABC3-FE66F519B552"
key.compilerargs:
  - "-target"
  - "x86_64-apple-macosx10.12"
  - "-sdk"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk"
  - "-I"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/local/include"
  - "-F"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/PrivateFrameworks"
  - ""
key.modulename: "Foundation"
key.toolchains:
  - "com.apple.dt.toolchain.XcodeDefault"
key.synthesizedextensions: 1

Then I ran sourcekitten request --yaml request.yml and got the generated module interface which you wanted, and a whole lot more information about the tokenization, structure and syntax highlighting information for the text. Here's the (heavily truncated) output to give you an idea:

{
  "key.annotations" : [
    {
      "key.kind" : "source.lang.swift.ref.module",
      "key.offset" : 7,
      "key.length" : 14,
      "key.is_system" : true
    },
    {
      "key.kind" : "source.lang.swift.ref.module",
      "key.offset" : 29,
      "key.length" : 12,
      "key.is_system" : true
    }
  ],
  "key.syntaxmap" : [
    {
      "key.kind" : "source.lang.swift.syntaxtype.keyword",
      "key.offset" : 0,
      "key.length" : 6
    },
    {
      "key.kind" : "source.lang.swift.syntaxtype.identifier",
      "key.offset" : 7,
      "key.length" : 14
    }
  ],
  "key.substructure" : [
    {
      "key.namelength" : 11,
      "key.nameoffset" : 5487,
      "key.length" : 25,
      "key.name" : "AnyHashable",
      "key.bodyoffset" : 5500,
      "key.bodylength" : 1,
      "key.kind" : "source.lang.swift.decl.extension",
      "key.offset" : 5477
    },
    {
      "key.namelength" : 14,
      "key.nameoffset" : 5514,
      "key.length" : 28,
      "key.name" : "POSIXErrorCode",
      "key.bodyoffset" : 5530,
      "key.bodylength" : 1,
      "key.kind" : "source.lang.swift.decl.extension",
      "key.offset" : 5504
    }
  ],
  "key.sourcetext" : "import CoreFoundation\nimport CoreGraphics\nimport Darwin.uuid\nimport Darwin\n..."
}

To match the behavior from the swift-ide-test command exactly, where you just want the key.sourcetext string, you can use jq:

$ sourcekitten request --yaml req.yml | jq -r '.["key.sourcetext"]'

Which outputs just the Swift interface:

import CoreFoundation
import CoreGraphics
import Darwin.uuid
/* many more imports, truncated for brevity */

extension AnyHashable {
}

extension POSIXErrorCode {
}

extension Bool {

    public init(_ number: NSNumber)
}

extension UInt16 {

    public init(_ number: NSNumber)
}

extension NSEnumerator : Sequence {

    /// Return an *iterator* over the *enumerator*.
    ///
    /// - Complexity: O(1).
    public func makeIterator() -> NSFastEnumerationIterator
}

extension NSArray : CustomReflectable {

    /// The custom mirror for this instance.
    ///
    /// If this type has value semantics, the mirror should be unaffected by
    /// subsequent mutations of the instance.
    public var customMirror: Mirror { get }
}

/* about 8,000 more lines of generated interface */

Now this is just a quick example demonstrating how this could/should be done. In practice, you'd probably want to use SourceKittenFramework to construct a Request programmatically rather than using a hardcoded YAML file, so you could dynamically pull in the appropriate SDK, module, module name, additional compiler arguments if necessary, etc.

Here's a minimal Swift CLI app that would do this programmatically:

import Foundation
import SourceKittenFramework

let compilerArguments = [
  "-target",
  "x86_64-apple-macosx10.12",
  "-sdk",
  "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk",
  "-I",
  "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/local/include",
  "-F",
  "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/PrivateFrameworks",
  ""
]
var skCompilerArguments = compilerArguments.map { sourcekitd_request_string_create($0) }

let toolchains = ["com.apple.dt.toolchain.XcodeDefault"]
var skToolchains = toolchains.map { sourcekitd_request_string_create($0) }

let dict = [
  sourcekitd_uid_get_from_cstr("key.request"): sourcekitd_request_uid_create(sourcekitd_uid_get_from_cstr("source.request.editor.open.interface")),
  sourcekitd_uid_get_from_cstr("key.name"): sourcekitd_request_string_create(UUID().uuidString),
  sourcekitd_uid_get_from_cstr("key.compilerargs"): sourcekitd_request_array_create(&skCompilerArguments, skCompilerArguments.count),
  sourcekitd_uid_get_from_cstr("key.modulename"): sourcekitd_request_string_create("Foundation"),
  sourcekitd_uid_get_from_cstr("key.toolchains"): sourcekitd_request_array_create(&skToolchains, skToolchains.count),
  sourcekitd_uid_get_from_cstr("key.synthesizedextensions"): sourcekitd_request_int64_create(1)
]

var keys = Array(dict.keys.map({ $0 as sourcekitd_uid_t? }))
var values = Array(dict.values)
let skRequest = sourcekitd_request_dictionary_create(&keys, &values, dict.count)!

let request = Request.customRequest(request: skRequest)
print(request.send()["key.sourcetext"]!)

_Note that I had to make all the internal declarations from library_wrapper_sourcekitd.swift public in order to access them from outside the framework._

I'm sure it'd be useful to package this up as a SourceKitten CLI command, if you're inclined to make a PR.

Kacper20 commented 7 years ago

Thanks, @jpsim! Actually, I've done the same thing earlier today for Foundation and it works as intended. I've tried the same thing for other Objective-C based frameworks.

I like a lot your approach of looking at logs. Was getting to it through lots of tries and crashes, unfortunately. I don't know why SIGILL happens so often when querying SourceKit with custom yaml files.

The problem I want to solve now is that this approach does not work for Swift frameworks. Doing the same thing on Alamofire, or SourceKitten results in nothing. I will now use your approach with logging and try to find how to make it work for pure Swift ones. Thank you again :)

I will work on PR to SourceKitten to make it a custom Command :)

jpsim commented 7 years ago

The exact same approach works with 3rd party Swift frameworks:

$ pwd
/Users/jp/Downloads
$ curl -O -L https://github.com/jpsim/SourceKitten/releases/download/0.18.0/SourceKittenFramework.framework.zip
$ unzip SourceKittenFramework.framework.zip
$ cat req.yml
key.request: source.request.editor.open.interface
key.name: "5F63C5B8-6D92-44FF-8012-DCA7D787D243"
key.compilerargs:
  - "-target"
  - "x86_64-apple-macosx10.12"
  - "-sdk"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk"
  - "-I"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/local/include"
  - "-F"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/PrivateFrameworks"
  - "-F"
  - "/Users/jp/Downloads/Carthage/Build/Mac"
  - ""
key.modulename: "SourceKittenFramework"
key.toolchains:
  - "com.apple.dt.toolchain.XcodeDefault"
key.synthesizedextensions: 1
$ sourcekitten request --yaml req.yml | jq -r '.["key.sourcetext"]'

Outputs the following Swift interface for SourceKittenFramework:

import Dispatch
import Foundation
import SWXMLHash
import SourceKittenFramework.CXCompilationDatabase
import SourceKittenFramework.Documentation
import SourceKittenFramework.Index
import SourceKittenFramework.sourcekitd
import SourceKittenFramework
import SourceKittenFramework.BuildSystem
import SourceKittenFramework.CXErrorCode
import SourceKittenFramework.CXString
import SourceKittenFramework.Platform
import SourceKittenFramework.Swift
import Yams

//
//  SourceKittenFramework.h
//  SourceKittenFramework
//
//  Created by JP Simard on 2015-01-02.
//  Copyright (c) 2015 SourceKitten. All rights reserved.
//

//! Project version number for SourceKittenFramework.
public var SourceKittenFrameworkVersionNumber: Double
extension CharacterSet {

    public func bridge() -> NSCharacterSet
}

extension NSString {

    /**
        Returns line number and character for utf16 based offset.

        - parameter offset: utf16 based index.
        */
    public func lineAndCharacter(forCharacterOffset offset: Int) -> (line: Int, character: Int)?

    /**
        Returns line number and character for byte offset.

        - parameter offset: byte offset.
        */
    public func lineAndCharacter(forByteOffset offset: Int) -> (line: Int, character: Int)?

    /**
        Returns a copy of `self` with the trailing contiguous characters belonging to `characterSet`
        removed.

        - parameter characterSet: Character set to check for membership.
        */
    public func trimmingTrailingCharacters(in characterSet: CharacterSet) -> String

    /**
        Returns self represented as an absolute path.

        - parameter rootDirectory: Absolute parent path if not already an absolute path.
        */
    public func absolutePathRepresentation(rootDirectory: String = default) -> String

    /**
        Converts a range of byte offsets in `self` to an `NSRange` suitable for filtering `self` as an
        `NSString`.

        - parameter start: Starting byte offset.
        - parameter length: Length of bytes to include in range.

        - returns: An equivalent `NSRange`.
        */
    public func byteRangeToNSRange(start: Int, length: Int) -> NSRange?

    /**
        Converts an `NSRange` suitable for filtering `self` as an
        `NSString` to a range of byte offsets in `self`.

        - parameter start: Starting character index in the string.
        - parameter length: Number of characters to include in range.

        - returns: An equivalent `NSRange`.
        */
    public func NSRangeToByteRange(start: Int, length: Int) -> NSRange?

    /**
        Returns a substring with the provided byte range.

        - parameter start: Starting byte offset.
        - parameter length: Length of bytes to include in range.
        */
    public func substringWithByteRange(start: Int, length: Int) -> String?

    /**
        Returns a substring starting at the beginning of `start`'s line and ending at the end of `end`'s
        line. Returns `start`'s entire line if `end` is nil.

        - parameter start: Starting byte offset.
        - parameter length: Length of bytes to include in range.
        */
    public func substringLinesWithByteRange(start: Int, length: Int) -> String?

    public func substringStartingLinesWithByteRange(start: Int, length: Int) -> String?

    /**
        Returns line numbers containing starting and ending byte offsets.

        - parameter start: Starting byte offset.
        - parameter length: Length of bytes to include in range.
        */
    public func lineRangeWithByteRange(start: Int, length: Int) -> (start: Int, end: Int)?

    /**
    Returns an array of Lines for each line in the file.
    */
    public func lines() -> [SourceKittenFramework.Line]

    /**
    Returns true if self is an Objective-C header file.
    */
    public func isObjectiveCHeaderFile() -> Bool

    /**
    Returns true if self is a Swift file.
    */
    public func isSwiftFile() -> Bool

    /**
    Returns a substring from a start and end SourceLocation.
    */
    public func substringWithSourceRange(start: SourceKittenFramework.SourceLocation, end: SourceKittenFramework.SourceLocation) -> String?
}

extension NSString {

    public func bridge() -> String
}

extension Bool : SourceKitRepresentable {
}

extension Array : SourceKitRepresentable {
}

extension Array {

    public func bridge() -> NSArray
}

extension String {

    /// Returns the `#pragma mark`s in the string.
    /// Just the content; no leading dashes or leading `#pragma mark`.
    public func pragmaMarks(filename: String, excludeRanges: [NSRange], limit: NSRange?) -> [SourceKittenFramework.SourceDeclaration]

    /**
        Returns whether or not the `token` can be documented. Either because it is a
        `SyntaxKind.Identifier` or because it is a function treated as a `SyntaxKind.Keyword`:

        - `subscript`
        - `init`
        - `deinit`

        - parameter token: Token to process.
        */
    public func isTokenDocumentable(token: SourceKittenFramework.SyntaxToken) -> Bool

    /**
        Find integer offsets of documented Swift tokens in self.

        - parameter syntaxMap: Syntax Map returned from SourceKit editor.open request.

        - returns: Array of documented token offsets.
        */
    public func documentedTokenOffsets(syntaxMap: SourceKittenFramework.SyntaxMap) -> [Int]

    /**
        Returns the body of the comment if the string is a comment.

        - parameter range: Range to restrict the search for a comment body.
        */
    public func commentBody(range: NSRange? = default) -> String?

    /// Returns a copy of `self` with the leading whitespace common in each line removed.
    public func removingCommonLeadingWhitespaceFromLines() -> String

    /**
        Returns the number of contiguous characters at the start of `self` belonging to `characterSet`.

        - parameter characterSet: Character set to check for membership.
        */
    public func countOfLeadingCharacters(in characterSet: CharacterSet) -> Int
}

extension String : SourceKitRepresentable {
}

extension String {

    public func bridge() -> NSString
}

extension Dictionary : SourceKitRepresentable where Key : Hashable {
}

extension CXString : CustomStringConvertible {

    /// A textual representation of this instance.
    ///
    /// Instead of accessing this property directly, convert an instance of any
    /// type to a string by using the `String(describing:)` initializer. For
    /// example:
    ///
    ///     struct Point: CustomStringConvertible {
    ///         let x: Int, y: Int
    ///
    ///         var description: String {
    ///             return "(\(x), \(y))"
    ///         }
    ///     }
    ///
    ///     let p = Point(x: 21, y: 30)
    ///     let s = String(describing: p)
    ///     print(s)
    ///     // Prints "(21, 30)"
    ///
    /// The conversion of `p` to a string in the assignment to `s` uses the
    /// `Point` type's `description` property.
    public var description: String { get }
}

extension Int64 : SourceKitRepresentable {
}

extension Dictionary where Key : Hashable {

    public func bridge() -> NSDictionary
}

/// A [strict total order](http://en.wikipedia.org/wiki/Total_order#Strict_total_order)
/// over instances of `Self`.
public func <(lhs: SourceKittenFramework.SourceDeclaration, rhs: SourceKittenFramework.SourceDeclaration) -> Bool

/// A [strict total order](http://en.wikipedia.org/wiki/Total_order#Strict_total_order)
/// over instances of `Self`.
public func <(lhs: SourceKittenFramework.SourceLocation, rhs: SourceKittenFramework.SourceLocation) -> Bool

public func ==(lhs: SourceKittenFramework.SourceDeclaration, rhs: SourceKittenFramework.SourceDeclaration) -> Bool

public func ==(lhs: SourceKittenFramework.SourceLocation, rhs: SourceKittenFramework.SourceLocation) -> Bool

/**
Returns true if `lhs` Structure is equal to `rhs` Structure.

- parameter lhs: Structure to compare to `rhs`.
- parameter rhs: Structure to compare to `lhs`.

- returns: True if `lhs` Structure is equal to `rhs` Structure.
*/
public func ==(lhs: SourceKittenFramework.Structure, rhs: SourceKittenFramework.Structure) -> Bool

/**
Returns true if `lhs` SyntaxMap is equal to `rhs` SyntaxMap.

- parameter lhs: SyntaxMap to compare to `rhs`.
- parameter rhs: SyntaxMap to compare to `lhs`.

- returns: True if `lhs` SyntaxMap is equal to `rhs` SyntaxMap.
*/
public func ==(lhs: SourceKittenFramework.SyntaxMap, rhs: SourceKittenFramework.SyntaxMap) -> Bool

/**
Returns true if `lhs` SyntaxToken is equal to `rhs` SyntaxToken.

- parameter lhs: SyntaxToken to compare to `rhs`.
- parameter rhs: SyntaxToken to compare to `lhs`.

- returns: True if `lhs` SyntaxToken is equal to `rhs` SyntaxToken.
*/
public func ==(lhs: SourceKittenFramework.SyntaxToken, rhs: SourceKittenFramework.SyntaxToken) -> Bool

public struct ClangAvailability {

    public let alwaysDeprecated: Bool

    public let alwaysUnavailable: Bool

    public let deprecationMessage: String?

    public let unavailableMessage: String?
}

/// Represents a group of CXTranslationUnits.
public struct ClangTranslationUnit {

    public let declarations: [String : [SourceKittenFramework.SourceDeclaration]]

    /**
        Create a ClangTranslationUnit by passing Objective-C header files and clang compiler arguments.

        - parameter headerFiles:       Objective-C header files to document.
        - parameter compilerArguments: Clang compiler arguments.
        */
    public init(headerFiles: [String], compilerArguments: [String])

    /**
        Failable initializer to create a ClangTranslationUnit by passing Objective-C header files and
        `xcodebuild` arguments. Optionally pass in a `path`.

        - parameter headerFiles:         Objective-C header files to document.
        - parameter xcodeBuildArguments: The arguments necessary pass in to `xcodebuild` to link these header files.
        - parameter path:                Path to run `xcodebuild` from. Uses current path by default.
        */
    public init?(headerFiles: [String], xcodeBuildArguments: [String], inPath path: String = default)
}

extension ClangTranslationUnit : CustomStringConvertible {

    /// A textual JSON representation of `ClangTranslationUnit`.
    public var description: String { get }
}

public struct CodeCompletionItem : CustomStringConvertible {

    public typealias NumBytesInt = Int64

    public let kind: String

    public let context: String

    public let name: String?

    public let descriptionKey: String?

    public let sourcetext: String?

    public let typeName: String?

    public let moduleName: String?

    public let docBrief: String?

    public let associatedUSRs: String?

    public let numBytesToErase: SourceKittenFramework.CodeCompletionItem.NumBytesInt?

    /// Dictionary representation of CodeCompletionItem. Useful for NSJSONSerialization.
    public var dictionaryValue: [String : Any] { get }

    /// A textual representation of this instance.
    ///
    /// Instead of accessing this property directly, convert an instance of any
    /// type to a string by using the `String(describing:)` initializer. For
    /// example:
    ///
    ///     struct Point: CustomStringConvertible {
    ///         let x: Int, y: Int
    ///
    ///         var description: String {
    ///             return "(\(x), \(y))"
    ///         }
    ///     }
    ///
    ///     let p = Point(x: 21, y: 30)
    ///     let s = String(describing: p)
    ///     print(s)
    ///     // Prints "(21, 30)"
    ///
    /// The conversion of `p` to a string in the assignment to `s` uses the
    /// `Point` type's `description` property.
    public var description: String { get }

    public static func parse(response: [String : SourceKitRepresentable]) -> [SourceKittenFramework.CodeCompletionItem]
}

public struct Documentation {

    public let parameters: [SourceKittenFramework.Parameter]

    public let returnDiscussion: [SourceKittenFramework.Text]
}

/// Represents a source file.
final public class File {

    /// File path. Nil if initialized directly with `File(contents:)`.
    public let path: String?

    /// File contents.
    public var contents: String

    /// File lines.
    public var lines: [SourceKittenFramework.Line]

    /**
        Failable initializer by path. Fails if file contents could not be read as a UTF8 string.

        - parameter path: File path.
        */
    public init?(path: String)

    /**
        Initializer by file contents. File path is nil.

        - parameter contents: File contents.
        */
    public init(contents: String)

    /**
     Formats the file.
     */
    public func format(trimmingTrailingWhitespace: Bool, useTabs: Bool, indentWidth: Int) -> String

    /**
        Parse source declaration string from SourceKit dictionary.

        - parameter dictionary: SourceKit dictionary to extract declaration from.

        - returns: Source declaration if successfully parsed.
        */
    public func parseDeclaration(_ dictionary: [String : SourceKitRepresentable]) -> String?

    /**
    Parse line numbers containing the declaration's implementation from SourceKit dictionary.

    - parameter dictionary: SourceKit dictionary to extract declaration from.

    - returns: Line numbers containing the declaration's implementation.
    */
    public func parseScopeRange(_ dictionary: [String : SourceKitRepresentable]) -> (start: Int, end: Int)?

    /**
        Returns a copy of the input dictionary with comment mark names, cursor.info information and
        parsed declarations for the top-level of the input dictionary and its substructures.

        - parameter dictionary:        Dictionary to process.
        - parameter cursorInfoRequest: Cursor.Info request to get declaration information.
        */
    public func process(dictionary: [String : SourceKitRepresentable], cursorInfoRequest: sourcekitd_object_t? = default, syntaxMap: SourceKittenFramework.SyntaxMap? = default) -> [String : SourceKitRepresentable]
}

/// File methods to generate and manipulate OffsetMap's.
extension File {

    /**
        Creates an OffsetMap containing offset locations at which there are declarations that likely
        have documentation comments, but haven't been documented by SourceKitten yet.

        - parameter documentedTokenOffsets: Offsets where there are declarations that likely
                                            have documentation comments.
        - parameter dictionary:             Docs dictionary to check for which offsets are already
                                            documented.

        - returns: OffsetMap containing offset locations at which there are declarations that likely
                   have documentation comments, but haven't been documented by SourceKitten yet.
        */
    public func makeOffsetMap(documentedTokenOffsets: [Int], dictionary: [String : SourceKitRepresentable]) -> OffsetMap
}

/// Language Enum.
public enum Language {

    /// Swift.
    case swift

    /// Objective-C.
    case objc
}

extension Language {
}

/// Representation of line in String
public struct Line {

    /// origin = 0
    public let index: Int

    /// Content
    public let content: String

    /// UTF16 based range in entire String. Equivalent to Range<UTF16Index>
    public let range: NSRange

    /// Byte based range in entire String. Equivalent to Range<UTF8Index>
    public let byteRange: NSRange
}

/// Represents source module to be documented.
public struct Module {

    /// Module Name.
    public let name: String

    /// Compiler arguments required by SourceKit to process the source files in this Module.
    public let compilerArguments: [String]

    /// Source files to be documented in this Module.
    public let sourceFiles: [String]

    /// Documentation for this Module. Typically expensive computed property.
    public var docs: [SourceKittenFramework.SwiftDocs] { get }

    public init?(spmName: String)

    /**
        Failable initializer to create a Module by the arguments necessary pass in to `xcodebuild` to build it.
        Optionally pass in a `moduleName` and `path`.

        - parameter xcodeBuildArguments: The arguments necessary pass in to `xcodebuild` to build this Module.
        - parameter name:                Module name. Will be parsed from `xcodebuild` output if nil.
        - parameter path:                Path to run `xcodebuild` from. Uses current path by default.
        */
    public init?(xcodeBuildArguments: [String], name: String? = default, inPath path: String = default)

    /**
        Initializer to create a Module by name and compiler arguments.

        - parameter name:              Module name.
        - parameter compilerArguments: Compiler arguments required by SourceKit to process the source files in this Module.
        */
    public init(name: String, compilerArguments: [String])
}

extension Module : CustomStringConvertible {

    /// A textual representation of `Module`.
    public var description: String { get }
}

/**
Objective-C declaration kinds.
More or less equivalent to `SwiftDeclarationKind`, but with made up values because there's no such
thing as SourceKit for Objective-C.
*/
public enum ObjCDeclarationKind : String {

    /// `category`.
    case category

    /// `class`.
    case `class`

    /// `constant`.
    case constant

    /// `enum`.
    case `enum`

    /// `enumcase`.
    case enumcase

    /// `initializer`.
    case initializer

    /// `method.class`.
    case methodClass

    /// `method.instance`.
    case methodInstance

    /// `property`.
    case property

    /// `protocol`.
    case `protocol`

    /// `typedef`.
    case typedef

    /// `function`.
    case function

    /// `mark`.
    case mark

    /// `struct`
    case `struct`

    /// `field`
    case field

    /// `ivar`
    case ivar

    /// `ModuleImport`
    case moduleImport

    /// `UnexposedDecl`
    case unexposedDecl

    public init(_ cursorKind: CXCursorKind)
}

extension ObjCDeclarationKind {
}

/// Type that maps potentially documented declaration offsets to its closest parent offset.
public typealias OffsetMap = [Int : Int]

public struct Parameter {

    public let name: String

    public let discussion: [SourceKittenFramework.Text]
}

/// Represents a SourceKit request.
public enum Request {

    /// An `editor.open` request for the given File.
    case editorOpen(file: SourceKittenFramework.File)

    /// A `cursorinfo` request for an offset in the given file, using the `arguments` given.
    case cursorInfo(file: String, offset: Int64, arguments: [String])

    /// A custom request by passing in the sourcekitd_object_t directly.
    case customRequest(request: sourcekitd_object_t)

    /// A request generated by sourcekit using the yaml representation.
    case yamlRequest(yaml: String)

    /// A `codecomplete` request by passing in the file name, contents, offset
    /// for which to generate code completion options and array of compiler arguments.
    case codeCompletionRequest(file: String, contents: String, offset: Int64, arguments: [String])

    /// ObjC Swift Interface
    case interface(file: String, uuid: String, arguments: [String])

    /// Find USR
    case findUSR(file: String, usr: String)

    /// Index
    case index(file: String, arguments: [String])

    /// Format
    case format(file: String, line: Int64, useTabs: Bool, indentWidth: Int64)

    /// ReplaceText
    case replaceText(file: String, offset: Int64, length: Int64, sourceText: String)

    /// A documentation request for the given source text.
    case docInfo(text: String, arguments: [String])

    /// A documentation request for the given module.
    case moduleInfo(module: String, arguments: [String])

    /**
        Sends the request to SourceKit and return the response as an [String: SourceKitRepresentable].

        - returns: SourceKit output as a dictionary.
        */
    public func send() -> [String : SourceKitRepresentable]

    /// A enum representation of SOURCEKITD_ERROR_*
    public enum Error : Error, CustomStringConvertible {

        case connectionInterrupted(String?)

        case invalid(String?)

        case failed(String?)

        case cancelled(String?)

        case unknown(String?)

        /// A textual representation of `self`.
        public var description: String { get }
    }

    /**
        Sends the request to SourceKit and return the response as an [String: SourceKitRepresentable].

        - returns: SourceKit output as a dictionary.
        - throws: Request.Error on fail ()
        */
    public func failableSend() throws -> [String : SourceKitRepresentable]
}

extension Request : CustomStringConvertible {

    /// A textual representation of `Request`.
    public var description: String { get }
}

/// Represents a source code declaration.
public struct SourceDeclaration {

    public let type: SourceKittenFramework.ObjCDeclarationKind

    public let location: SourceKittenFramework.SourceLocation

    public let extent: (start: SourceKittenFramework.SourceLocation, end: SourceKittenFramework.SourceLocation)

    public let name: String?

    public let usr: String?

    public let declaration: String?

    public let documentation: SourceKittenFramework.Documentation?

    public let commentBody: String?

    public var children: [SourceKittenFramework.SourceDeclaration]

    public let swiftDeclaration: String?

    public let availability: SourceKittenFramework.ClangAvailability?

    /// Range
    public var range: NSRange { get }

    /// Returns the USR for the auto-generated getter for this property.
    ///
    /// - warning: can only be invoked if `type == .Property`.
    public var getterUSR: String { get }

    /// Returns the USR for the auto-generated setter for this property.
    ///
    /// - warning: can only be invoked if `type == .Property`.
    public var setterUSR: String { get }
}

extension SourceDeclaration : Hashable {

    /// The hash value.
    ///
    /// Hash values are not guaranteed to be equal across different executions of
    /// your program. Do not save hash values to use during a future execution.
    public var hashValue: Int { get }
}

extension SourceDeclaration : Comparable {
}

public protocol SourceKitRepresentable {

    public func isEqualTo(_ rhs: SourceKitRepresentable) -> Bool
}

extension SourceKitRepresentable {

    public func isEqualTo(_ rhs: SourceKitRepresentable) -> Bool
}

public struct SourceLocation {

    public let file: String

    public let line: UInt32

    public let column: UInt32

    public let offset: UInt32

    public func range(toEnd end: SourceKittenFramework.SourceLocation) -> NSRange
}

extension SourceLocation : Comparable {
}

/// Swift declaration kinds.
/// Found in `strings SourceKitService | grep source.lang.swift.stmt.`.
public enum StatementKind : String, SwiftLangSyntax {

    /// `brace`.
    case brace

    /// `case`.
    case `case`

    /// `for`.
    case `for`

    /// `foreach`.
    case forEach

    /// `guard`.
    case `guard`

    /// `if`.
    case `if`

    /// `repeatewhile`.
    case repeatWhile

    /// `switch`.
    case `switch`

    /// `while`.
    case `while`
}

extension StatementKind {
}

/// Represents the structural information in a Swift source file.
public struct Structure {

    /// Structural information as an [String: SourceKitRepresentable].
    public let dictionary: [String : SourceKitRepresentable]

    /**
    Create a Structure from a SourceKit `editor.open` response.

    - parameter sourceKitResponse: SourceKit `editor.open` response.
    */
    public init(sourceKitResponse: [String : SourceKitRepresentable])

    /**
        Initialize a Structure by passing in a File.

        - parameter file: File to parse for structural information.
        */
    public init(file: SourceKittenFramework.File)
}

extension Structure : CustomStringConvertible {

    /// A textual JSON representation of `Structure`.
    public var description: String { get }
}

extension Structure : Equatable {
}

/// Swift declaration kinds.
/// Found in `strings SourceKitService | grep source.lang.swift.decl.`.
public enum SwiftDeclarationKind : String, SwiftLangSyntax {

    /// `associatedtype`.
    case `associatedtype`

    /// `class`.
    case `class`

    /// `enum`.
    case `enum`

    /// `enumcase`.
    case enumcase

    /// `enumelement`.
    case enumelement

    /// `extension`.
    case `extension`

    /// `extension.class`.
    case extensionClass

    /// `extension.enum`.
    case extensionEnum

    /// `extension.protocol`.
    case extensionProtocol

    /// `extension.struct`.
    case extensionStruct

    /// `function.accessor.address`.
    case functionAccessorAddress

    /// `function.accessor.didset`.
    case functionAccessorDidset

    /// `function.accessor.getter`.
    case functionAccessorGetter

    /// `function.accessor.mutableaddress`.
    case functionAccessorMutableaddress

    /// `function.accessor.setter`.
    case functionAccessorSetter

    /// `function.accessor.willset`.
    case functionAccessorWillset

    /// `function.constructor`.
    case functionConstructor

    /// `function.destructor`.
    case functionDestructor

    /// `function.free`.
    case functionFree

    /// `function.method.class`.
    case functionMethodClass

    /// `function.method.instance`.
    case functionMethodInstance

    /// `function.method.static`.
    case functionMethodStatic

    /// `function.operator`.
    case functionOperator

    /// `function.operator.infix`.
    case functionOperatorInfix

    /// `function.operator.postfix`.
    case functionOperatorPostfix

    /// `function.operator.prefix`.
    case functionOperatorPrefix

    /// `function.subscript`.
    case functionSubscript

    /// `generic_type_param`.
    case genericTypeParam

    /// `module`.
    case module

    /// `precedencegroup`.
    case precedenceGroup

    /// `protocol`.
    case `protocol`

    /// `struct`.
    case `struct`

    /// `typealias`.
    case `typealias`

    /// `var.class`.
    case varClass

    /// `var.global`.
    case varGlobal

    /// `var.instance`.
    case varInstance

    /// `var.local`.
    case varLocal

    /// `var.parameter`.
    case varParameter

    /// `var.static`.
    case varStatic
}

extension SwiftDeclarationKind {
}

/// SourceKit response dictionary keys.
public enum SwiftDocKey : String {

    /// Annotated declaration (String).
    case annotatedDeclaration

    /// Body length (Int64).
    case bodyLength

    /// Body offset (Int64).
    case bodyOffset

    /// Diagnostic stage (String).
    case diagnosticStage

    /// File path (String).
    case filePath

    /// Full XML docs (String).
    case fullXMLDocs

    /// Kind (String).
    case kind

    /// Length (Int64).
    case length

    /// Name (String).
    case name

    /// Name length (Int64).
    case nameLength

    /// Name offset (Int64).
    case nameOffset

    /// Offset (Int64).
    case offset

    /// Substructure ([SourceKitRepresentable]).
    case substructure

    /// Syntax map (NSData).
    case syntaxMap

    /// Type name (String).
    case typeName

    /// Inheritedtype ([SourceKitRepresentable])
    case inheritedtypes

    /// Column where the token's declaration begins (Int64).
    case docColumn

    /// Documentation comment (String).
    case documentationComment

    /// Declaration of documented token (String).
    case docDeclaration

    /// Discussion documentation of documented token ([SourceKitRepresentable]).
    case docDiscussion

    /// File where the documented token is located (String).
    case docFile

    /// Line where the token's declaration begins (Int64).
    case docLine

    /// Name of documented token (String).
    case docName

    /// Parameters of documented token ([SourceKitRepresentable]).
    case docParameters

    /// Parsed declaration (String).
    case docResultDiscussion

    /// Parsed scope start (Int64).
    case docType

    /// Parsed scope start end (Int64).
    case usr

    /// Result discussion documentation of documented token ([SourceKitRepresentable]).
    case parsedDeclaration

    /// Type of documented token (String).
    case parsedScopeEnd

    /// USR of documented token (String).
    case parsedScopeStart

    /// Swift Declaration (String).
    case swiftDeclaration

    /// Always deprecated (Bool).
    case alwaysDeprecated

    /// Always unavailable (Bool).
    case alwaysUnavailable

    /// Always deprecated (String).
    case deprecationMessage

    /// Always unavailable (String).
    case unavailableMessage
}

extension SwiftDocKey {
}

/// Represents docs for a Swift file.
public struct SwiftDocs {

    /// Documented File.
    public let file: SourceKittenFramework.File

    /// Docs information as an [String: SourceKitRepresentable].
    public let docsDictionary: [String : SourceKitRepresentable]

    /**
        Create docs for the specified Swift file and compiler arguments.

        - parameter file:      Swift file to document.
        - parameter arguments: compiler arguments to pass to SourceKit.
        */
    public init?(file: SourceKittenFramework.File, arguments: [String])

    /**
        Create docs for the specified Swift file, editor.open SourceKit response and cursor info request.

        - parameter file:              Swift file to document.
        - parameter dictionary:        editor.open response from SourceKit.
        - parameter cursorInfoRequest: SourceKit dictionary to use to send cursorinfo request.
        */
    public init(file: SourceKittenFramework.File, dictionary: [String : SourceKitRepresentable], cursorInfoRequest: sourcekitd_object_t?)
}

extension SwiftDocs : CustomStringConvertible {

    /// A textual JSON representation of `SwiftDocs`.
    public var description: String { get }
}

public protocol SwiftLangSyntax {

    public var rawValue: String { get }
}

/// Syntax kind values.
/// Found in `strings SourceKitService | grep source.lang.swift.syntaxtype.`.
public enum SyntaxKind : String, SwiftLangSyntax {

    /// `argument`.
    case argument

    /// `attribute.builtin`.
    case attributeBuiltin

    /// `attribute.id`.
    case attributeID

    /// `buildconfig.id`.
    case buildconfigID

    /// `buildconfig.keyword`.
    case buildconfigKeyword

    /// `comment`.
    case comment

    /// `comment.mark`.
    case commentMark

    /// `comment.url`.
    case commentURL

    /// `doccomment`.
    case docComment

    /// `doccomment.field`.
    case docCommentField

    /// `identifier`.
    case identifier

    /// `keyword`.
    case keyword

    /// `number`.
    case number

    /// `objectliteral`
    case objectLiteral

    /// `parameter`.
    case parameter

    /// `placeholder`.
    case placeholder

    /// `string`.
    case string

    /// `string_interpolation_anchor`.
    case stringInterpolationAnchor

    /// `typeidentifier`.
    case typeidentifier
}

extension SyntaxKind {
}

/// Represents a Swift file's syntax information.
public struct SyntaxMap {

    /// Array of SyntaxToken's.
    public let tokens: [SourceKittenFramework.SyntaxToken]

    /**
        Create a SyntaxMap by passing in tokens directly.

        - parameter tokens: Array of SyntaxToken's.
        */
    public init(tokens: [SourceKittenFramework.SyntaxToken])

    /**
        Create a SyntaxMap by passing in NSData from a SourceKit `editor.open` response to be parsed.

        - parameter data: NSData from a SourceKit `editor.open` response
        */
    public init(data: [SourceKitRepresentable])

    /**
        Create a SyntaxMap from a SourceKit `editor.open` response.

        - parameter sourceKitResponse: SourceKit `editor.open` response.
        */
    public init(sourceKitResponse: [String : SourceKitRepresentable])

    /**
        Create a SyntaxMap from a File to be parsed.

        - parameter file: File to be parsed.
        */
    public init(file: SourceKittenFramework.File)
}

extension SyntaxMap : CustomStringConvertible {

    /// A textual JSON representation of `SyntaxMap`.
    public var description: String { get }
}

extension SyntaxMap : Equatable {
}

/// Represents a single Swift syntax token.
public struct SyntaxToken {

    /// Token type. See SyntaxKind.
    public let type: String

    /// Token offset.
    public let offset: Int

    /// Token length.
    public let length: Int

    /// Dictionary representation of SyntaxToken. Useful for NSJSONSerialization.
    public var dictionaryValue: [String : Any] { get }

    /**
        Create a SyntaxToken by directly passing in its property values.

        - parameter type:   Token type. See SyntaxKind.
        - parameter offset: Token offset.
        - parameter length: Token length.
        */
    public init(type: String, offset: Int, length: Int)
}

extension SyntaxToken : Equatable {
}

extension SyntaxToken : CustomStringConvertible {

    /// A textual JSON representation of `SyntaxToken`.
    public var description: String { get }
}

public enum Text {

    case para(String, String?)

    case verbatim(String)
}

public struct Version {

    public let value: String

    public static let current: SourceKittenFramework.Version
}

public func declarationsToJSON(_ decl: [String : [SourceKittenFramework.SourceDeclaration]]) -> String

public func insertMarks(declarations: [SourceKittenFramework.SourceDeclaration], limit: NSRange? = default) -> [SourceKittenFramework.SourceDeclaration]

/**
Parse XML from `key.doc.full_as_xml` from `cursor.info` request.

- parameter xmlDocs: Contents of `key.doc.full_as_xml` from SourceKit.

- returns: XML parsed as an `[String: SourceKitRepresentable]`.
*/
public func parseFullXMLDocs(_ xmlDocs: String) -> [String : SourceKitRepresentable]?

/**
Extracts Objective-C header files and `xcodebuild` arguments from an array of header files followed by `xcodebuild` arguments.

- parameter sourcekittenArguments: Array of Objective-C header files followed by `xcodebuild` arguments.

- returns: Tuple of header files and xcodebuild arguments.
*/
public func parseHeaderFilesAndXcodebuildArguments(sourcekittenArguments: [String]) -> (headerFiles: [String], xcodebuildArguments: [String])

public func sdkPath() -> String

/**
 JSON Object to JSON String.

 - parameter object: Object to convert to JSON.

 - returns: JSON string representation of the input object.
 */
public func toJSON(_ object: Any) -> String

/**
 Convert [String: SourceKitRepresentable] to `NSDictionary`.

 - parameter dictionary: [String: SourceKitRepresentable] to convert.

 - returns: JSON-serializable value.
 */
public func toNSDictionary(_ dictionary: [String : SourceKitRepresentable]) -> NSDictionary

Kacper20 commented 7 years ago

Hmm... It finally worked with following exactly your steps. Earlier I was working on SourceKittenFramework.framework alone(without Yams and SWXMLHash being in the same directory) and then it didn't work. Been doing this all day with other examples, too. I believe it needs to have all of the dependent frameworks available. Out of curiosity - do you know why is it necessary to generate interfaces?

Thank you very much for help @jpsim. It saved me hours of researching, and your tips will save me next hours in the future :) 👍

Kacper20 commented 7 years ago

Tomorrow I will work on integrating this into SourceKitten. Excellent tool!

jpsim commented 7 years ago

Out of curiosity - do you know why is it necessary to generate interfaces?

The compiler can't resolve all the dependencies if they're not available, or if it doesn't know where to find them.

This interface isn't a string in the binary, it's a more complex serialization that links to external symbols, so rather than succeeding when it can only resolve a partial interface, SourceKit just fails altogether.

jpsim commented 7 years ago

Feel free to close this issue if you feel your questions have been adequately addressed.

Kacper20 commented 7 years ago

Okay, that's clear right now. Thanks :)

MedAmineWinners commented 5 years ago

Hello @jpsim hello @Kacper20 this is really what I'm looking for, But how can I get the .yaml request created by Xcode when you CMD tap the foundation import. I need to have the request I don't get how can I generate it( I want it manually not with the framework)

jpsim commented 5 years ago

Step by step instructions are in this comment: https://github.com/jpsim/SourceKitten/issues/405#issuecomment-316473679

MedAmineWinners commented 5 years ago

@jpsim , I'm following but I don't get logs when I command tap the Foundation on my Command Line Xcode new project

jpsim commented 5 years ago

Here it is, reposted.

Contents of req.yml

key.request: source.request.editor.open.interface
key.name: "F6DA8E2C-95B6-40FB-ABC3-FE66F519B552"
key.compilerargs:
  - "-target"
  - "x86_64-apple-macosx10.12"
  - "-sdk"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk"
  - "-I"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/local/include"
  - "-F"
  - "/Applications/Xcode-8.3.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/PrivateFrameworks"
  - ""
key.modulename: "Foundation"
key.toolchains:
  - "com.apple.dt.toolchain.XcodeDefault"
key.synthesizedextensions: 1

Then run

$ sourcekitten request --yaml req.yml | jq -r '.["key.sourcetext"]'

You'll need to change the paths in the yaml file to match where Xcode is installed and what macOS version you're running on.

MedAmineWinners commented 5 years ago

thank you @jpsim it works like a charm, what if I want to generate a Swift interface of a swift framework ? I tried the same thing that you did with modulename: "SourceKittenFramework", but I get sourceKittenFramework.Request.error error 2.

Actually I'm creating a swift framework and I need to make its Classes and methods declarations visible to users, like header files in obj-c. So I said that generating an interface will be probably a good idea. Am' I on the good way ?