Project-specific completion

[nathankot]

Will need to derive and send the correct flags to sourcekitten. It's best that this happens automatically simply by inspecting the .xcodeproj or .xcworkspace.

Here is what XCode sends to sourcekit when doing completion in a project:

I don't think everything is required but we should try to emulate the important bits:

  key.compilerargs: [
    "-module-name",
    "Rumuki",
    "-O",
    "-DAPI_NATHAN",
    "-DDEBUG",
    "-sdk",
    "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator9.1.sdk",
    "-target",
    "x86_64-apple-ios8.0",
    "-g",
    "-module-cache-path",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/ModuleCache",
    "-Xfrontend",
    "-serialize-debugging-options",
    "-I",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Products/Nathan-iphonesimulator",
    "-F",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Products/Nathan-iphonesimulator",
    "-F",
    "/Users/nathan/Development/Apps/rumuki/ios/Carthage/Build/iOS",
    "-c",
    "-j4",
    "/Users/nathan/Development/Apps/rumuki/ios/Rumuki/Cryptography/CryptHelpers.swift",
    "/Users/nathan/Development/Apps/rumuki/ios/Rumuki/HTTP/Api.swift",
    "/Users/nathan/Development/Apps/rumuki/ios/Rumuki/UserInterface/Components/RecordingListEmptyBackgroundComponent.swift",
    "/Users/nathan/Development/Apps/rumuki/ios/Rumuki/AudioVideo/Library/AVCNALHelpers.swift",
   "/* ... long list of source files */",
    "-emit-module",
    "-emit-module-path",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/Objects-normal/x86_64/Rumuki.swiftmodule",
    "-Xcc",
    "-I/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/swift-overrides.hmap",
    "-Xcc",
    "-iquote",
    "-Xcc",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/Rumuki-generated-files.hmap",
    "-Xcc",
    "-I/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/Rumuki-own-target-headers.hmap",
    "-Xcc",
    "-I/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/Rumuki-all-non-framework-target-headers.hmap",
    "-Xcc",
    "-ivfsoverlay",
    "-Xcc",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/all-product-headers.yaml",
    "-Xcc",
    "-iquote",
    "-Xcc",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/Rumuki-project-headers.hmap",
    "-Xcc",
    "-I/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Products/Nathan-iphonesimulator/include",
    "-Xcc",
    "-I/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/DerivedSources/x86_64",
    "-Xcc",
    "-I/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/DerivedSources",
    "-Xcc",
    "-DDEBUG=1",
    "-Xcc",
    "-DDEBUG=1",
    "-emit-objc-header",
    "-emit-objc-header-path",
    "/Users/nathan/Library/Developer/Xcode/DerivedData/Rumuki-famrqdkrtmqhvkawkbxggtwiipsp/Build/Intermediates/Rumuki.build/Nathan-iphonesimulator/Rumuki.build/Objects-normal/x86_64/Rumuki-Swift.h",
    "-import-objc-header",
    "/Users/nathan/Development/Apps/rumuki/ios/Rumuki/BridgingHeader.h",
    "-Xcc",
    "-working-directory/Users/nathan/Development/Apps/rumuki/ios"
  ]

[terhechte]

I started work on a SourceKittenDaemon that keeps the XPC running and also uses existing libraries for parsing an xcodeproj and figuring out the parameters. I'll try to add this to the project today.

[nathankot]

Sweet! Look forward to this

[terhechte]

Hey, it's not working yet, but I've uploaded what I finished so far. Sadly I had less time today to work on it than I anticipated. Nevertheless, I've made you a collaborator, you can see it here: https://github.com/terhechte/SourceKittenDaemon

All the basics are in place. It is a web server connected to sourcekitten and a xcode project parser. In theory, upon startup you'd hand it the xocde project, it would parse it, and retain the arguments. Then, for each web request from emacs (that contains the location of a file to be completed, and an offset) it would ask via XPC and return the results as JSON. Most of that is in place already, but the Xcode parsing is still in it's infancy, and the webserver doesn't understand arguments yet :)

[terhechte]

Did you have success getting sourcekitten to provide completions for more than just one Swift file? I.e. for a project consisting out of two Swift files. I've tried that in various ways today, but either got an error or empty results. I've created this ticket here:

https://github.com/jpsim/SourceKitten/issues/102

However, I may just have done something stupid and it is really simple.

[nathankot]

@terhechte I haven't tried this yet, but maybe you could try passing in arguments in the same order as xcode does? So:

sourcekitten complete --text "struct A { let u: Int } ; let aa = A(u: 5) ; aa." --offset 48 --compilerargs -- "-sdk $SDKPATH -c /path/to/main.swift /path/to/ExampleProtocols.swift"

Also, not sure what -c does as it's not listed in swift -h (I believe thats where compiler args get passed to?)

[terhechte]

I was successful. This seems to be the minim required to get back completions for multiple files:

sourcekitten complete --text "struct A:Identifiable  { let name: String ; let u: Int } ; let aa = A(name: \"carl\", u: 5) ; aa." --offset 97 
--compilerargs -- "

-module-name FitureCompletion 
-sdk /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator9.1.sdk 
/Users/terhechte/Development/Cocoa/SourceKittenDaemon/SourceKittenDaemonTests/Fixtures/FitureCompletionTests/FitureCompletionTests/ExampleProtocols.swift 
/Users/terhechte/Development/Cocoa/SourceKittenDaemon/SourceKittenDaemonTests/Fixtures/FitureCompletionTests/FitureCompletion/main.swift 
-Xcc -I/Users/terhechte/Library/Developer/Xcode/DerivedData/FitureCompletionTests-elfludapdfytimbonfbtpehrgipu/Build/Intermediates/FitureCompletionTests.build/Debug-iphonesimulator/FitureCompletion.build/swift-overrides.hmap 
-Xcc -I/Users/terhechte/Library/Developer/Xcode/DerivedData/FitureCompletionTests-elfludapdfytimbonfbtpehrgipu/Build/Intermediates/FitureCompletionTests.build/Debug-iphonesimulator/FitureCompletion.build/FitureCompletion-all-target-headers.hmap 
-DDEBUG=1 "

[nathankot]

Nice :smile: How far along is SourcekittenDaemon? I'm going to find some time this weekend to work on it as well hopefully we can get it integrated into company-soucekit!

[terhechte]

Well, mostly ready, with the exception of the parameters listed above. It seems in order to get completion for more than one file, we need the hmap files. Those can be generated by running xcodebuild in the project folder. After that, they're located in .//build/ProjectName.build/Debug/ProjectName.build/.

However, I wonder, how often do those files need to be re-generated? I'm planning to do some tests tomorrow to figure out whether the completions are still up to date when changing the contents of project files.

Completions for one file are fine, but few projects consist of only one file, and as soon as we have more than one file, we need the modules map it seems.

Would be great if we could combine the two projects. I'll hopefully be able to finish the remaining tasks for a 0.1 in the coming days. That means, no support for more than one target, and no support for frameworks yet.

[nathankot]

@terhechte had a play around with sourcekitten daemon today, but unfortunately didn't really manage to get anywhere... I got stuck trying to move all the dependencies to Carthage lol

I came across xcodebuild -showBuildSettings, I wonder how much of the compiler args can the output of that command satisfy?

And the hmap files seem tricky... so we have to prebuild them before the completion command is run? I guess this is done when xcode shows the 'Indexing' message..

I can see many other editors using SourceKittenDaemon as a completion backend, so it's probably best if these remain separate.

[terhechte]

Yeah, it seems that that's what happens when Xcode does the indexing. Running that is probably not too difficult, but I'm still pondering when we have to do it. I'm researching the contents of these modules right now, to better understand them.

As for carthage, apparently the SwiftSockets project changed something and now their xcode project does not work with carthage anymore. A quick fix is the following:

1. carthage update
2. fails to compile
3. open the SwiftSockets Project and change the scheme as in this screenshot:
4. carthage build

[terhechte]

Ok, so now I'm entirely confused. I just spend some more time with the completion generation and added a small script test-completion.sh to the Fixtures folder to easier test things out on the command line.

This works:

sourcekitten complete --text "struct A:Identifiable  { let name: String ; let u: Int } ; let aa = A(name: \"\", u: 5); company1." --offset 97 \
--compilerargs -- "-module-name FitureCompletion \
`pwd`/FitureCompletionTests/ExampleProtocols.swift \
`pwd`/FitureCompletionTests/ExampleStructs.swift \
`pwd`/FitureCompletionTests/CompletionTarget.swift \
-Xcc -I./build/FitureCompletionTests.build/Debug/FitureCompletionTests.build/swift-overrides.hmap \
-Xcc -I./build/FitureCompletionTests.build/Debug/FitureCompletionTests.build/FitureCompletionTests-all-target-headers.hmap \
-DDEBUG=1"%

(once you do xcodebuild -configuration Debug in the folder at least once)

However, when I remove the Xcc entries like this:

sourcekitten complete --text "struct A:Identifiable  { let name: String ; let u: Int } ; let aa = A(name: \"\", u: 5); company1." --offset 97 \
--compilerargs -- "-module-name FitureCompletion \
`pwd`/FitureCompletionTests/ExampleProtocols.swift \
`pwd`/FitureCompletionTests/ExampleStructs.swift \
`pwd`/FitureCompletionTests/CompletionTarget.swift \
-DDEBUG=1"%

It still works. I'm really sure that this did not work when I tried it earlier this week. I also removed the .build folder, but it still worked. I'm lost as to why this did not work earlier this week, and now it apparently works. Does the script work for you when you execute it?

Cheers, Benedikt

[nathankot]

I've tried both variants of test-completion.sh and they both work. I've also tried clearing DerivedData and the build folder and it still works after that.

In your previous comment you pass in the -sdk option, test-completion.sh doesn't seem to have it. I added -sdk back in but both variants still work as well.

Perhaps the hmap files are not required?

[terhechte]

I just commited a couple of changes. The args that are generated are now all correct, the file paths are correct etc. When I insert the very same args into the command line sourcekitten and run it, I'm getting completions. When I'm doing it in SourceKittenD, I'm getting a EXC_BAD_ACCESS. I'm pretty sure that that's a minor error and I'm just doing something stupid somewhere. I'll look into it again tomorrow. Overall, once that works, we should have multi-file completions (with lots of exceptions for multiple targets, frameworks, and what not, but it is a first step).

[nathankot]

@terhechte

Awesome :) I'm wondering if the tests are meant to be passing at this stage? I get this when running the tests:

[terhechte]

That's the remaining issue. At that point all the argument are correct, and I'm calling sourcekitten like the cli app, but something is clearly wrong. Did not figure it out yet.

Sent from my iPhone

On 25.11.2015, at 10:00, Nathan Kot notifications@github.com wrote:

@terhechte

Awesome :) I'm wondering if the tests are meant to be passing at this stage? I get this when running the tests:

— Reply to this email directly or view it on GitHub.

[nathankot]

@terhechte I've merged the sourcekitten daemon branch to master as I see you've made an initial release =D

[terhechte]

Yeah, I thought with Linux Swift out, there might be a chance to catch a few additional contributors in order to target Linux support in the near future.

[nathankot]

Sweet, company-sourcekit's linux support should hopefully just depend on sourcekittendaemon