search

terhechte / SourceKittenDaemon

Swift Auto Completions for any Text Editor
MIT License
529 stars 37 forks source link

[WIP] Port to Linux #62

Closed felix91gr closed 7 years ago

felix91gr commented 7 years ago

Using Norio Nomura's 3.1.1 branch for compiling a toolchain with SourceKit, passing tests and all, I'm starting the porting process for Linux. I'll post here on the progress and misses that I make or meet.

Let's make it work! :smile:

felix91gr commented 7 years ago

Errors from make install

The Makefile has some errors, I think because some of the commands don't work on Linux:

ROOT_DIR := $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
VERSION := $(shell agvtool what-marketing-version -terse1)
BUILD := .build
DIST := dist
PREFIX := /usr/local
IDENTIFIER := com.stylemac.SourceKittenDaemon
BINARIES_FOLDER := /bin
# ...
.PHONY: install
install: $(DIST)
    mkdir -p "$(PREFIX)$(BINARIES_FOLDER)"
    cp -f "$(DIST)$(BINARIES_FOLDER)/sourcekittendaemon" "$(PREFIX)$(BINARIES_FOLDER)/"
#...
.PHONY: $(BUILD)
$(BUILD):
    mkdir -p $@
    swift build -c release --build-path $(BUILD)

I get this output:

make: agvtool: Command not found
mkdir -p .build
swift build -c release --build-path .build
#... (it works, it downloads the dependencies but I stopped it midway because of the preceding errors)

Updates on this issue

Commands that are not available on Linux

felix91gr commented 7 years ago

Build errors

This is the console output that I get (I included all the parts that could be relevant)

mkdir -p .build
swift build -c release --build-path .build
Fetching https://github.com/Carthage/Commandant.git
# More Fetching
Fetching https://github.com/vapor/socks.git
Cloning https://github.com/jpsim/SourceKitten.git
Resolving https://github.com/jpsim/SourceKitten.git at 0.17.2
# More Cloning-Resolving...
Cloning https://github.com/vapor/tls.git
Resolving https://github.com/vapor/tls.git at 1.1.2
Compile CYaml src/parser.c
Compile CYaml src/api.c
Compile CYaml src/writer.c
Compile CYaml src/emitter.c
Compile CYaml src/scanner.c
Compile CYaml src/loader.c
Compile CYaml src/dumper.c
Compile Swift Module 'SWXMLHash' (2 sources)
Compile CLibreSSL aes_misc.c
Compile CLibreSSL ssl_asn1.c
Compile CLibreSSL ecp_mont.c
Compile CLibreSSL ssl_rsa.c
Compile Swift Module 'Result' (2 sources)
Compile CLibreSSL txt_db.c
Compile CLibreSSL e_xcbc_d.c
Compile CLibreSSL x509spki.c
Compile CLibreSSL a_strex.c
Compile CYaml src/reader.c
# ...
# Many files in between, following the pattern "Compile CLibreSSL *.c"
# ...
Compile CLibreSSL p12_p8e.c
Compile Swift Module 'Xcode' (4 sources)
Compile Swift Module 'SocksCore' (15 sources)
Compile Swift Module 'libc' (1 sources)
Compile Swift Module 'Polymorphic' (2 sources)
Compile Swift Module 'PathIndexable' (2 sources)
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:40:38: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
    return T(id: key, dict: obj.dict as AnyObject, allObjects: self)
                            ~~~~~~~~~^~~~~~~~~~~~
                                     as!
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:81:64: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
    self.project = PBXProject(id: rootObjectId, dict: projDict as AnyObject, allObjects: allObjects)
                                                      ~~~~~~~~~^~~~~~~~~~~~
                                                               as!
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:102:43: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
      return type.init(id: id, dict: dict as AnyObject, allObjects: allObjects)
                                     ~~~~~^~~~~~~~~~~~
                                          as!
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:106:55: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit?
    assertionFailure("Unknown PBXObject subclass isa=\(isa)")
                                                      ^~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:106:56: note: use 'String(describing:)' to silence this warning
    assertionFailure("Unknown PBXObject subclass isa=\(isa)")
                                                      ~^~~~
                                                       String(describing:  )
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:106:56: note: provide a default value to avoid this warning
    assertionFailure("Unknown PBXObject subclass isa=\(isa)")
                                                      ~^~~~
                                                           ?? <#default value#>
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/Xcode.swift.git-8174145864755727667/Sources/XCProjectFile.swift:107:41: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
    return PBXObject(id: id, dict: dict as AnyObject, allObjects: allObjects)
                                   ~~~~~^~~~~~~~~~~~
                                        as!
Compile Swift Module 'TurnstileCrypto' (3 sources)
Compile Swift Module 'Jay' (21 sources)
Compile CLibreSSL d1_enc.c
Compile CLibreSSL dso_null.c
# ...
# Many files in between, following the pattern "Compile CLibreSSL *.c"
# ...
Compile CLibreSSL p12_utl.c
Compile CLibreSSL tasn_prn.c
Compile CLibreSSL ts_rsp_sign.c
<unknown>:0: error: build had 1 command failures
error: exit(1): /build/swift-FullSKLinux/usr/bin/swift-build-tool -f /home/felix/Documents/SKD/SourceKittenDaemon/.build/release.yaml
Makefile:43: recipe for target '.build' failed
make: *** [.build] Error 1

Is it a problem with CLibreSSL? :thinking: For today I don't have much time left, I'll leave this as-is.

For tomorrow, I'll try to (1) use the less-dependencies branch, because I get the feeling that CLibreSSL might not be used there, and (2) find exactly what's the error that makes swift-build-tool crash.

felix91gr commented 7 years ago

Finding build errors in dependencies

I'm testing the dependencies separately, to find what fails to build and why. Here are the current results:

Dependencies being used now

I'm testing if specific versions that we have in our Package.swift are able to build independently. The key for successfully porting SKD to Linux might lie within updating which versions we choose to use.

Compile Swift Module 'Sessions' (6 sources)
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Sessions/CacheSessions.swift:24:29: warning: 'bytes' is deprecated: Use the throwing method `bytes(count: Int)` instead.
        return CryptoRandom.bytes(16).base64String
                            ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Sessions/CacheSessions.swift:24:39: warning: 'base64String' is deprecated: Use `Bytes.base64Encoded.string` instead.
        return CryptoRandom.bytes(16).base64String
                                      ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Sessions/MemorySessions.swift:44:29: warning: 'bytes' is deprecated: Use the throwing method `bytes(count: Int)` instead.
        return CryptoRandom.bytes(16).base64String
                            ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Sessions/MemorySessions.swift:44:39: warning: 'base64String' is deprecated: Use `Bytes.base64Encoded.string` instead.
        return CryptoRandom.bytes(16).base64String
                                      ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Auth/Authentication/CacheSessionManager.swift:29:39: warning: 'bytes' is deprecated: Use the throwing method `bytes(count: Int)` instead.
        let identifier = CryptoRandom.bytes(16).base64String
                                      ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Auth/Authentication/CacheSessionManager.swift:29:49: warning: 'base64String' is deprecated: Use `Bytes.base64Encoded.string` instead.
        let identifier = CryptoRandom.bytes(16).base64String
                                                ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Auth/Authentication/Header/Basic.swift:16:34: warning: 'base64DecodedString' is deprecated: Use `String.bytes.base64Decoded.string` instead.
        let decodedToken = token.base64DecodedString
                                 ^
Compile Swift Module 'Vapor' (86 sources)
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Vapor/Hash/Hasher.swift:32:25: warning: 'base64String' is deprecated: Use `Bytes.base64Encoded.string` instead.
            return hash.base64String
                        ^
/home/felix/Documents/SKD/TestingDependencies/vapor/Sources/Vapor/Cipher/CipherProtocol.swift:50:22: warning: 'base64String' is deprecated: Use `Bytes.base64Encoded.string` instead.
            return m.base64String
                     ^

Maybe we can update the versions that SKD uses in this update. I'll test how each dependency behaves in different releases from the ones we're currently using, and maybe we can work from there to use a more recent version of each one.

Conclusions

The four dependencies seem to build without problems. Embassy's warnings are probably not relevant, as they are of the accessor consistency type. Vapor's warnings are worrisome: it's using deprecated APIs, which could lead to a future SKD release not working after support for them ends. This can probably be easily fixed by changing the dependency to use version 1.5.9, which compiles without warnings.

Another conclusion we can derive from this is the following: the building error of SKD comes from either SKD itself, or from Xcode.swift, which I haven't yet figured out how to build as a standalone package. In the next comment, I address this exact point.

felix91gr commented 7 years ago

Regarding Xcode.swift

As seen in my build console output, Xcode.swift seems to have some syntax errors with AnyObject.

Xcode.swift is a fork of XcodeEdit. Using Github's compare, we can see that the fork is actually behind the original.

Also, the latest tag from Xcode.swift is from Nov 12th, 2016. Whereas the latest tag from XcodeEditis from Mar 28th, 2017. Given the recent changes to AnyObject, maybe Xcode.swift lacks some updating that is important for this compiling error.

After changing to XcodeEdit

I got the exact same errors in my build output:

/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/XcodeEdit-7028054331236040754/Sources/XCProjectFile.swift:40:38: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
    return T(id: key, dict: obj.dict as AnyObject, allObjects: self)
                            ~~~~~~~~~^~~~~~~~~~~~
                                     as!
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/XcodeEdit-7028054331236040754/Sources/XCProjectFile.swift:81:64: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
    self.project = PBXProject(id: rootObjectId, dict: projDict as AnyObject, allObjects: allObjects)
                                                      ~~~~~~~~~^~~~~~~~~~~~
                                                               as!
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/XcodeEdit-7028054331236040754/Sources/XCProjectFile.swift:102:43: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
      return type.init(id: id, dict: dict as AnyObject, allObjects: allObjects)
                                     ~~~~~^~~~~~~~~~~~
                                          as!
/home/felix/Documents/SKD/SourceKittenDaemon/.build/checkouts/XcodeEdit-7028054331236040754/Sources/XCProjectFile.swift:107:41: error: 'JsonObject' (aka 'Dictionary<String, AnyObject>') is not convertible to 'AnyObject'; did you mean to use 'as!' to force downcast?
    return PBXObject(id: id, dict: dict as AnyObject, allObjects: allObjects)
                                   ~~~~~^~~~~~~~~~~~
                                        as!

I'm gonna try doing what the compiler suggests in those error messages: changing as to as! in those 4 lines.

After fixing the as-as! error

Everything compiles. Except SKD:

Compile Swift Module 'SourceKittenDaemon' (8 sources)
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Extensions/Path+.swift:2:8: error: no such module 'Xcode'
import Xcode
       ^

I suspect this has to do with the fact that Xcode.swift includes a file named Xcode.h, whereas XcodeEdit doesn't.

I shall try now to fix the as-as! error on Xcode.swift, and see if then SKD compiles!

Final step (for today, anyway)

I kept the original Xcode.swift dependency. I also realized that XcodeEdit is a higher-order dependency (as in, it's a dependency of one of the many dependencies down the dependency graph). No problem: I fixed the as-as! keyword error on both of them.

Then the build process started, and it ended up in the last step, again. But this time, the error was much more interesting. I think we're up to something:

Compile Swift Module 'SourceKittenDaemon' (8 sources)
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:27:28: error: use of unresolved identifier 'FSEventStreamCreate'
        self.eventStream = FSEventStreamCreate(
                           ^~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:28:17: error: use of unresolved identifier 'kCFAllocatorDefault'
                kCFAllocatorDefault,
                ^~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:32:17: error: use of unresolved identifier 'FSEventStreamEventId'
                FSEventStreamEventId(kFSEventStreamEventIdSinceNow),
                ^~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:32:38: error: use of unresolved identifier 'kFSEventStreamEventIdSinceNow'
                FSEventStreamEventId(kFSEventStreamEventIdSinceNow),
                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:34:17: error: use of unresolved identifier 'FSEventStreamCreateFlags'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                ^~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:34:42: error: use of unresolved identifier 'kFSEventStreamCreateFlagFileEvents'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:34:79: error: use of unresolved identifier 'kFSEventStreamCreateFlagNoDefer'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                                                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:37:9: error: use of unresolved identifier 'FSEventStreamScheduleWithRunLoop'
        FSEventStreamScheduleWithRunLoop(eventStream,
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:40:9: error: use of unresolved identifier 'FSEventStreamStart'
        FSEventStreamStart(eventStream)
        ^~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:43:36: warning: result of call to 'addObserver(forName:object:queue:usingBlock:)' is unused
        NotificationCenter.default.addObserver(
                                   ^          ~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:53:9: error: use of unresolved identifier 'FSEventStreamInvalidate'
        FSEventStreamInvalidate(eventStream)
        ^~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~

I suspect the error lies within the XCode.h file in Xcode.swift. It imports Cocoa/Cocoa.h, which afaik is only available on Darwin.

:thinking: maybe we could fork Xcode.swift and try to use the GNUstep library instead of Cocoa, for this. I'll read the documentation as soon as I can... maybe it can be done.

The other option is to not depend on Cocoa.h or the Xcode.swift project. I'll examine the XcodeEdit repo, maybe it works on Linux without depending on Darwin-only libraries.

TO-DO

felix91gr commented 7 years ago

In-Depth research of XcodeEdit

  1. Does this package compile? Yes! But only master. The latest release doesn't: it fails, as seen before, on the as-as! error.
  2. How different is it from Xcode.swift? Can it replace it? Stay tuned...
  3. Etc

(I'm following up on this as soon as I can. This week'll be work-heavy... therefore, I'll be back at most, by the weekend!)

felix91gr commented 7 years ago

Regarding... Foundation

So... I had some time today to closely examine the console errors. They are at the end of this comment for everyone to check.

I found the following:

  1. Both using Vapor and Embassy, if we fix de as-as! error in Xcode.swift, the compiling process stops in a similar manner: it fails right after starting the step Compile Swift Module 'SourceKittenDaemon' (8 sources).
  2. The errors stem from two Foundation components:
    • FSEventStream. This one is tricky to ignore. Afaik, the FSEventStream tools are only available on Darwin.
    • kCAllocatorDefault. Only available on Darwin, but not as tricky. This seems to some kind of alias for NULL. If so, it should be trivial to replace.
  3. Embassy introduces a third type of error, ~regarding a dependency on Python 3's KqueueSelector~ because KqueueSelector isn't built for Linux. Indeed, the latest Embassy commit stopped defining that class there. That said though, it looks like the author intends to support it for Linux in the future.

The hardest barrier seems to be the use of FSEventStream. I hope we can replace it... I couldn't find it in the swift-corelibs-foundation project.

Appendix: console output, errors at final step

Vapor setup

/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:27:28: error: use of unresolved identifier 'FSEventStreamCreate'
        self.eventStream = FSEventStreamCreate(
                           ^~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:28:17: error: use of unresolved identifier 'kCFAllocatorDefault'
                kCFAllocatorDefault,
                ^~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:32:17: error: use of unresolved identifier 'FSEventStreamEventId'
                FSEventStreamEventId(kFSEventStreamEventIdSinceNow),
                ^~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:32:38: error: use of unresolved identifier 'kFSEventStreamEventIdSinceNow'
                FSEventStreamEventId(kFSEventStreamEventIdSinceNow),
                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:34:17: error: use of unresolved identifier 'FSEventStreamCreateFlags'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                ^~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:34:42: error: use of unresolved identifier 'kFSEventStreamCreateFlagFileEvents'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:34:79: error: use of unresolved identifier 'kFSEventStreamCreateFlagNoDefer'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                                                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:37:9: error: use of unresolved identifier 'FSEventStreamScheduleWithRunLoop'
        FSEventStreamScheduleWithRunLoop(eventStream,
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:40:9: error: use of unresolved identifier 'FSEventStreamStart'
        FSEventStreamStart(eventStream)
        ^~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:43:36: warning: result of call to 'addObserver(forName:object:queue:usingBlock:)' is unused
        NotificationCenter.default.addObserver(
                                   ^          ~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:53:9: error: use of unresolved identifier 'FSEventStreamInvalidate'
        FSEventStreamInvalidate(eventStream)
        ^~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SourceKittenDaemon/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~

Embassy setup

/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:27:28: error: use of unresolved identifier 'FSEventStreamCreate'
        self.eventStream = FSEventStreamCreate(
                           ^~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:28:17: error: use of unresolved identifier 'kCFAllocatorDefault'
                kCFAllocatorDefault,
                ^~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:32:17: error: use of unresolved identifier 'FSEventStreamEventId'
                FSEventStreamEventId(kFSEventStreamEventIdSinceNow),
                ^~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:32:38: error: use of unresolved identifier 'kFSEventStreamEventIdSinceNow'
                FSEventStreamEventId(kFSEventStreamEventIdSinceNow),
                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:34:17: error: use of unresolved identifier 'FSEventStreamCreateFlags'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                ^~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:34:42: error: use of unresolved identifier 'kFSEventStreamCreateFlagFileEvents'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:34:79: error: use of unresolved identifier 'kFSEventStreamCreateFlagNoDefer'
                FSEventStreamCreateFlags(kFSEventStreamCreateFlagFileEvents | kFSEventStreamCreateFlagNoDefer))!
                                                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:37:9: error: use of unresolved identifier 'FSEventStreamScheduleWithRunLoop'
        FSEventStreamScheduleWithRunLoop(eventStream,
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:40:9: error: use of unresolved identifier 'FSEventStreamStart'
        FSEventStreamStart(eventStream)
        ^~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:43:36: warning: result of call to 'addObserver(forName:object:queue:usingBlock:)' is unused
        NotificationCenter.default.addObserver(
                                   ^          ~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:53:9: error: use of unresolved identifier 'FSEventStreamInvalidate'
        FSEventStreamInvalidate(eventStream)
        ^~~~~~~~~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
                     ^~~~~~~~~~~~~~~~
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Server/CompletionServer.swift:28:28: error: use of unresolved identifier 'KqueueSelector'
        let selector = try KqueueSelector()
                           ^~~~~~~~~~~~~~
Embassy.SelectSelector:1:20: note: did you mean 'SelectSelector'?
final public class SelectSelector : Selector {
                   ^
/home/felix/Documents/SKD/SKD_lessDep/Sources/SourceKittenDaemon/Completer/Completer.swift:22:22: error: use of undeclared type 'FSEventStreamRef'
    let eventStream: FSEventStreamRef
felix91gr commented 7 years ago

Update on Embassy's build error

I opened an issue there asking about KqueueSelector's support for Linux. As you can see, Linux is entirely supported! Here's what Fang-Pen Lin showed me:

  1. SelectSelector is the cross-platform alternative to KqueueSelector.
  2. The APIs of both classes are the same, as we can see in the usage example.

So far, adoption of Embassy to replace Vapor doesn't seem to impose any barriers towards support on Linux :)

terhechte commented 7 years ago

Hey! awesome work so far! We don't have to use Embassy if that makes too much problems. I like that there're less dependencies when using Embassy, but if it's not 100% tested on Linux, we can also stay with Vapor.

Other than that, it seems the only thing missing is the missing Xcode.h, is it? And that'd be replaceable with XcodeEdit if I see it correctly? I'll give it a try to take the current SKD master and replace Xcode.swift with XcodeEdit to see if that's feasible. Are there any other blockers?

felix91gr commented 7 years ago

Are there any other blockers?

Yes! Using FSEventStream gives errors, because it's not available outside of Darwin.

I looked up alternatives to that API for Linux, and it seems like we can use inotify. It monitors filesystem events. We could maybe use #if statements in a small wrapper to make FSEvent calls when in Darwin and inotify calls when in Linux.

(regarding Xcode.swift) And that'd be replaceable with XcodeEdit if I see it correctly?

I'm almost certain. My tests, after fixing the as-as! error I found, only showed problems with the FSEvent APIs. The XCode.h header didn't give any errors itself. It doesn't seem like it needs to be replaced right away. That said: XcodeEdit used to be called Xcode.swift, but now that it reached 1.0.0 it was renamed and also made compliant with SPM. It looks like a good upgrade to the dependency.

I propose we test both Xcode.swift and XcodeEdit after we fix the FSEvent blocker. What do you think? :)

felix91gr commented 7 years ago

Dangit, I forgot to answer your points regarding Embassy.

It looks good, specially since the SelectSelector seems to be a trivial replacement for KqueueSelector.

But just to make our lifes easier, I think it's more convenient if we keep Vapor for now. After we make the port work, we can probably try to swap it for Embassy and debug it without the worry of "is this a port bug, or just an Embassy bug?". U know what I mean? 😄

felix91gr commented 7 years ago

Update on inotify

I'm preparing a mockup for using inotify for Linux. I'm now almost 100% sure that we can use it for the exact same functionality as the FSEventStream APIs on Darwin. One great thing is: just as in Darwin, it's included in the system already (it's part of the kernel, in fact), so there's no extra dependency to install :)

Update no.2: It works!

I found and forked a repo that implements something quite similar to FSEventStream's API. I think it's just right for our use case. @terhechte, take a look: FSWatcher

(I also made a project that shows how to use it)

I think I can modify it so that it includes the 2 second delay and kFSEventStreamCreateFlagNoDefer EventStream flag that SKD is using right now. That's all that is missing (I think) to match exactly the behavior of what we have in Mac OS*.

*: Maybe I could implement also (w/ a flag in the watch method) the kFSEventStreamCreateFlagFileEvents flag, which allows for file-level notifications. I'll look it up... maybe it's not necessary (maybe with directory-level notifications we are notified anyway about changes in files).

Update no.3: it's kinda done?

I think I finished implementing the deferring mechanism that SKD is using from the FSEventStream API. Check the repo out :)

felix91gr commented 7 years ago

Commits have started

The project now compiles on Linux. Linux users, read the commit notes to know the caveats (this is still WIP).

I made a wrapper that makes the OS transparent with regards to File System events. It uses my fork of the FileSystemWatcher, which I modified to have the same functionality as the FSEventStream APIs present in Darwin.

Now, on to testing ^^ I still have to test that what got built actually works. But it compiles :) I'm happy for now.

@terhechte, could you test that this commit builds on your machine? I hope I didn't break the Mac OS build. If it does, the errors should only be typos. I only redirected what was already there to the wrapper... but since that API doesn't exist in Linux, I can't test it myself xD

terhechte commented 7 years ago

Hey!

I'm testing on macOS now. Will report back. Thank you so much already!

terhechte commented 7 years ago

I've added some fixes to make it compile properly on macOS. The details are in the commit. I haven't had time to test it on Linux yet, but will hopefully be able to do so tomorrow (already 10pm here). Thanks for your hard work!

felix91gr commented 7 years ago

Awesome :) I tested it in my Linux, works allright!

I'm gonna try to keep at what's left to do, this weekend. I'll be very busy with Uni stuff though, so I'm not hoping to get much done soon. Below is what's left to do (in theory) to finish the port.

Remaining tasks

I want to work on these after I finish polishing my FileSystemWatcher, which I'm making tests for. If they take me too long to write, I'll see if I can finish the porting anyway but with a good eye on the bugs that may arise, because the Watcher wouldn't be thoroughly tested.

If it is: _(or if maybe there are performance benefits in KqueueSelector):

If it's not, that point is done.

After this checklist is done, it's up to testing the functionality and that's it! It's important to finish these tasks first though, just because they make the program more robust.

And these would-be-nice

But aren't mandatory just yet.

felix91gr commented 7 years ago

Small updates

  1. I'm gonna try to make progress on this during this week, to start testing SKD functionality ASAP.

  2. The new Swift 4.0 binary snapshots include full SourceKit support for Linux. That's awesome! Linux users wanting to use these tools might not need to compile the toolchain using custom cherrypicks anymore :D

  3. I'll also start side testing with the 4.0 snapshots in what's left of the Linux port, to be aware of any changes needing to happen for migrating users. Since swift 4.0 includes swift 3.X source compatibility, I'll be able to test it in both environments to have a fair assessment of the quality of the toolchain binaries themselves.

Update on tests

Update 0

I realized I should've ran tests for each dependency. I'm doing that and troubleshooting them atm. Later today (May 19th as of this edit) I'll post the test results.

Update 1

Some tests in the dependencies have failed. I'm suspecting that the special branch I'm using must be involved there. Since (I think) I might have commits that belong to Swift 4.0, maybe because of that some packages get compiled in a different way than how they were intended to do.

I will get this sorted out after I make a sandboxed setup with the 4.0 snapshots binaries. I'm almost certain that those allow for -swift-version 3 to be passed as a flag to swiftc

Update 2

Commandant and SourceKitten pass all of their tests. If anyone wants to try it, the Commandant test suite has some quirks in the latest release, but nothing to worry about :+1:

Update 3

Xcode.swift and XcodeEdit (the old and new releases of the same package) don't have any tests. Therefore, if any errors are present after everything is checked, we might need to look into this.

Update 4

vapor passes all of the tests in their latest release (2.0.0).

Embassy is giving me trouble. Might be related to the AnyObject changes. :thinking: more testing is coming.

Update 5

Tested vapor under the version that our master branch uses (1.5.9). It gives me 2 errors, but they are expected errors, so it should be just fine.

Update 6

Embassy's tests can't compile under my setup. It gives me errors like 

'Error?' is not convertible to 'NSError?'; did you mean to use 'as!' to force downcast?
                receivedError = error as NSError?

I'll be testing it again with precompiled binaries under the -swift-version 3 flag, to see if it works there. I'll also try it with the 3.X binaries. It will probably work (unlike SourceKitten which needs at least the commits included in Nomura's branch)

Update 7, and last on this topic

I made Embassy run its tests under Swift 3.0.2. Only there everything compiled. Both Swift 3.1.X and the Swift 4.0 snapshots made the test suite not compile.

I'll report the errors I got to the maintainer. Maybe I can help her fix some.

felix91gr commented 7 years ago

Waiting for updates... or maybe not :smile_cat:?

@terhechte These results made me realize that some of our dependencies will need to be updated to Swift 3.1 or Swift 4.0 (after its release, of course) before we can use them on Linux.

This is because:

  1. Linux's full SourceKit support only started in Swift 3.1. The branch that Norio Nomura made completed it, and it's based precisely on Swift 3.1. (Swift 4.0 includes Nomura's cherrypicks, and therefore from then on Linux is 100% supported.)

  2. Embassy can only run its tests under Swift 3.0.2. Any other release can't compile them.

We might be able to use the other dependencies, though. These are my current results:

Current results

Here is a list of the dependencies that pass all of their tests in every Swift release that works for SKD in Linux (that is, Swift ≥ 3.1.x).

Swift 3.1.1

Pass

Fails

No testing available

Swift 4.0

Pass

Fails

No testing available

Possible courses of action

Wait for Embassy to update

I like Embassy, because it fits our use case so well. We could wait for it to update and try then. Maybe I can help the owner (Fang-Pen Lin) to update it for Swift 3.1.1

Use (and fix the failing tests) in Vapor, and change to Embassy later

Since Vapor at least compiles all of its tests, we could try to fix it and see how we go from there.

Final verdict on XcodeEdit

The only moving part, aside from the server packages, would then be XcodeEdit. We should change to using XcodeEdit. I asked about the errors that I got while compiling SKD with it, and all the errors were just because of encapsulation. Apart from that, the API is the same as in Xcode.swift, so we're good. We can update the fork that we're using, or make a new fork, and it'll be enough. I'll do one in the meantime; we can go back to the original as soon as tomlokhorst updates it.

Edit log

  1. Fixed my mistakes on interpreting what an 'expected error' was.
felix91gr commented 7 years ago

Appendix: how did I test the different environments

I'm using the precompiled binaries along with the command env. This allows you to change your environment variables just for one command. Using the fish shell, I made the following function (you can make a similar one for bash):

function changeToolChain
  switch $argv[1]
    case '3.0'
      env LD_LIBRARY_PATH=/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-3.0.2-RELEASE-ubuntu16.04/usr/lib/:/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-3.0.2-RELEASE-ubuntu16.04/usr/lib/swift/linux/: 
        PATH=/home/felix/.rbenv/plugins/ruby-build/bin:/home/felix/.rbenv/shims:/home/felix/.rbenv/bin:/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-3.0.2-RELEASE-ubuntu16.04/usr/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games 
        $argv[2..-1]
    case '3.1'
      env LD_LIBRARY_PATH=/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-3.1.1-RELEASE-ubuntu16.04/usr/lib/:/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-3.1.1-RELEASE-ubuntu16.04/usr/lib/swift/linux/: \
        PATH=/home/felix/.rbenv/plugins/ruby-build/bin:/home/felix/.rbenv/shims:/home/felix/.rbenv/bin:/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-3.1.1-RELEASE-ubuntu16.04/usr/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games \
        $argv[2..-1]
    case '4.0'
      env LD_LIBRARY_PATH=/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-4.0-DEVELOPMENT-SNAPSHOT-2017-05-15-a-ubuntu16.04/usr/lib/:/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-4.0-DEVELOPMENT-SNAPSHOT-2017-05-15-a-ubuntu16.04/usr/lib/swift/linux/: 
        PATH=/home/felix/.rbenv/plugins/ruby-build/bin:/home/felix/.rbenv/shims:/home/felix/.rbenv/bin:/home/felix/Documents/SKD/SwiftAlternativeToolchains/swift-4.0-DEVELOPMENT-SNAPSHOT-2017-05-15-a-ubuntu16.04/usr/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games 
        $argv[2..-1]
  end  
end

Then, I tested every dependency using the following commands:

rm -rf .build Package.pins # Removes all of last compiling's artifacts 
changeToolChain swift test
felix91gr commented 7 years ago

I'm closing this in lieu of #61. I'll post what's left to do there, and leave this as an archive of my research and testing for the Linux port :)