Distros other than Vanilla Ubuntu have some issues

felix91gr commented 6 years ago

I'm on Elementary OS, an Ubuntu-based Linux. The Swift binaries compiled for Ubuntu 16.04 are fully compatible with it. But if using swiftenv, I get two results depending on how I install Swift:

`swiftenv install N`:

It first downloads the repos:

swift
swift-llvm
swift-clang
swift-lldb
swift-cmark
swift-llbuild
swift-package-manager
swift-compiler-rt
swift-corelibs-xctest
swift-corelibs-foundation
swift-corelibs-dispatch

Then proceeds to compile Swift:

Building Swift
This may take a very long time...

And after that when I run swift (to invoke the REPL), I get this error:

<unknown>:0: error: unable to load standard library for target 'x86_64-unknown-linux-gnu'
swift: /home/felix/.swiftenv/tmp/swiftenv-build-4.1/swift/include/swift/AST/Module.h:1193: swift::SourceFile &swift::ModuleDecl::getMainSourceFile(swift::SourceFileKind) const: Assertion `!Files.empty() && "No files added yet"' failed.
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x3f259a4]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x3f25ce6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f0a50ae4390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f0a4f009428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f0a4f00b02a]
/lib/x86_64-linux-gnu/libc.so.6(+0x2dbd7)[0x7f0a4f001bd7]
/lib/x86_64-linux-gnu/libc.so.6(+0x2dc82)[0x7f0a4f001c82]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x4dd35b]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x4dacb8]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x4c146c]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x4bf96c]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x478364]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f0a4eff4830]
/home/felix/.swiftenv/versions/4.1/usr/bin/swift[0x475c19]
Stack dump:
0.  Program arguments: /home/felix/.swiftenv/versions/4.1/usr/bin/swift -frontend -repl -disable-objc-interop -color-diagnostics -module-name REPL 
fish: “swift” terminated by signal SIGABRT (Abort)

`swiftenv install SWIFT_BINARIES_URL`

After this happened, I decided to try again with a directed binary installation:

swiftenv install https://swift.org/builds/swift-4.1-release/ubuntu1604/swift-4.1-RELEASE/swift-4.1-RELEASE-ubuntu16.04.tar.gz
Downloading https://swift.org/builds/swift-4.1-release/ubuntu1604/swift-4.1-RELEASE/swift-4.1-RELEASE-ubuntu16.04.tar.gz
/tmp/swiftenv-4.1-RELEASE-felix ~
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  171M  100  171M    0     0  3648k      0  0:00:48  0:00:48 --:--:-- 5053k
~
4.1-RELEASE has been installed.
felix@felix-X550LD ~> swift
Welcome to Swift version 4.1 (swift-4.1-RELEASE). Type :help for assistance.
  1>

Which works no problem :+1:

What can be done

I think the best alternative is to build under non-ubuntu-based distros and figure out what's wrong with the build process that doesn't let it build or configure properly the Stdlib, s.t. they can actually use swiftenv.

And under ubuntu-based distros, having a way to tell what ubuntu are they based on. Then, proceeding to just install the binaries as if they were that particular ubuntu.

What I don't know

Does building work properly under vanilla Ubuntu? Maybe the stdlib gets properly built or configured iff that's the OS swiftenv is under.

Extras

I tested this both under Swift 4.0 and 4.1, to make sure it wasn't my 4.1 PR that made swiftenv fail.

kylef commented 6 years ago

And under ubuntu-based distros, having a way to tell what ubuntu are they based on. Then, proceeding to just install the binaries as if they were that particular ubuntu.

That is possible, you can set the PLATFORM environment variable which will skip the platform check and instead use the value of the environment variable.

For example:

$ set -x PLATFORM ubuntu16.10
$ swiftenv install 4.1

That said, perhaps it makes sense to rename this environment variable and prefix it with SWIFTENV_ or SWIFT_ so you can set this in your global profile and no other tools will pick it up.

https://github.com/kylef/swiftenv/blob/4114933644f11046709c63ff8f1deae4facf0f4b/libexec/swiftenv-install#L38-L42

As for building Swift, I have tested some older Swift versions on Archlinux with success in the past. Unfortunately with the resources I have it is very slow and cumbersome to test building each release, the build instructions are the most untested part of swiftenv as it is not really possible to test this on a wide range of platforms.

It sounds like the swift repl is perhaps no longer statically linked and requires some other built libraries. As we're only installing the Swift binaries:

https://github.com/kylef/swiftenv/blob/master/libexec/swiftenv-build#L62-L65

I think the changes in the PR at https://github.com/kylef/swiftenv/pull/87 should solve this problem, however there are no build profiles for Swift 4/4.1 for this and to be honest I haven't had the time to test out the changes properly so no idea if they have problems of their own. I'll go ahead and rebase this PR and add the 4.0, 4.1 profiles right now so perhaps you can give those a try.

felix91gr commented 6 years ago

That is possible, you can set the PLATFORM environment variable which will skip the platform check and instead use the value of the environment variable. For example: $ set -x PLATFORM ubuntu16.10 $ swiftenv install 4.1

It works great! Thanks! :)

I was talking about another thing though: deriving from the PLATFORM environment variable, or something akin to that, what Ubuntu distro is this one based on. That way you extend the easiest of setups to all Ubuntu-based distros :)

As for building Swift, I have tested some older Swift versions on Archlinux with success in the past. Unfortunately with the resources I have it is very slow and cumbersome to test building each release, the build instructions are the most untested part of swiftenv as it is not really possible to test this on a wide range of platforms.

That's totally understandable. I think it's okay if you can't test it. Sometime with CI, maybe, this can be tested periodically :)

I think the changes in the PR at #87 should solve this problem, however there are no build profiles for Swift 4/4.1 for this and to be honest I haven't had the time to test out the changes properly so no idea if they have problems of their own. I'll go ahead and rebase this PR and add the 4.0, 4.1 profiles right now so perhaps you can give those a try.

Cool! I'll follow it, I hope it goes well. I'll test it after you release it so that I can give you feedback :)

felix91gr commented 6 years ago

Update: fwiw, this command shows you the Ubuntu version you're running:

lsb_release -r -u -s

In my case, it outputs 16.04, as it should.

And to confirm you're running an Ubuntu-based distro:

lsb_release -i -u -s

Which in my case outputs Ubuntu

If I get the time, I'd like to add this to swiftenv next week. But there are my findings, for if someone else's interested as well.

felix91gr commented 6 years ago

Ah, what the heck. I added it in #121.

The change was very easy to make, now that you pointed me to the right file to edit :)

kylef commented 6 years ago

That's neat, thanks for the PR #121!

I'm not having much luck building Swift, I've tempted to get Swift building on Circle CI (https://github.com/kylef/swiftenv/commit/c0b19d43945e15f27f8aed1a6d2c4415d58e8882) but eventually it fails during the build with clang getting killed. I'm guessing this is due to memory pressure.

To make the built version work, I think the installation instructions at https://github.com/kylef/swiftenv/blob/master/libexec/swiftenv-build#L62-L65 will need to be to be updated to install other libraries in the package. Do you happen to still have the tmp directory from installation containing the built files? It is normally cleared after installation (unless --no-clean was provided to swiftenv install). It should be at ~/.swiftenv/tmp/swiftenv-build-4.1-dev/build/Ninja-ReleaseAssert/. Inside will be the swift directory.

felix91gr commented 6 years ago

I'm guessing this is due to memory pressure.

It might be! I’ve faced a similar problem when compiling Swift. Try to limit the number of threads used to compile, those are the big memory eaters :)

Do you happen to still have the tmp directory from installation containing the built files?

Nu, I deleted it all when trying to debug other issues.

segabor commented 6 years ago

I finally managed to build Swift 4.2 latest on Arch Linux. I recorded the steps here https://gist.github.com/segabor/cd08eae2ce8717a1e4f8476687d98221

segabor commented 6 years ago

@kylef I think PR #87 is a good way to go. I simply used --preset=buildbot_linux,no_test parameter to build deployable Swift 4.2 dev package. But on Arch Linux sources had to be patched to use python2.7 before building. So a solution that determines actual python version will be needed in swiftenv also.

kylef / swiftenv