Debian packaging support

[melizasw]

@buttaface suggested that I start a new issue here to discuss problems that arise while I am packaging Swift for Debian. I have found the Fedora packaging that @tachoknight has done to be very helpful in this effort. A related work that I just learned of from @etcwilde is work on packaging for Ubuntu in the swift-installer-scripts repo.

The challenges are mainly getting Swift to build into a .deb package using the debuild tool and getting the packing into compliance with Debian policy. It has a very powerful linting tool that checks for many common errors and issues and most Debian Developers won't sign off on a package that isn't lint clean or reasonably close to it.

Here is the biggest issue that I know of right now. The file ConvertUTF.cpp uses a non-free license and is located in three places:

1. indexstore-db/lib/LLVMSupport/Support/ConvertUTF.cpp
2. llbuild/lib/llvm/Support/ConvertUTF.cpp
3. llvm-project/llvm/lib/Support/ConvertUTF.cpp

Debian suggests replacing these files with libicu, which is already a part of Swift. But there are other alternatives out there as well such as utfcpp. I am not a software engineer by trade so it is unlikely that I could make this replacement on my own without making a mess of things.

Another linting error that I can probably resolve post-bulid on my own is that many libraries have executable permissions. For example: E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/10.0.0/lib/linux/libclang_rt.dyndd-x86_64.so]

And another example of how challenging this packaging can be, I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped. sigh

[melizasw]

My package sponsor has been trying to build swiftlang using a tool called cowbuilder but is experiencing failed builds. Cowbuilder uses a clean environment in /var/cache/pbuilder/ using just the packages listed as build dependencies. I've been able to reproduce the error and it happens at the very end of the build when running some tests.

With cowbuilder:

Failed Tests (2):
  swift-package-tests :: swift-package-init-exec.md
  swift-package-tests :: swift-package-init-lib.md

Testing Time: 5.11s
  Unsupported: 10
  Passed     : 19
  Failed     :  2

With my usual debuild:

Testing Time: 4.19s
   Unsupported: 10
   Passed     : 21

I've attached the cowbuilder log file from where it started the tests until it the end. To me this seems to be the pertinent error:

Input was:
<<<<<<
        1: warning: Failed creating default security location, Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission." 
not:42     !~~~~~~                                                                                                                      error: no match expected
        2: warning: Failed creating default configuration location, Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission." 
        3: warning: Failed creating default cache location, Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission." 
        4: Building for debugging... 
        5: [1/4] Emitting module Project 
        6: [2/4] Compiling Project main.swift 
        7: [5/6] Wrapping AST for Project for debugging 
        8: [6/6] Linking Project 
        9: Build complete! (1.77s) 
>>>>>>

Apparently these two tests are trying to create directories but are failing to do so. I can't tell what directory they are trying to create so that I can figure out why permissions are inadequate. Any help would be appreciated

swiftlang_cowbuild_errors.txt

[finagolfin]

SPM tries to create some directories in the home directory, is that disallowed by cowbuilder?

[melizasw]

SPM tries to create some directories in the home directory, is that disallowed by cowbuilder?

I checked the area where the build took place and there is a /root directory with .bashrc in it and an empty /home directory. Since cowbuilder was ran as sudo I think ~/.swiftpm must have resolved to the /home directory, but since that is empty I would have expected a failure to create directory instead of a permissions error. I'm not familiar enough with cowbuilder to know if there is something I can tweak. Maybe I'll su - before running cowbuilder and see if that works.

[melizasw]

Maybe I'll su - before running cowbuilder and see if that works.

Nope, no change in the error messages. I'm not sure what to make of it.

[finagolfin]

Can you run cowbuilder as a normal user that has a home directory, or won't it work that way?

[melizasw]

I'll have to see if that is possible, it is a good idea. I'm not keen on having to use sudo to do a package build and when I partitioned my drive I wasn't planning to put multiple ~20GB builds into /var. I can only do one build at a time without filling /var, and even then I can't guarantee that I didn't fill up /var as cowbuilder does an rm -rf on the build directory when it detects a failure.

Oh fun.

$ sudo which cowbuilder
/usr/sbin/cowbuilder

I haven't got this tip that I found working yet, but it is very telling:

If you reach the conclusion that your build fails because of some effort to write to HOME, and fixing the software to not do that is unfeasible, then you can set

BUILD\_HOME = /build

in .pbuilderrc or /etc/pbuilderrc 

And in /usr/share/pbuilder/pbuilderrc

# what be used as value for HOME during builds.  See #441052
# The default value prevents builds to write on HOME, which is prevented on
# Debian buildds too.  You can set it to $BUILDDIR to get a working HOME, if
# you need to.
BUILD_HOME=/nonexistent

[melizasw]

I did get cowbuilder to complete successfully by adding BUILD_HOME = /build to /etc/pbuilderrc. It didn't work to put it into ~/.pbuilderrc, if that is what it meant for me to do.

But, it turns out that Debian uses this tool for testing so it must pass without messing with the default settings so I need to disable tests swift-package-init-exec.md and swift-package-init-lib.md. Rather than deleting these files is there a better way such as putting skip-test-swift-package-init-exec.md into build-presets.ini?

[melizasw]

I ended up just deleting the parts of those two tests that parsed the logs for warning and errors. That has got cowbuilder running happily on the default settings so there shouldn't be any issues from that when running builds on Debian servers.

I also got a build made without the Swift provided lldb. This included removing "lldb" and "install-lldb" from build-presets.ini as well as some edits to tests so they wouldn't fail from the missing repl_swift and to point it to the Debian provided lldb. As expected, Swift REPL did not work, but what I did not expect is that normal lldb on a Swift built binary didn't work in that I couldn't set any break points.

(lldb) b 2
error: No selected frame to use to find the default file.
error: No file supplied and no default file available.

I'm sure that is related to removing lldb from the packaging process as the Debian provided lldb used to work with Swift built binaries. Unexpected, but not surprising. I am not planning to debug this and at present have no open issues related to packaging Debian for Swift.

[jblache]

Hi,

@jblache has added DEB control files here https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB for Swift. @melizasw have you seen these?

Yes, @etcwilde had pointed those out to me. The issue isn't how to make a .deb file, but how to make one that complies with Debian Policies and builds on Debian testing. I had a working .deb file back on Swift 5.5.1, but it was miles away from being a completed Debian package that could become an official package. We are nearly there now, but my debian directory contains 32 files to accomplish that. In the eyes of Debian this is the official installer script location.

I'll direct you to previous discussions on the subject for the rationale and the current approach; as we've said all along, help is welcome tackling the packaging, and you seem to have done some good work in that direction.

As far as criteria for inclusion in Debian, having all the files in the right place is necessary but not sufficient; as has been noted in the thread, source duplication and embedded shared libraries are two big issues we're facing here.

In the case of LLVM sources, the likely solution is to have the LLVM package in Debian follow in the footsteps of the GCC package and provide an llvm-X-source package. Swift can then be shipped as a patch against the LLVM sources.

Shared libraries de-embedding is an equally complicated affair, I believe I covered that previously.

[melizasw]

I'm working on splitting up the Swift packing into four. A doc package, a dev package containing headers, a lib package containing runtime libraries, and the main package containing the binaries and everything else that doesn't fit well into the other packages.

So in regards to the doc package, is docc something that should be considered documentation that goes into the doc package or is it better considered a tool that belongs in the main package?

And for the runtime libraries, does this list look correct?

/usr/libexec/swift/lib/swift/linux/libBlocksRuntime.so
/usr/libexec/swift/lib/swift/linux/libFoundation.so
/usr/libexec/swift/lib/swift/linux/libFoundationNetworking.so
/usr/libexec/swift/lib/swift/linux/libFoundationXML.so
/usr/libexec/swift/lib/swift/linux/libXCTest.so
/usr/libexec/swift/lib/swift/linux/lib_InternalSwiftScan.so
/usr/libexec/swift/lib/swift/linux/lib_InternalSwiftSyntaxParser.so
/usr/libexec/swift/lib/swift/linux/libdispatch.so
/usr/libexec/swift/lib/swift/linux/libswiftCore.so
/usr/libexec/swift/lib/swift/linux/libswiftDispatch.so
/usr/libexec/swift/lib/swift/linux/libswiftGlibc.so
/usr/libexec/swift/lib/swift/linux/libswiftRemoteMirror.so
/usr/libexec/swift/lib/swift/linux/libswiftSwiftOnoneSupport.so
/usr/libexec/swift/lib/swift/linux/libswift_Concurrency.so
/usr/libexec/swift/lib/swift/linux/libswift_Differentiation.so
/usr/libexec/swift/lib/swift/linux/libswift_MatchingEngine.so
/usr/libexec/swift/lib/swift/linux/libswift_StringProcessing.so

Thanks

[tbkka]

docc is definitely a tool that belongs in the main package alongside swiftc.

[finagolfin]

lib_InternalSwiftScan.so and lib_InternalSwiftSyntaxParser.so are host libraries for the tools, as tipped off by the _Internal prefix, so I don't think you should include those with the runtime package. Others look good.

[melizasw]

Thanks for the help. I didn't have much time today to work on this, but I did get the four packages build and with nothing but the runtime package installed I was able to run a binary that I had previously built. One point I'm not sure about is if the runtime libraries should be needed for REPL and swiftc as both complained if libswiftCore.so was not present, I'll dig into that a bit more as maybe there are a few more libraries that belong in the main package instead of runtime.

Unfortunately the REPL and swiftc of these new packages are both broken, it seems to be an issue in finding libraries. My guess is that I've missed a few files. When making a single package you just tell it to take everything in directory A and install it to path B, a simple one line command for the whole package. But slicing and dicing up the files between multiple packages means having to use multiple commands to copy the needed files and that introduces lots of places for errors. Hopefully I'll get that sorted out soon.

[finagolfin]

One point I'm not sure about is if the runtime libraries should be needed for REPL and swiftc

The runtime package should be a dependency of the main toolchain package, both because the compiler itself and several other tools link against the runtime libraries and you can't build new Swift executables without linking against the runtime.

[melizasw]

It turns out that my broken packages was due to some missing module.map files from locations such as /usr/lib/swift/CoreFoundation. Forgive my ignorance, should these module.map files go into the dev package with all of the *.h files or into the main package with the binaries?

[etcwilde]

module.modulemap files generally go in the header file with the .h files.

[etcwilde]

CoreFoundation should have it's modulemap files though. Running find . -iname 'module.modulemap' in CoreFoundation

./URL.subproj/module.modulemap
./Parsing.subproj/module.modulemap
./Base.subproj/module.modulemap

Check that those are getting installed with their headers.

[melizasw]

My sponsor tried to upload the Swift package and the Debian servers rejected it due to an embedded libyaml library. The idea is to use Debian's libyaml-dev, but I suspect this won't work as it seems that yams is providing libyaml and yams != libyaml. I'm doing a test build now, but just wanted to confirm my suspicion that I can't replace yams with libyaml and that we will need to override this lint error.

swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/sourcekit-lsp: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/swift-build-sdk-interfaces: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/swift-driver: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/swift-package: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.

[stevapple]

The CYaml target from Yams is identically libyaml, so it's true that every executable target depending on Yams will embed a copy of libyaml. I have tried to build Yams against libyaml-dev (with SwiftPM) and it just works.

I read through the Debian policies and, although it's technically possible to override the tag, it seems this is still likely being rejected because libyaml is already provided by Debian.

Since Yams is owned and maintained by @jpsim, I think it's eventually up to him if CYaml can be replaced by system-wide libyaml. Not too many technical blockers here, but it adds to build-system complexity.

[melizasw]

@stevapple Thanks for explaining it to me. It sounds like I need to modify Yams to pull in the system libyaml instead of building its own CYaml library which is a duplication of libyaml. I haven't got it working yet, but that is what I am trying to do now.

[melizasw]

It took me a while to get both the Swift code and Cmake files changed to pull in the system's libyaml, but it is now working. The swiftlang package is in the NEW queue for Debian Experimental. From here it is a matter of waiting days or weeks until it gets reviewed and either accepted or rejected. And then after that it still needs to get into Unstable before it goes anywhere useful. This is a major milestone and my thanks to everyone that has helped.

[tachoknight]

Congrats @melizasw! Hope it all goes well, will be very cool to apt-get install swiftlang! Out of curiosity, what architectures are you building it for, x864_64 as well as aarch64?

[stevapple]

@melizasw Some questions around libswiftlang:

• Is it required to be extracted into a separate package? I believe so far the community still don’t agree on which part of Swift is runtime, toolchain or SDK except for the established pattern on macOS — and the macOS case is so special that it may not work on other platforms. I wish to delay separate packaging until the issue is fully discussed.
• Is the name libswiftlang the only option in respect of Debian conventions? If feels a little weird to me though.
• I see ./usr/libexec/swift/lib/swift/linux/libXCTest.so in libswiftlang and I would say, on both macOS and Windows we’re not treating XCTest as a runtime library. It is part of the development suite which shouldn’t be leaked into the system.
• If we want to make libswiftlang a usable thing, is ./usr/lib a more suitable place than ./usr/libexec/swift/lib?

---

Update: (some more questions)

• swiftlang: Why is swift not linked into /usr/bin?
• swiftlang-dev: I may come a bit late but is it really necessary to separate this?
• swiftlang-doc: Will it be better if we put man documents here?

[melizasw]

Congrats @melizasw! Hope it all goes well, will be very cool to apt-get install swiftlang! Out of curiosity, what architectures are you building it for, x864_64 as well as aarch64?

Uhhh. Yes? I admit I'm not very familiar with the packaging process yet. But in the past new applications would get built for all architectures for which all dependencies could be met. The swiftlang package is marked as having an architecture of "any" so I think it will be the same way.

[melizasw]

@melizasw Some questions around libswiftlang:

* Is it required to be extracted into a separate package? I believe so far the community still don’t agree on _which part of Swift is runtime, toolchain or SDK_ except for the established pattern on macOS — and the macOS case is so special that it may not work on other platforms. I wish to delay separate packaging until the issue is fully discussed.

It is not required, I did it to provide runtime libraries in a much smaller and lighter package than the full swiftlang package. I do not wish to delay and am willing to move libraries between packages when a formal decision has been made. Worst case is that someone installs libswiftlang only and a Swift binary won't run as it is missing a library that was put into the swiftlang package. In the interim I don't think there is any harm in having internal libraries exposed in libswiftlang unless people think that is an endorsement to link to them for their own binaries.

• Is the name libswiftlang the only option in respect of Debian conventions? If feels a little weird to me though.

No, it could have been swift_lang as it is in Fedora or swiftc as was done with rust. The package name swift is already taken. So I opted to use swiftlang which is the name that was used in the ITP bug that was created in 2015.

• I see ./usr/libexec/swift/lib/swift/linux/libXCTest.so in libswiftlang and I would say, on both macOS and Windows we’re not treating XCTest as a runtime library. It is part of the development suite which shouldn’t be leaked into the system.

I will remove this from the libswiftlang package and put it in the swiftlang package.

• If we want to make libswiftlang a usable thing, is ./usr/lib a more suitable place than ./usr/libexec/swift/lib?

That would be preferred and I will give it a try, but I'm not sure how well the rest of Swift will handle having its libraries moved out of adjacent directories. I've found libswiftlang usable as-is by building a simple binary on Debian and running it on Pop_OS where only libswiftlang was installed.

• swiftlang: Why is swift not linked into /usr/bin?

Debian policy prohibits me from supplying /usr/bin/swift as there already exists one as provided by python3-swiftclient. Maybe we can talk them into picking a new name and freeing up /usr/bin/swift, but until then I cannot provide it. What I have done instead is provide a prompt during the installation of swiftlang to create the link to /usr/bin/swift if it does not already exist and the user selects to create it.

• swiftlang-dev: I may come a bit late but is it really necessary to separate this?

Only one package is necessary, which is the approach taken by @tachoknight for Fedora. My Debian sponsor requested a -dev and -doc package so he gets them

• swiftlang-doc: Will it be better if we put man documents here?

I don't believe so, no. Debian policy requires a man page for each binary in /usr/bin so those man pages have to go with their binaries in swiftlang. They also live in the debian directory of the package so any man pages provided by upstream have to be moved as part of the packaging process.

[jpsim]

Since Yams is owned and maintained by @jpsim, I think it's eventually up to him if CYaml can be replaced by system-wide libyaml. Not too many technical blockers here, but it adds to build-system complexity.

@melizasw @stevapple I'm open to this, just file a PR and we can discuss the specifics.

[stevapple]

No, it could have been swift_lang as it is in Fedora or swiftc as was done with rust. The package name swift is already taken. So I opted to use swiftlang which is the name that was used in the ITP bug that was created in 2015.

Ah, I was explicitly asking for libswiftlang, but now I get the naming at some point.

That would be preferred and I will give it a try, but I'm not sure how well the rest of Swift will handle having its libraries moved out of adjacent directories. I've found libswiftlang usable as-is by building a simple binary on Debian and running it on Pop_OS where only libswiftlang was installed.

My concern is that placing the libraries in /usr/libexec/swift/lib diverges from the current installation (and also the slim Docker image). And since SE-0342 will be a thing (hopefully soon), there should be less need for a runtime package.

Debian policy prohibits me from supplying /usr/bin/swift as there already exists one as provided by python3-swiftclient. Maybe we can talk them into picking a new name and freeing up /usr/bin/swift, but until then I cannot provide it. What I have done instead is provide a prompt during the installation of swiftlang to create the link to /usr/bin/swift if it does not already exist and the user selects to create it.

I see. Asking them to rename seems not so realistic, and your workaround makes sense👍 I wonder if this is a valid use case for update-alternative...

Only one package is necessary, which is the approach taken by @tachoknight for Fedora. My Debian sponsor requested a -dev and -doc package so he gets them

swiftlang-dev arouses new questions to me now:

• Naming. I think libswiftlang-dev should be the correct name, to correctly indicate that the package is used for developing apps that use libswiftlang, not swiftlang (eg. some imaginary Swift IDE).
• Contents. The C++ headers are one of the undecided pieces (on Windows we package it along with SDK, while on macOS both toolchain and SDK hold it). Also, Swift modules should inevitably be a part of development files IMO.

---

Some more thoughts:

• It would be nice have new packages like libswiftlang-dev-arm64-cross to easily enable cross compilation. Actually adopting this feature will reflect some short-sighted designs in toolchain and layout.
• I wonder if it's necessary to extract a swiftlang-common package, and keep swiftlang as simplified as what the user can directly see (and use).
• It's interesting to see what could be in swiftlang-dev — the package that contains development files for the Swift compiler itself.
• Are we going to extract sourcekit-lsp and docc (and even SwiftPM) into separate packages? Depending on use cases, not every Swift user is using these tools. And all these components are pluggable, so it should be fairly easy.🤔

[melizasw]

My concern is that placing the libraries in /usr/libexec/swift/lib diverges from the current installation (and also the slim Docker image). And since SE-0342 will be a thing (hopefully soon), there should be less need for a runtime package.

That was an interesting read. One of the premises of SE-0342 is that Linux does not ship with Swift runtime libraries. This will no longer be true of Debian and its derivatives at some point. It is somewhat circular to say that libswiftlang is not needed due to SE-0342 and SE-0342 is needed (in part) because there are no runtime libraries on Linux. Obviously there is much more to consider with static vs dynamic linking, but giving the option of a runtime library package for now seems like a good idea to me. I'll look into changing that lib path though.

I see. Asking them to rename seems not so realistic, and your workaround makes sense👍 I wonder if this is a valid use case for update-alternative...

I don't think it would be allowed since the requirement for update-alternative is that they provide the same kind of functionality. I initially tried to flag swiftlang and being a conflict with package python3-swiftclient, but that isn't allowed unless they are identically functional but having different implementations. I'm not even sure asking the user to create a dynamic link during install will be permitted, but it makes sense to me and we decided to give it a try.

As for the rest of your comments, I appreciate you taking the time to leave them and I will spend more time later looking into them and thinking about what changes could be made to make the packaging better. But for now, lunch break is over.

[melizasw]

Another significant milestone was reached today. My Swift 5.6.3 package was approved by the Debian FTP team and is now in experimental. That is still not accessible for most people to install, but it means that the FTP team approved of the various packaging methodologies and workarounds that were developed with your help. From here on out there is no manual review by the FTP team so when approved by my sponsor, it can move out of experimental and start making its way out into the world.

You can see some info about the package here: https://tracker.debian.org/pkg/swiftlang On the left side under "binaries" you can click to see more info about each of the four packages and which files that they contain.

For now it is only building on "amd64" and failing or not installable due to missing build dependencies on all other architectures. It will take some time for me to work through the various build logs and see what is wrong. https://buildd.debian.org/status/package.php?p=swiftlang&suite=experimental

Next steps of course include updating the package to 5.9.1. I've not been following along so I don't know yet if Swift 5.9.x requires swift to build or not. Hopefully I can get this out in time for it to get pulled into Ubuntu 24.04 LTS.

[stevapple]

My Swift 5.6.3 package was approved by the Debian FTP team and is now in experimental.

🎉🎉🎉

I've not been following along so I don't know yet if Swift 5.9.x requires swift to build or not.

It does if you’re going to enable macro/regex literal/… and other related features.

Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package. Because Swift in Swift supports only the last minor release, you’ll at least have to:

1. Use the existing Swift 5.6.3 toolchain to build Swift 5.7.3;
2. Use the Swift 5.7.3 toolchain to build Swift 5.8.1;
3. Use the Swift 5.8.1 toolchain to build Swift 5.9.1.

Hopefully I can get this out in time for it to get pulled into Ubuntu 24.04 LTS.

You can always ask for assistance if you wish:)

[melizasw]

Because Swift in Swift supports only the last minor release, you’ll at least have to:

1. Use the existing Swift 5.6.3 toolchain to build Swift 5.7.3;

2. Use the Swift 5.7.3 toolchain to build Swift 5.8.1;

3. Use the Swift 5.8.1 toolchain to build Swift 5.9.1.

Thanks for the roadmap. Once out of experimental and into unstable I can start working my way up the versions. My sponsor says that we can leave experimental whenever I feel the package is ready, and I think that will be after I fix the non-amd64 builds that should be working, especially arm64.

You can always ask for assistance if you wish:)

Ok. I've been working on the failures to build on other architectures. Some fail because Swift does not support them:

File "/<<PKGBUILDDIR>>/swift/utils/swift_build_support/swift_build_support/targets.py", line 348, in host_target
    raise NotImplementedError('System "%s" with architecture "%s" is not '
NotImplementedError: System "Linux" with architecture "armv8l" is not supported

And the others fail because -fstack-clash-protection is being called on an architecture in which Clang 13.0 does not support that option:

-- Build files have been written to: /<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64
+ popd
/<<PKGBUILDDIR>>
<string>:1: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
+ /usr/bin/cmake --build /<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64 -- -j4 all
[1/82][  1%][0.084s] Building C object src/CMakeFiles/dispatch.dir/apply.c.o
FAILED: src/CMakeFiles/dispatch.dir/apply.c.o 
/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-aarch64/bin/clang -DDISPATCH_USE_DTRACE=0 -DHAVE_CONFIG_H -D_GNU_SOURCE=1 -Ddispatch_EXPORTS -I/<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64 -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src -I/<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64/src -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/private -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src/BlocksRuntime -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -O3 -DNDEBUG -std=gnu11 -fPIC -fvisibility=hidden -Werror -Wall -Wextra -Warray-bounds-pointer-arithmetic -Wassign-enum -Watomic-properties -Wcomma -Wconditional-uninitialized -Wconversion -Wcovered-switch-default -Wdate-time -Wdeprecated -Wdocumentation -Wdouble-promotion -Wduplicate-enum -Wexpansion-to-defined -Wfloat-equal -Widiomatic-parentheses -Winfinite-recursion -Wmissing-prototypes -Wnewline-eof -Wnullable-to-nonnull-conversion -Wobjc-interface-ivars -Wover-aligned -Wpacked -Wpointer-arith -Wselector -Wshadow -Wshorten-64-to-32 -Wsign-conversion -Wstatic-in-inline -Wsuper-class-method-mismatch -Wswitch -Wunguarded-availability -Wunreachable-code -Wunused -Wno-unknown-warning-option -Wno-trigraphs -Wno-four-char-constants -Wno-disabled-macro-expansion -Wno-pedantic -Wno-bad-function-cast -Wno-c++-compat -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-cast-align -Wno-cast-qual -Wno-documentation-unknown-command -Wno-format-nonliteral -Wno-missing-variable-declarations -Wno-old-style-cast -Wno-padded -Wno-reserved-id-macro -Wno-shift-sign-overflow -Wno-undef -Wno-unreachable-code-aggressive -Wno-unused-macros -Wno-used-but-marked-unused -Wno-void-pointer-to-int-cast -Wno-vla -Wno-error=assign-enum -fno-exceptions -fblocks -MD -MT src/CMakeFiles/dispatch.dir/apply.c.o -MF src/CMakeFiles/dispatch.dir/apply.c.o.d -o src/CMakeFiles/dispatch.dir/apply.c.o -c /<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src/apply.c
clang-13: error: argument unused during compilation: '-fstack-clash-protection' [-Werror,-Wunused-command-line-argument]

It seems to be happening when building libdispatch. The compile is being called with -fstack-clash-protection for some reason. I am still working to figure out where this is coming from so that I can address it. Any help on this would be appreciated.

[stevapple]

Some fail because Swift does not support them:

File "/<<PKGBUILDDIR>>/swift/utils/swift_build_support/swift_build_support/targets.py", line 348, in host_target
    raise NotImplementedError('System "%s" with architecture "%s" is not '
NotImplementedError: System "Linux" with architecture "armv8l" is not supported

For this specific case, ARMv8L is not likely to be supported, because Swift doesn't support 32-bit host. I believe it's supported as a target (i.e. for libswiftlang).

[melizasw]

Swift 5.6.3 list the following valid hosts for Linux:

    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts. However, the only hosts from this list that Debian supports are x86_64, i686, aarch64, powerpc64, and s390x (powerpc and riscv64 as well). So I think I can just restrict my package to building on these 5 architectures.

x86_64 builds i686 fails because it needs libatomic aarch64 fails because its build is using the unsupported -fstack-clash protection powerpc64 is also missing libatomic as well as whatever provides gnu_f2h_ieee, gnu_h2f_ieee, and __truncdfhf2 s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

I'm working to fix these, but any help would be appreciated.

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

[stevapple]

Swift 5.6.3 list the following valid hosts for Linux:

    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts.

IIRC 32-bit host support was dropped long before (or hadn’t ever existed). Latest PRs that add new architectures show they’re adding support for targets that are meant to be cross-compiled against. 32-bit ARMv8 host triples should be handled similarly, while 64-bit ones can be mapped into aarch64.

s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

Maybe the port is not maintained any more and should be removed?

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

powerpc is target-only IIRC. riscv64 and other newly added architectures should be cross-compiled first. Not sure if the Debian packaging team will allow though. BTW this port doesn’t seem to be complete, so you can skip for now.

[stevapple]

Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package.

@melizasw In fact, you can tweak the process to first build a toolchain without Swift in Swift support, and then build another one with it. This still works as for Swift 5.9, but will be removed at some time.

[melizasw]

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Debian does indeed support the addition of new architectures and I was able to find some documentation on bootstrapping and cross-building which may be needed for riscv64 support. But for now I think I'm going to just focus on x86_64 and aarch64 and maybe s390x (Debian's LLVM toolchain builds for s390x).

Thanks for the help.

[tachoknight]

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

[melizasw]

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

Yep, I ran into the same problem and added to your bug report from 3+ years ago. There was some activity on the bug 2 months ago, but no fix yet. I guess no s390x for now.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

It'll take a while before I get there, but I will let you know.

On a RPI 4, it takes around 24 hours. 🙃

That beats the 2.5 days it took to build through QEMU! I ordered an RPI 5 that will be here in a few days to help out with that problem. I've also got a LicheePi 4A arriving later today to help me with building for riscv64 which I have been making some progress on by porting patches from @futurejones to 5.6.3.

[melizasw]

Great news - Swift is now on Debian unstable (sid) and in a few days should go out to testing (trixie) on amd64 and arm64. https://tracker.debian.org/pkg/swiftlang

Unfortunately when Ubuntu pulled it in and attempted to build on Noble it failed. https://launchpad.net/ubuntu/+source/swiftlang

The issue appears to me to be related to the gold linker

[4402/4423][ 99%][4529.955s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4403/4423][ 99%][4529.991s] Linking CXX static library lib/libgtest.a
[4404/4423][ 99%][4530.020s] Linking CXX static library lib/libLLVMTestingSupport.a
[4405/4423][ 99%][4530.061s] Creating directories for 'compiler-rt'
[4406/4423][ 99%][4530.071s] No download step for 'compiler-rt'
[4407/4423][ 99%][4530.082s] No update step for 'compiler-rt'
[4408/4423][ 99%][4530.084s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
[4409/4423][ 99%][4530.094s] No patch step for 'compiler-rt'
[4409/4423][ 99%][4530.094s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - broken
CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL'

    Run Build Command(s): /usr/bin/ninja -v cmTC_d2d03
    [1/2][ 50%][0.017s] /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang   -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -MD -MT CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -c /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL/testCCompiler.c
    clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
    [2/2][100%][0.030s] : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    FAILED: cmTC_d2d03 
    : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    /usr/bin/ld: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
    clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:10 (project)

-- Configuring incomplete, errors occurred!
[4415/4423][ 99%][4530.818s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o
FAILED: tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure 
cd /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -DCMAKE_C_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_CXX_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ -DCMAKE_ASM_COMPILER=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DLLVM_CONFIG_PATH=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/llvm-config "-DLLVM_LIT_ARGS=-v --time-tests -j 4" -DCOMPILER_RT_OUTPUT_DIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./lib/clang/13.0.0 -DCOMPILER_RT_EXEC_OUTPUT_DIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin -DCOMPILER_RT_INSTALL_PATH:PATH=lib/clang/13.0.0 -DCOMPILER_RT_INCLUDE_TESTS=ON -DCMAKE_INSTALL_PREFIX=/usr -DLLVM_LIBDIR_SUFFIX= -DLLVM_RUNTIME_OUTPUT_INTDIR=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin -DCMAKE_OSX_DEPLOYMENT_TARGET= -DCMAKE_OSX_SYSROOT:PATH= -DCOMPILER_RT_INTERCEPT_LIBDISPATCH=ON -DCOMPILER_RT_PREFIX=/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/projects/compiler-rt -DCOMPILER_RT_SRC_ROOT=/<<PKGBUILDDIR>>/llvm-project/llvm/../compiler-rt -GNinja -S /<<PKGBUILDDIR>>/llvm-project/llvm/../compiler-rt -B /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -E touch /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps/compiler-rt-configure
[4417/4423][ 99%][4531.167s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o
ninja: build stopped: subcommand failed.
-- Warning: {} Host toolchain could not locate a compiler to build swift-driver. (Try `--skip-early-swift-driver`)
--- Building cmark ---
Building the standard library for: swift-stdlib-linux-x86_64
Running Swift tests for: check-swift-only_early_swiftdriver-linux-x86_64 check-swift-all-linux-x86_64 check-swift-all-optimize-linux-x86_64
ERROR: command terminated with a non-zero exit status 1, aborting

The same build on Debian produces the following build log:

[4405/4423][ 99%][2053.044s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4406/4423][ 99%][2053.094s] Linking CXX static library lib/libgtest.a
[4407/4423][ 99%][2053.137s] Linking CXX static library lib/libgtest_main.a
[4408/4423][ 99%][2053.138s] Linking CXX static library lib/libLLVMTestingSupport.a
[4409/4423][ 99%][2053.157s] Creating directories for 'compiler-rt'
[4410/4423][ 99%][2053.175s] No download step for 'compiler-rt'
[4411/4423][ 99%][2053.193s] No update step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] No patch step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for unwind.h
-- Looking for unwind.h - found
-- Looking for rpc/xdr.h
-- Looking for rpc/xdr.h - not found
CMake Warning at cmake/Modules/CompilerRTUtils.cmake:352 (message):
  llvm-config finding testingsupport failed with status 1
Call Stack (most recent call first):
  CMakeLists.txt:81 (load_llvm_config)

I'm not sure why Debian is skipping the check but Ubuntu is not and then failing it. My Debian build doesn't have LLVMgold.so so no surprise that Ubuntu doesn't have it either.

I'm going to try the following to see if they help: 1) Add binutils-dev to the build dependencies 2) patch llvm-project-orig/llvm/cmake/config-ix.cmake as @tachoknight did for his Fedora 5.6.3 build to set LLVM_BINUTILS_INCDIR to /usr/include (Debian LLVM 16 also sets this path for its build).

Any help or advice would be appreciated as at the moment I am not set up to run test builds on Ubuntu and cannot directly test these without first releasing them to Debian then waiting for Ubuntu to pull it in and build.

[melizasw]

More good news. Swift has now migrated to Debian testing and the two fixes mentioned above did indeed get the Ubuntu Noble build past that hang-up. I drug an old system out of retirement and installed Ubuntu on it so that I could work through the build errors on Noble.

Unfortunately now I'm getting another error, something about trying to link a shared library and needed to compile with -fPIC.

[770/863][ 89%][27.919s] Linking CXX shared library swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so
FAILED: swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so 

/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: warning: relocation against `__gwp_asan_default_options' in read-only section `.text._ZN8gwp_asan7options11initOptionsEPKcPFvS2_zE'
/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

That doesn't seem too hard, I just need to find the right place to do so when I get more time.

[futurejones]

@melizasw I have installed and tried to test your debian packages but they don't work. I am just trying to run a simple "hello world" using the swift package manager. The errors are very easy to replicate.

The test environment is a Debian Trixie docker container.

docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it debian:trixie /bin/bash

apt update

apt install swiftlang

mkdir hello && cd hello && swift package init --type executable && swift run

root@d5e164273bb5:/# mkdir hello && cd hello && swift package init --type executable && swift run
Creating executable package: hello
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/hello/main.swift
Creating Tests/
Creating Tests/helloTests/
Creating Tests/helloTests/helloTests.swift
'hello': error: invalidManifestFormat("<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"CoreFoundation.h\"\n        
 ^\n/usr/libexec/swift/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: \'sys/types.h\' file not found\n#include <sys/types.h>\n         
^\n/usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module \'CDispatch\' imported from /usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:\n#include <dispatch/dispatch.h>\n         
^\n<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"dispatch.h\"\n         ^\n/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:10: note: in file included from 
/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:\n#include <os/generic_unix_base.h>\n  
^\n/usr/libexec/swift/lib/swift/os/generic_unix_base.h:24:10: 
error: \'sys/param.h\' file not found\n#include <sys/param.h>\n         
^\n<unknown>:0: error: could not build C module \'CoreFoundation\'", diagnosticFile: nil)
root@d5e164273bb5:/hello# 

This error also occurs when using Debian Unstable.

When trying to build using swiftc we get an another error.

root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift 
error: link command failed with exit code 1 (use -v to see invocation)
clang-13: error: invalid linker name in argument '-fuse-ld=gold'

This indicates that binutils is missing from the required dependancies. But after installing binutils we get another error.

root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift 
error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: error: cannot open Scrt1.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crti.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtbeginS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtendS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtn.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
/usr/bin/ld.gold: error: cannot find -lc
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

I have filed a bug with Debian to track the issue. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057117

[melizasw]

@futurejones this is due to missing dependencies on binutils and gcc. If you install these two packages it will get you up and running for the next few days until I can get a new release out.

[futurejones]

@melizasw there appears to be a problem with the relationship between the different swiftlang packages.

The swiftlang package is listing the swiftlang-dev package as a dependency, whereas the swiftlang-dev package only lists swiftlang asrecommends`.

The swiftlang-dev package should list both swiftlang and libswiftlang as dependencies. https://packages.debian.org/trixie/swiftlang-dev

The swiftlang package should not list swiftlang-dev as a dependency. https://packages.debian.org/trixie/swiftlang

[melizasw]

@futurejones I can certainly do that. Can you help me understand though what use swiftlang is without swiftlang-dev? I was under the impression that swiftlang wasn't of much use without both libswiflang and swiftland-dev.

[etcwilde]

So first, I'm really happy to see this work making progress. I think this will really help with the Swift-on-Linux story.

For some ideas on the separation of the libraries and tools, it might be worth looking at the Windows installer. I don't remember the exact split but it does a good job of separating concerns for developers, CI, and end-consumers of Swift programs. You shouldn't need to download the compiler if you just want to run a pre-compiled program that happens to be written in Swift, and you shouldn't need to download the headers for CoreFoundation if you just want to build a hello-world, for instance. It looks like that separation is mostly covered with the separation of swiftlang and libswiftlang, though perhaps could be split further for the stdlib and corelibs.

I may have missed a few runtime libraries in the listing here, so worth double-checking, but I think it's reasonable breakdown:

Runtimes:

• swift runtimes (both with and without swiftmodules/header files (dev and non-dev versions))
  • swiftCore
  • swiftGlibc
  • swiftRemoteMirror
  • swiftSwiftOnoneSupport
  • swift_Concurrency
  • swift_Differentiation
  • swift_MatchingEngine
  • swift_StringProcessing
  • libdispatch*
• corelibs
  • blocksRuntime
  • Foundation
  • FoundationNetworking
  • FoundationXML

* The challenge with libdispatch is that concurrency depends on it. The C bits of corelibs-libdispatch actually get built twice in the toolchain build, so I wonder if it would make sense to configure the build to emit a static archive so that we don't expose that. Then libdispatch.so could be installed as part of the corelibs package.

We should keep the runtime libraries together installed as one. I don't have a strong opinion on the corelibs libraries being together or apart, though mildly in favor of keeping them together as a swiftlang-corelibs package. Then we would want a dev variant of the swiftlang runtimes and swiftlang corelibs packages containing swiftmodules and headers. I think that keeping the corelibs separate from the runtimes is important, especially as work on swift-Foundation progresses, I'm not entirely sure what the future of corelibs-Foundation looks like.

Looking at swiftlang-dev, I'm concerned about the libcxx headers getting installed? If folks are doing C++ interop, which I don't think Swift 5.6 supports anyway, those headers will need to match the C++ runtime they're building for, which natively would be libstdc++ on the system.

Build Tools: (what I assume is the umbrella swiftlang package)

• swiftc
• swiftpm
• clang

The swift-clang will be an interesting tricky bit. We'll want to provide the special swift-clang that has support for swift and swift-async calling conventions so that folks can use the C and C++ interop features. That of course conflicts with the existing clang package. I think the provides mechanism might work here though? A swift-clang package provides clang perhaps?

My concern with naming clang swift-clang is that various build-scripts may try to use clang while consuming a header emitted by Swift. Those headers may contain functions with swift or swiftasync calling conventions, so if they use the wrong C/C++ compiler, they will get a warning about the unknown attribute, but the program will compile with a misaligned ABI. On a good day, that will crash their program. On a bad day, their program will look like it runs, but has totally random garbage floating around. I'm open to push-back on that.

swiftlang can probably drop the headers and swiftmodules, and instead install those in the runtime-dev package. It's true that the compiler isn't terribly useful without them, but it's more conventional to install the headers associated with the library in the <library>-dev package. See the gcc vs libc6 vs libc6-dev filelists, for example. gcc doesn't come with headers, but libc6-dev comes with the C headers.

Dev Tools:

• lldb
• sourcekit-lsp
• other tools and files useful for development but not strictly necessary for running or building a program

Same sort of situation with the swift-lldb as with clang. The lldb from apple/llvm-project has extra bits to make debugging Swift work, but that will conflict with the existing lldb package. Probably the same resolution since it provides lldb + swift extras.

Anyway, my initial thoughts. Thank you for working on this.

[melizasw]

Despite what was said above about building Swift 5.7.3 using Swift 5.6.3, I tried without Swift and was able to build 5.7.3 just fine and am in the process of getting that pushed out into Debian. In the mean time I decided to build Swift 5.8.1 without using Swift 5.7.3, which I think should be possible as @tachoknight did so for Fedora. However, I got this bootstrapping error quite similar to what @stevapple got nearly 2 years ago in Windows - https://github.com/swiftlang/swift/issues/61410

Here is my version of the error, or at least the start of about 5,000 lines of them.

[1215/1425][ 85%][1092.863s] Building swift module Basic FAILED: bootstrapping1/SwiftCompilerSources/Basic.o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.o cd /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources && /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping0/bin/swiftc -c -o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.o -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping0/bin/../lib -I /usr/lib -target x86_64-pc-linux-gnu -module-name Basic -emit-module -emit-module-path /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.swiftmodule -parse-as-library /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources/Sources/Basic/SourceLoc.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources/Sources/Basic/Utils.swift -wmo -Xfrontend -validate-tbd-against-ir=none -Xfrontend -enable-experimental-cxx-interop -Xcc -UIBOutlet -Xcc -UIBAction -Xcc -UIBInspectable -Xfrontend -disable-implicit-string-processing-module-import -O -cross-module-optimization -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-x86_64/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-x86_64/tools/clang/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/SwiftCompilerSources/../include -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources

:0: warning: unable to perform implicit import of "_Concurrency" module: no such module found /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include/swift/AST/ClangModuleLoader.h:19:10: note: while building module 'Clang_AST' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include/swift/AST/ClangModuleLoader.h:19: #include "clang/AST/DeclTemplate.h" ^ /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/AST/APValue.h:16:10: note: while building module 'Clang_Basic' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/AST/APValue.h:16: #include "clang/Basic/LLVM.h" ^ /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/Basic/Sanitizers.h:20:10: note: while building module 'LLVM_Transforms' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/Basic/Sanitizers.h:20: #include "llvm/Transforms/Instrumentation/AddressSanitizerOptions.h" ^ /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h:20:10: note: while building module 'LLVM_intrinsic_gen' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h:20: #include "llvm/IR/PassManager.h" ^ :1:10: note: in file included from :1: #include "IR/Argument.h" ^ /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Argument.h:17:10: note: in file included from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Argument.h:17: #include "llvm/IR/Attributes.h" ^ /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Attributes.h:30:10: note: in file included from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Attributes.h:30: #include ^ /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/set:62:10: note: in file included from /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/set:62: #include ^ /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_tree.h:2005:5: error: missing '#include "gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_pair.h"'; 'pair' must be declared before it is used pair

I can't figure out what is causing this error. Is there a way around it or do I need to give up and build 5.8.1 with 5.7.3?