Debian packaging support

[melizasw]

@buttaface suggested that I start a new issue here to discuss problems that arise while I am packaging Swift for Debian. I have found the Fedora packaging that @tachoknight has done to be very helpful in this effort. A related work that I just learned of from @etcwilde is work on packaging for Ubuntu in the swift-installer-scripts repo.

The challenges are mainly getting Swift to build into a .deb package using the debuild tool and getting the packing into compliance with Debian policy. It has a very powerful linting tool that checks for many common errors and issues and most Debian Developers won't sign off on a package that isn't lint clean or reasonably close to it.

Here is the biggest issue that I know of right now. The file ConvertUTF.cpp uses a non-free license and is located in three places:

1. indexstore-db/lib/LLVMSupport/Support/ConvertUTF.cpp
2. llbuild/lib/llvm/Support/ConvertUTF.cpp
3. llvm-project/llvm/lib/Support/ConvertUTF.cpp

Debian suggests replacing these files with libicu, which is already a part of Swift. But there are other alternatives out there as well such as utfcpp. I am not a software engineer by trade so it is unlikely that I could make this replacement on my own without making a mess of things.

Another linting error that I can probably resolve post-bulid on my own is that many libraries have executable permissions. For example: E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/10.0.0/lib/linux/libclang_rt.dyndd-x86_64.so]

And another example of how challenging this packaging can be, I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped. sigh

[finagolfin]

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.

Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

[melizasw]

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.

Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

That was a great idea. The llvm-toolchain-14 package has 840 lint errors including this one: llvm-toolchain-14 | 1:14.0.6-2 |   | error | license-problem-convert-utf-code | [llvm/lib/Support/ConvertUTF.cpp]

I am able to see what they put in their copyright file and will do the same thing. The lint error will just have to be overridden. I see a lot of familiar lint errors in that llvm package so maybe this won't be as big of an issue as I had thought.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

I'm sure they do. I was just complaining how this turns into a three step process of disabling the stripping, disabling the lint errors that result, then probably having to explain to a Debian Developer why stripping was disabled.

Once I have the low hanging lint errors cleaned up I'll start working on getting the remaining errors reviewed by a Debian Developer to see if they have any concerns that I need to address.

[melizasw]

Here is something else that would help me. Anything that lintian flags as an error is considered to be either a bug in the lint tool or a Debian policy violation. That said, it seems like there is some understanding that it isn't always that simple and overrides of rules are allowed.

For the following lintian errors it would be very helpful to know if the error is something I should be able to fix, or is expected. And if expected, what is the reason why. Any explanation that I can add to the comments of the lintian overrides will help to get approval to add the package to Debian. Most errors have multiple failures, but only the first is shown for brevity.

E: swift source: binary-with-bad-dynamic-table [llvm-project/llvm/test/tools/llvm-xray/X86/Inputs/elf64-objcopied-instrmap.bin]
N: 
N:   This appears to be an ELF file. According to readelf, the program headers
N:   suggests it should have a dynamic section, but readelf cannot find it.
N:   
N:   If it is meant to be external debugging symbols for another file, it
N:   should be installed under /usr/lib/debug. Otherwise, this could be a
N:   corrupt ELF file.

E: swiftlang: custom-library-search-path RUNPATH swift_5.6.2/build/buildbot_linux/llvm-linux-x86_64/lib [usr/libexec/swift/bin/lldb-argdumper]
N: 
N:   The binary or shared library sets RPATH or RUNPATH. This overrides the
N:   normal library search path, possibly interfering with local policy and
N:   causing problems for multilib, among other issues.
N:   
N:   The only time a binary or shared library in a Debian package should set
N:   RPATH or RUNPATH is if it is linked to private shared libraries in the
N:   same package. In that case, place those private shared libraries in
N:   /usr/lib/*package*. Libraries used by binaries in other packages should be
N:   placed in /lib or /usr/lib as appropriate, with a proper SONAME, in which
N:   case RPATH/RUNPATH is unnecessary.
N:   
N:   To fix this problem, look for link lines like: gcc test.o -o test
N:   -Wl,--rpath,/usr/local/lib or gcc test.o -o test -R/usr/local/lib and
N:   remove the -Wl,--rpath or -R argument. You can also use the chrpath
N:   utility to remove the RPATH.
N: 
N:   Please refer to https://wiki.debian.org/RpathIssue for details.

This should be pretty easy for me to fix if this is just an artifact of packaging and not a responsibility of the swift build scripts.

E: swift source: missing-notice-file-for-apache-license [swift-crypto/NOTICE.txt]
N: 
N:   The package appears to be licensed under the Apache 2.0 license and a
N:   NOTICE file (or similar) exists in the source tree. However, no files
N:   called NOTICE or NOTICE.txt are installed in any of the binary packages.
N:   
N:   The Apache 2.0 license requires distributing of such files:
N:   
N:    (d) If the Work includes a "NOTICE" text file as part of its
N:        distribution, then any Derivative Works that You distribute must
N:        include a readable copy of the attribution notices contained
N:        within such NOTICE file [..]
N:   
N:   Please include the file in your package, for example by adding
N:   path/to/NOTICE to a debian/package.docs file.
N: 
N:   Please refer to usr/share/common-licenses/Apache-2.0 for details.

E: swiftlang: no-code-sections [usr/libexec/swift/lib/swift_static/linux/libicudataswift.a]
N: 
N:   
N:   The named members of the static library have no usable code sections.
N:   
N:   It happens when shared objects are built with -flto=auto but without
N:   -ffat-lto-objects. dh_strip strips the LTO sections but may leave the
N:   static library without any usable code.
N: 
N:   Please refer to Bug#977596 for details.

These don't need to be executable do they? I'm not even sure just how they are getting +x.

E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/13.0.0/lib/linux/libclang_rt.asan-x86_64.so]
N: 
N:   Shared libraries should be mode 0644.
N: 
N:   Please refer to Run-time shared libraries (Section 8.1) in the Debian
N:   Policy Manual for details.

There are many of this kind. I'm not sure what the issue is as they don't go into the package. Maybe there is a way to clean them up just prior to running lintian?

E: swift source: source-is-missing [cmake/Tests/CMakeTests/ELF/elf32lsb.bin]
N: 
N:   The source of the following file is missing. Lintian checked a few
N:   possible paths to find the source, and did not find it.
N:   
N:   Please repack your package to include the source or add it to
N:   "debian/missing-sources" directory.
N:   
N:   Please note, that very-long-line-length-in-source-file tagged files are
N:   likely tagged source-is-missing. It is a feature not a bug.

This is just a warning, but it makes me wonder if I built Swift incorrectly? Maybe it doesn't matter as this is pertaining to the debug symbol package (swiftlang-dbgsym_5.6.2-1_amd64.deb) and I don't think that would be getting added.

W: swiftlang-dbgsym: debug-file-with-no-debug-symbols [usr/lib/debug/.build-id/00/bb2a059d939872ab355e8fa581567eac042f7d.debug]
N: 
N:   The binary is installed as a detached "debug symbols" ELF file, but it
N:   does not appear to have debug information associated with it.
N:   
N:   A common cause is not passing -g to GCC when compiling.
N:   
N:   Implementation detail: Lintian checks for the ".debug_line" and the
N:   ".debug_str" sections. If either of these are present, the binary is
N:   assumed to contain debug information.
N: 
N:   Please refer to Bug#668437 for details.

If anyone can help make sense of any of these lintian issues that would be most helpful.

[finagolfin]

• The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.
• The RUNPATH issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.
• The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.
• The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but ~will~may require you to bump the Swift package anytime the system libicu is updated.
• Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.
• I have no idea about that debug file with no symbols, haven't used that.

[melizasw]

• The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.

I looked up Debian's llvm-toolkit-14 package and wouldn't you know it, they have these same lintian errors and many of the others that I am seeing. If they can have them in llvm then swift should be allowed them as well. The lint tool looks at both the completed package as well as the source, so it seems to pick up all sorts of files that shouldn't matter.

* The `RUNPATH` issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.

I'm going to look into this some more, but maybe it isn't an issue. Based on the Debian wiki this forces the use of the custom Swift libraries, which I presume is better than picking up the system's libraries for llvm and others. Again, the Debian llvm package has these errors as well so I will contact the maintainer to find out what I can.

* The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.

For now I'm going to let the Debian build scripts copy over the file and see if that resolves things.

* The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but ~will~may require you to bump the Swift package anytime the system libicu is updated.

Well, wouldn't you know it. Debian's libicu-dev package has the exact same error on libicudata.a. So I guess nothing to worry about there.

* Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.here

I was able to get those fixed by manually running chmod at the appropriate place in the Debian packaging flow. There is a tool that is supposed to fix this automatically for me so maybe it has a bug. I will have to look into that. At no point in the build process do I see the swift tools setting these files to anything but 0644 and they don't become 0755 until they are copied out into the final directory structure so I'm not sure yet who is doing it.

* I have no idea about that debug file with no symbols, haven't used that.

Yah, probably not an issue.

I've just realized that the person that originally wanted to package Swift for Debian back in 2015 was and is a Debian Developer with the LLVM packaging team. If I can pique his interest to sponsor/mentor this package then he already has half of the answers and might even jump at the opportunity to get involved with it again. Crossing my fingers!

[melizasw]

I'm at the point of trying to get a Debian Developer to sponsor my package and clean up any policy violations. The violation that was discovered first is that there is an existing Debian packaged named "python3-swiftclient" and they supply a binary named "swift". Since their binary does something completely different I am not allowed to have a binary named "swift" no matter how unlikely it is for someone to want to use the swift that is designed for server object storage and also compile Swift programs on that same server. I could use the "conflict" mechanism to prevent my package and theirs from being installed at the same time, but policy does not allow that unless the binaries provide the same functionality.

The recommended action is to contact the debian-devel mailing list to start a discussion of who has to change their name, or if both have to change their name. At the same time I could ask for an exception to the "conflict" mechanism rule.

I guess my main question here is that since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead? Would that break anything other than reams of documentation?

[finagolfin]

since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead?

swift has two purposes: it invokes the REPL when run alone and it interprets a Swift file when given one. As such, you could always drop it as long as you don't care about those two uses, or look into renaming it instead.

Would that break anything other than reams of documentation?

No, I doubt there's even that much doc on it.

[melizasw]

I don't know yet if this will be allowed by the Debian policy, but what I have done is create a set of scripts such that during the installation process if /usr/bin/swift does not exist the user is presented with a prompt asking if they wish to add it or not. If an actual binary file is located at /usr/bin/swift instead of a symbolic link then a message is printed out warning that "swift" already exists and it cannot be used to call the Swift REPL or compiler/interpreter.

[melizasw]

I've been copying /usr/bin/ls to /usr/bin/swift to test the Debian package install in the case where the "swift" package is installed. If I forget to remove this copy before building Swift then it tries to build newswiftdriver with /usr/bin/swift and fails with a less than helpful message. I'm not sure what all I can do on my end to address this (and may not need to do anything), but I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

[finagolfin]

I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

[melizasw]

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

I tried to reproduce this error on a Swift 5.5.1 build, but was unable to. Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case):

swift/utils/build-script --preset=buildbot_linux,no_test install_destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 installable_package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz
[swift/utils/build-script] NOTE: using preset "buildbot_linux,no_test", which expands to 

swift/utils/build-script --assertions --swift-enable-ast-verifier=0 --no-swift-stdlib-assertions '--swift-install-components=autolink-driver;compiler;clang-resource-dir-symlink;stdlib;swift-remote-mirror;sdk-overlay;parser-lib;toolchain-tools;license;sourcekit-inproc' '--llvm-install-components=llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld;LTO' --llbuild --swiftpm --swift-driver --xctest --libicu --libcxx --swiftdocc --build-ninja --install-llvm --install-swift --install-lldb --install-llbuild --install-swiftpm --install-swift-driver --install-xctest --install-libicu --install-prefix=/usr --install-libcxx --install-sourcekit-lsp --install-swiftdocc --build-swift-static-stdlib --build-swift-static-sdk-overlay --build-swift-stdlib-unittest-extra --test-installable-package --toolchain-benchmarks --install-destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 --installable-package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz --relocate-xdg-cache-home-under-build-subdir --build-subdir=buildbot_linux --lldb --release --test --validation-test --long-test --stress-test --test-optimized --foundation --libdispatch --indexstore-db --sourcekit-lsp --swiftdocc '--lit-args=-v --time-tests' --lldb-test-swift-only --install-foundation --install-libdispatch --reconfigure --skip-test-cmark --skip-test-lldb --skip-test-swift --skip-test-llbuild --skip-test-swiftpm --skip-test-swift-driver --skip-test-xctest --skip-test-foundation --skip-test-libdispatch --skip-test-playgroundsupport --skip-test-libicu --skip-test-indexstore-db --skip-test-sourcekit-lsp --skip-test-swiftdocc

[swift/utils/build-script] NOTE: Using toolchain default
+ cmake --version
+ mkdir -p /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux
+ rm -rf /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ cp -r /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/ninja /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ pushd /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ env CXX=/usr/bin/clang++ /usr/bin/python3 configure.py --bootstrap
[1/29][  3%][0.045s] INLINE build/browse_py.h
[2/29][  6%][0.057s] CXX build/debug_flags.o
[3/29][ 10%][0.635s] CXX build/browse.o
[4/29][ 13%][0.787s] CXX build/depfile_parser.o
[5/29][ 17%][0.971s] CXX build/disk_interface.o
[6/29][ 20%][1.180s] CXX build/clparser.o
[7/29][ 24%][1.208s] CXX build/edit_distance.o
[8/29][ 27%][1.449s] CXX build/dyndep.o
[9/29][ 31%][1.652s] CXX build/deps_log.o
[10/29][ 34%][2.033s] CXX build/clean.o
[11/29][ 37%][2.073s] CXX build/line_printer.o
[12/29][ 41%][2.137s] CXX build/dyndep_parser.o
[13/29][ 44%][2.163s] CXX build/build_log.o
[14/29][ 48%][2.252s] CXX build/lexer.o
[15/29][ 51%][2.430s] CXX build/graphviz.o
[16/29][ 55%][2.691s] CXX build/metrics.o
[17/29][ 58%][2.967s] CXX build/version.o
[18/29][ 62%][3.019s] CXX build/eval_env.o
[19/29][ 65%][3.108s] CXX build/string_piece_util.o
[20/29][ 68%][3.210s] CXX build/parser.o
[21/29][ 72%][3.246s] CXX build/util.o
[22/29][ 75%][3.353s] CXX build/graph.o
[23/29][ 79%][3.606s] CXX build/subprocess-posix.o
[24/29][ 82%][3.873s] CXX build/manifest_parser.o
[25/29][ 86%][3.964s] CXX build/build.o
[26/29][ 89%][4.120s] CXX build/state.o
[27/29][ 93%][4.192s] AR build/libninja.a
[28/29][ 96%][4.692s] CXX build/ninja.o
[29/29][100%][4.813s] LINK ninja
bootstrapping ninja...
warning: A compatible version of re2c (>= 0.11.3) was not found; changes to src/*.in.cc will not affect your build.
wrote build.ninja.
bootstrap complete.  rebuilding...
+ popd
--- Building earlyswiftdriver ---
Traceback (most recent call last):
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 722, in <module>
    sys.exit(main())
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 717, in main
    return main_normal()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 673, in main_normal
    invocation.execute()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 669, in execute
    self._execute(pipeline, all_host_names)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 727, in _execute
    self.execute_product_build_steps(product_class, host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 783, in execute_product_build_steps
    product.build(host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 69, in build
    run_build_script_helper('build', host_target, self, self.args)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 107, in run_build_script_helper
    swiftc_path = os.path.abspath(product.toolchain.swiftc)
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: command terminated with a non-zero exit status 1, aborting

I'm not likely to be doing any debug or pull requests on this any time soon as I'm focused on packaging.

[tachoknight]

I had a similar issue with Swift being installed while packaging, but that was a mistake on my part (testing one version while building another) and the Fedora builders[1] won't have Swift installed by default.

[1] Dunno how Debian works, but all packages in the Fedora/RHEL environment are built from source on Fedora-controlled hardware, essentially in a special spun-up container; if I don't include it in the %BuildRequires section, it won't be in the container (which was how I discovered building Swift requires rsync 🙃).

[melizasw]

Yes, it is the same with Debian. I should be using chroot and a clean build environment each time, but I'm not there yet. There is pretty much no chance that the official Debian package would ever get built with that other "swift" package installed. I have rsync in my build dependencies as well, not because I discovered that on my own, but from benefiting from your work. :-)

I've still got some things to sort out, but I'm going to ask if what I have now is good enough to upload to Debian as a starting point.

[tachoknight]

out of curiosity, and because I don't have a Debian box handy, what is the other swift program? In Fedora's case there is another swift, but it's part of the OpenStack Object Storage API Python library, so the chance of a conflict with Swift® is effectively nil.

[melizasw]

On Debian/Ubuntu/Pop_OS there is another package "swift" but it doesn't actually provide the binary /usr/bin/swift, that is provided by the related python-swiftclient and they are indeed part of the OpenStack Object Storage API Python library.

$ sudo nala install python3-swiftclient
$ which swift
/usr/bin/swift
$ swift --version
python-swiftclient 3.13.1

I'd prefer to have swiftlang refuse to install if python3-swiftclient is installed, but Debian policy prohibits two packages from providing the same binary unless they are are the same program having different implementations. I could ask for an exception to this policy, but I think prompting to make the link works too.

[tachoknight]

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

[finagolfin]

Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case)

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

[melizasw]

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

[melizasw]

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

Good to know. I've asked my package sponsor how best to handle this and mentioned this as being what Fedora has done. I think the only issue would be not having separate runtime libraries and needing to install swiftlang to get them. Have you done any work on exploring separating out the runtime libraries into their own package?

[finagolfin]

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

Ah, you're right, found it. The current build checks for both swift and swiftc and tries to build the early swift driver if it finds just the former, as it assumes both will be installed together. Obviously, that assumption will break if your other OpenStack package is installed instead. I wouldn't worry about this, as very few will likely build the Swift toolchain, and almost nobody will do it with that other Swift package installed.

[melizasw]

@tachoknight my Debian sponsor has asked me to move from the Swift provided clang, llvm, cmake, and icu packages to those that come with Debian. I know that lldb is customized by Swift as mentioned in your blog post, but what of these others? I'm assuming that icu and cmake would be fine as long as Swift doesn't depend on a specific version's features.

Do you know and/or do you have a blog post about that sort of issues that I can point him to? He is one of the Debian LLVM package maintainers so he is very experienced in this area.

Of course anyone else that knows is welcome to chime in too.

[tachoknight]

This was my experience as well. The Swift-provided version of LLVM/LLDB is custom for Swift; the standard LLVM/LLDB you might install on any Debian or Fedora box knows nothing about Swift and it simply won't work. The solution I use in Fedora is to hide everything in /usr/libexec/swift and just symlink swift and swiftc to /usr/bin; the custom lldb executable for Swift is not on the path so there's no collision with an existing LLVM/LLDB installation.

Two things I can remember too: When I was originally getting it working with Fedora, I discovered that a lot of of the code assumes executables are in the same directory, or if dynamically loading a library, it will explicitly say something like ../lib/libFoo.so or whatever. The Swift toolchain is a complete package unto itself and thus it really needs to stay together; many early bugs with Swift on Fedora was simply due to constantly missing a spot where I had tried to redirect the loader to a more general location (e.g. Fedora wants all its 64-bit libs in /usr/lib64) and it was really frustrating. Thus when I was able to just drop everything into /usr/libexec/swift, it made life so much easier, and so many patches (and related bugs) just magically went away. The other thing that has come up, and I want to say it's somewhere on the Swift forums, that there was some discussion about moving Swift upstream to be part of the LLVM project. I have no idea what, if any, plans there are to do that and there's no point in thinking about it (this is to head off the possibility of someone asking "Why don't they just move everything to the LLVM project?").

[melizasw]

Thanks @tachoknight, I have forwarded your reply to my sponsor in the hopes that we will agree to keeping Swift together until upstream makes it possible for us to more easily split it apart.

[finagolfin]

As for CMake and ICU, the Swift build just uses the upstream source for each of those, so no problem with not building those from source and using the prebuilt Debian packages instead, provided they're not really old.

The only issue is that if you use the system ICU, you may have to bump the Foundation build everytime the system ICU is updated, to make sure it's linked against the latest version, whereas you don't need to do that if you build another libicu for Swift and stash it separately with the Swift-forked lldb and so on.

[finagolfin]

there was some discussion about moving Swift upstream to be part of the LLVM project.

Not Swift, I believe they just want to slowly upstream all Swift changes from their forked LLVM/clang/lldb to the upstream LLVM repo and get rid of those LLVM forks one day, but I'm guessing that's probably years away.

[tachoknight]

Ah, that's right, I was remembering it incorrectly.

[melizasw]

Thank you all for the information and help. My sponsor would like me to move to as many Debian packages as possible so here is our plan: 1) Switch to Debian cmake and ninja (Debian ftpmasters will reject this package otherwise) 2) Release swiftlang to Debian Experimental 3) Switch to Debian icu, llvm, and clang 4) Release to Debian Testing (from here it migrates to unstable after 5 days then to stable releases and derivative distributions)

[tachoknight]

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

[melizasw]

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

I'm hoping that will be open for negotiation. The desire is to not be packaging yet another fork of llvm/lldb/clang and having to deal with its bugs, but how do I make use of a tool chain that doesn't support my language? Start submitting all of Swift's changes as pull requests to Debian's LLVM packages? At that point it seems like you're still having to maintain a fork, but now those bugs impact far more Debian users than just the swiflang users. In any case, I must at least give it the old college try.

[finagolfin]

1. These are build dependencies so not sure why anyone cares which you use, but should be no problem using the Debian packages, provided you specify the right minimum versions to use.

3. That's only for the final package, you still have to build the Swift-forked LLVM and clang from source for the Swift compiler to work. Also, if you don't build the Swift-forked lldb, not only won't the debugger recognize Swift but you won't be able to use the REPL.

[tachoknight]

The way I see it, the fact that there's an lldb executable in the Swift toolchain's bin directory is just part of the Swift installation; if it had been renamed to xyzzy this wouldn't be anything to discuss. You're not maintaining a fork of LLVM, Apple is, as part of the Swift project. Nothing related to bugs or issues that crop up in the Swift-specific version of LLVM/LLDB applies to the standard LLVM/LLDB project; nobody interacts with the underlying binaries in the Swift toolchain other than swift and swiftc; if I want to write C++ code, I use the system-provided clang/llvm.

Basically my argument is: Swift is a complete package, a veritable iceberg of tools and libraries that should mean nothing to the user of Swift other than what is exposed via /usr/bin; whether the package includes clang, lldb, icu, whatever, it's not going to collide with the existing system-provided equivalents.

[tachoknight]

1. These are build dependencies so not sure why anyone cares which you use, but should be no problem using the Debian packages, provided you specify the right minimum versions to use.

Right, Fedora uses the system-provided Clang/LLVM to build the Swift equivalent.

3. That's only for the final package, you still have to build the Swift-forked LLVM and clang from source for the Swift compiler to work. Also, if you don't build the Swift-forked lldb, not only won't the debugger recognize Swift but you won't be able to use the REPL.

And I tried to make this work and it simply does not and it makes total sense, the default LLDB has no knowledge of Swift so why should it work?

Ultimately what I think the reviewers are confused is why should there be a whole separate LLVM/LLDB/etc. just for Swift, and the answer is to not think of it as the standard LLVM/LLDB; they're special, part of the Swift toolchain, and maybe it would have been easier to name the binaries something else (e.g. swift-lldb) so it was clear this was something not for general use.

[etcwilde]

It might be an idea to break the swift installation into smaller parts. Something like a swift-lldb package that "conflicts" with the lldb package might be a way to move forward since they provide effectively the same base functionality.

Does Foundation need to be in the same package as the compiler? A separate swift-Foundation package would mean that you could tackle the ICU details later, while still getting the compiler in start writing programs that don't need Foundation.

Clang might be a tricky spot. We do pretty regularly make special changes to the clang packaged in Apple/llvm-project that may or may not eventually get upstreamed into the actual clang. I think we're currently around feature-parity from my understanding, (though with the experimental C++ interop stuff, that might be changing day-by-day). At the very least, you will need to ensure that whatever C++ compiler you use to compile the Swift runtimes, it supports the Swift calling convention and the Swift async calling conventions. That's guaranteed to work from the right branch of the apple/llvm-project. It might work from another clang. It won't work from GCC.

[tachoknight]

It might be an idea to break the swift installation into smaller parts. Something like a swift-lldb package that "conflicts" with the lldb package might be a way to move forward since they provide effectively the same base functionality.

Tried it. The confusion is that the versions of LLVM/LLDB/clang between Swift and the standard LLVM installation are mismatched and any bugs that might be encountered when working with C++ code, for example, would go to the wrong place; the user may think it's a LLVM bug and file an issue with the LLVM maintainer, who would be left puzzled as to what they were dealing with. This happened to me on Ubuntu where I had Swift installed in my home directory, with its bin in the path, and had all kinds of weird issues when I was switching back and forth between wanting to work with C++ code or Swift code. The symlink of just swift and swiftc made all the difference and I could easily upgrade either Swift or system LLVM with no issues.

Ultimately, any attempt at changing the way Swift expects to work results in sadness. Even worse, what's valid today may not work tomorrow, and keeping Swift in its original tries-to-live-with-existing-LLVM configuration on Fedora was, frankly, overwhelming. Let the entire toolchain live somewhere (e.g. /usr/libexec/swift) and expose only what's necessary and everything just works. The thank-you emails, and the lack of reported bugs, attest to this. :)

[melizasw]

Thank you all, your input and suggestions are very helpful. I have passed along some of them to my sponsor so that maybe he will relax the requirement that I try to make use of Debian llvm/clang/lldb. I guess I'll get started by trying to use Debian's cmake and ninja and go from here.

Debian policy prohibits shipping binaries with the same name as a binary from another package unless they have identical functionality. They also don't want you shipping duplicate libraries for security reasons. Policies are not carved into stone, but they are very difficult to get an exception to.

[finagolfin]

Does Foundation need to be in the same package as the compiler?

No, it doesn't. If he ever splits off the runtime libraries as discussed above, he'll likely bundle everything in /usr/lib/swift/, including the stdlib, together in a separate package from the compiler, and just have the compiler package list that runtime package as a dependency. You could further split the corelibs up individually as you suggest too: I added libdispatch as a separate package on Android, though that doesn't include the Swift wrapper.

[finagolfin]

They also don't want you shipping duplicate libraries for security reasons.

The Swift versions of LLVM and lldb are not duplicates, they are proper forks, as @tachoknight emphasized. Clang might not be that different anymore, as @etcwilde noted, but there have been significant changes in the past and still might be.

Only libicu is a stock version, but as noted, that is only done to make sure a particular tested version is always available for Swift alone and may not matter as much anymore now that the Swift stdlib stopped using it, #40340, with only the Swift Foundation library using it now. I use the system packages of clang/lld, libdispatch, and libicu as runtime dependencies of my Android package of Swift and have not had a problem (I've not tried building the Swift-forked lldb for Android, so no debugger support or REPL yet, need to get on that), though I have no choice but to compile the Swift-forked LLVM when building the Swift compiler.

[etcwilde]

The Swift versions of LLVM and lldb are not duplicates, they are proper forks

They are proper forks that are kept roughly in sync with the upstream version (bi-annual-ish rebranch). They should be a proper superset of the functionality of the llvm/llvm-project at the point where the branch was cut, with the additional bits on top of that providing better Swift support. If things are broken on the C++ side, that sounds like a bug. We should be setting the BUG_REPORT_URL appropriately so that folks report bugs in the right places though.

[melizasw]

I have deleted the Swift provided/expected local copies of cmake and ninja and turned them build dependencies with a minimum version of what was expected by Swift. I modified build-presets.ini to disable build-ninja for the build that I was using. Is that an appropriate way to stop Swift from trying to build the now missing ninja?

No errors yet relating to the missing cmake so it must have sorted itself out. I'll know for sure in about 4 hours.

I am really looking forward to my new CPU showing up soon as it should cut my build times to about half of what they are now. I have a fanless HDPLEX H3 case so I can't use those high end high TDP CPUs.

[tachoknight]

I have deleted the Swift provided/expected local copies of cmake and ninja and turned them build dependencies with a minimum version of what was expected by Swift. I modified build-presets.ini to disable build-ninja for the build that I was using. Is that an appropriate way to stop Swift from trying to build the now missing ninja?

Yes that's what I used to do; I decided that since CMake and Ninja were built just to build Swift, not part of the final package, and essentially thrown away, to just let them be built, but I get why you may not want to do that (yet).

No errors yet relating to the missing cmake so it must have sorted itself out. I'll know for sure in about 4 hours.

On a RPI 4, it takes around 24 hours. 🙃

I am really looking forward to my new CPU showing up soon as it should cut my build times to about half of what they are now. I have a fanless HDPLEX H3 case so I can't use those high end high TDP CPUs.

Fastest build time I've seen so far is my M1 MBP; 1.5 hours to build in a Docker container.

[melizasw]

My build just generated a .deb package that seems to install and work just fine so I am going to declare the removal of cmake and ninja a success. I've still got an hour or so of linting to wait on for before the build is officially completed.

I've got to get me one of those new MBPs. About 1.5 years ago I replaced my 2012 15" MBPr with a System76 laptop as I didn't like the Apple options at the time. I hate to think how long a Swift build would take on old faithful. My new CPU has arrived so I'll be making the upgrade this evening.

[shahmishal]

@jblache has added DEB control files here https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB for Swift. @melizasw have you seen these?

[melizasw]

@jblache has added DEB control files here https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB for Swift. @melizasw have you seen these?

Yes, @etcwilde had pointed those out to me. The issue isn't how to make a .deb file, but how to make one that complies with Debian Policies and builds on Debian testing. I had a working .deb file back on Swift 5.5.1, but it was miles away from being a completed Debian package that could become an official package. We are nearly there now, but my debian directory contains 32 files to accomplish that. In the eyes of Debian this is the official installer script location.

One example of the difference is that Debian takes copyrights very seriously so you will see that our version of the copyright file is much more accurate and detailed than the one that is provided in swift-installer-scripts. That doesn't make one wit of difference to the .deb file someone builds for their own use, but will get a package kicked out of the official Debian repository.

[melizasw]

Happily it was an easy update from 5.6.2 to 5.6.3 and for fun I tested the Debian built .deb on Pop_OS.

[melizasw]

I've been working on switching ICU over to the Debian provided packages and just want to check to see if I've done it correctly.

To disable ICU in the Swift build I removed "libicu" and "install-libicu" from build-presets.ini, made my build depend on libicu-dev, and my package have a dependency on the runtime library libicu71. Swift builds just fine, but is this a good way to make the switch in ICU and how can I test to make sure that ICU is indeed working in my Swift package?

[finagolfin]

is this a good way to make the switch in ICU and how can I test to make sure that ICU is indeed working in my Swift package?

Yep, the build will pick up the system libicu automatically. To test, check if Foundation is linked against libicu readelf -d /usr/lib/swift/linux/libFoundation.so | ag NEEDED, and maybe check if some small programs built against Foundation work.

[melizasw]

I'm not sure what "ag" is so I substituted in "grep".

$ readelf -d /usr/libexec/swift/lib/swift/linux/libFoundation.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libswiftDispatch.so]
 0x0000000000000001 (NEEDED)             Shared library: [libicuuc.so.71]
 0x0000000000000001 (NEEDED)             Shared library: [libicui18n.so.71]
 0x0000000000000001 (NEEDED)             Shared library: [libicudata.so.71]
 0x0000000000000001 (NEEDED)             Shared library: [libdispatch.so]
 0x0000000000000001 (NEEDED)             Shared library: [libBlocksRuntime.so]
 0x0000000000000001 (NEEDED)             Shared library: [libswiftGlibc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libswift_Concurrency.so]
 0x0000000000000001 (NEEDED)             Shared library: [libswiftCore.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

So it did get linked, which I can also find in the build log:

Found the following ICU libraries:
   uc (required): /usr/lib/x86_64-linux-gnu/libicuuc.so
   i18n (required): /usr/lib/x86_64-linux-gnu/libicui18n.so
   data (optional): /usr/lib/x86_64-linux-gnu/libicudata.so
Found ICU: /usr/include (found version "71.1")

Also, this message implies that symbols from libicuuc.so and libicui18n.so are getting used in libFoundation.so: dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/swiftlang/usr/libexec/swift/lib/swift/linux/libFoundation. so was not linked against libicudata.so.71 (it uses none of the library's symbols)

And if I'm not mistaken, structs come from the Foundation framework:

[finagolfin]

And if I'm not mistaken, structs come from the Foundation framework

No, they come from the compiler and stdlib. Try running plutil --help, that's an executable provided by Foundation.

[melizasw]

Not this one...

$ sudo apt-file search /usr/bin/plutil
gnustep-base-runtime: /usr/bin/plutil

Are you saying that simply running plutil proves that libicu is linking correctly?

$ /usr/libexec/swift/bin/plutil -help
plutil: [command_option] [other_options] file...
The file '-' means stdin
Command options are (-lint is the default):
 -help                         show this message and exit
 -lint                         check the property list files for syntax errors
 -convert fmt                  rewrite property list files in format
                               fmt is one of: xml1 binary1 json
 -p                            print property list in a human-readable fashion
                               (not for machine parsing! this 'format' is not stable)
There are some additional optional arguments that apply to -convert
 -s                            be silent on success
 -o path                       specify alternate file path name for result;
                               the -o option is used with -convert, and is only
                               useful with one file argument (last file overwrites);
                               the path '-' means stdout
 -e extension                  specify alternate extension for converted files
 -r                            if writing JSON, output in human-readable form
 --                            specifies that all further arguments are file names

[finagolfin]

Are you saying that simply running plutil proves that libicu is linking correctly?

Yes, the dynamic linker generally will not run the executable otherwise. Of course, that's no guarantee that it actually invoked any functions from libicu, but it's a simple sanity check to try. If you really want to check it, you'd need to track down which functions in the C CoreFoundation library use libicu functions, then invoke those in some sample code.