Debian 2.1 release on Fedora Rawhide

[russel]

I picked up the Debian binary, even though I am currently using Fedora Rawhide. I'll reboot to Debian Sid in a minute.

Rawhide has:

/usr/lib64/libbz2.so.1 /usr/lib64/libbz2.so.1.0.6 /usr/lib/libbz2.so.1 /usr/lib/libbz2.so.1.0.6

but not the libbz2.so.1.0 that dstep requires. I hacked a link and used LD_LIBRARY_PATH and I got the help message with an incorrect command line. I then tried a correct command line.

|> LD_LIBRARY_PATH=$HOME/lib.Linux.x86_64 dstep --version /usr/include/libdvbv5/dvb-fe.h clang version 3.7.0 (tags/RELEASE_370/final) Target: x86_64-redhat-linux-gnu Thread model: posix

An unknown error occurred: dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred

??:? void dstep.driver.Application.Application.startConversion(immutable(char)[]) [0x589994] ??:? void dstep.driver.Application.Application.run() [0x5892a4] ??:? int dstack.application.Application.Application.releaseStart() [0x59fcf6] ??:? _Dmain [0x58c0b7] ??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9lambda1MFZv [0x5b9dee] ??:? void rt.dmain2._d_run_main(int, char*, extern (C) int function(char[][])).tryExec(scope void delegate()) [0x5b9d44] ??:? void rt.dmain2._d_runmain(int, char**, extern (C) int function(char[][])).runAll() [0x5b9daa] ??:? void rt.dmain2._d_run_main(int, char, extern (C) int function(char[][])_).tryExec(scope void delegate()) [0x5b9d44] ??:? _d_run_main [0x5b9ca1] ??:? main [0x58c151] ??:? __libc_start_main [0x5f0847ff]

dstep.core.Exceptions.DStepException@dstep/core/Exceptions.d(13): An unknown error occurred

??:? void dstep.driver.Application.Application.startConversion(immutable(char)[]) [0x589994] ??:? void dstep.driver.Application.Application.run() [0x5892a4] ??:? int dstack.application.Application.Application.releaseStart() [0x59fcf6] ??:? _Dmain [0x58c0b7] ??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9lambda1MFZv [0x5b9dee] ??:? void rt.dmain2._d_run_main(int, char*, extern (C) int function(char[][])).tryExec(scope void delegate()) [0x5b9d44] ??:? void rt.dmain2._d_runmain(int, char**, extern (C) int function(char[][])).runAll() [0x5b9daa] ??:? void rt.dmain2._d_run_main(int, char, extern (C) int function(char[][])_).tryExec(scope void delegate()) [0x5b9d44] ??:? _d_run_main [0x5b9ca1] ??:? main [0x58c151] ??:? __libc_start_main [0x5f0847ff]

[jacob-carlborg]

Hmm, I looks like it doesn't like when both the --version flag and a file is specified.

[jacob-carlborg]

I tagged a new version as well, v0.2.2. I think you should be able to try the new version with dub fetch dstep && dub run dstep. This will of course compile DStep.

[russel]

On Debian Sid, you have to install libclang-3.7-dev and libbz2-dev, yet the build still fails to find -lclang. I am guessing it is not looking in the 3.7 directories, which means the search path is wrong. Debian Sid still defaults to 3.6 and I have not installed the default dev packages, because I want to force 3.7 usage.

|> dub run dstep -- --version Building package dstep in /home/users/russel/.dub/packages/dstep-0.2.2/ Performing "debug" build using dmd for x86_64. tango 1.0.3+2.068: target for configuration "static" is up to date. mambo 0.0.7: target for configuration "library" is up to date. dstack 0.0.5: target for configuration "library" is up to date. dstep 0.2.2: building configuration "default"... .dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed .dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed Linking... /usr/bin/ld: cannot find -lclang collect2: error: ld returned 1 exit status --- errorlevel 1 dmd failed with exit code 1.

[russel]

On Fedora Rawhide, libclang and libbz2 dependency problems seems solved by having the clang-libs and bzip2-libs packages installed. However, there seems to be a problem in the latest DMD beta release. I am using 2.069 on Sid but 2.070-b1 on Rawhide.

|> dub run dstep -- --version Building package dstep in /home/russel/.dub/packages/dstep-0.2.2/ Performing "debug" build using dmd for x86_64. tango 1.0.3+2.068: building configuration "static"... tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io.vfs.* instead. tango.math.BigInt is deprecated. Please use std.bigint instead. .dub/packages/tango-1.0.3_2.068/tango/text/xml/SaxParser.d(808,8): Error: class tango.text.xml.SaxParser.SAXException use of object.Throwable.message() is hidden by SAXException; use 'alias message = Throwable.message;' to introduce base class overload set dmd failed with exit code 1.

[jacob-carlborg]

It needs to be able to find libclang. Either it in same directory as DStep or in the standard search paths. I should be the same as any other application. It's quite difficult to take in to considerations all different versions of Linux and their different locations of placing libraries.

[jacob-carlborg]

About the beta, it's a known issue [1]. I think it's fixed by this [2].

[1] https://issues.dlang.org/show_bug.cgi?id=15555 [2] https://github.com/D-Programming-Language/druntime/commit/80e5059bed9235f2cdebea2ca5366898b36ed506

[jacob-carlborg]

I could look into linking statically with libclang. I could also set up a docker environment, that would allow me to build for more platforms.

[russel]

I think the issue is one of allowing the "compiler" to know which CLang to work with. Different Linux distributions have different places to store things.

[jacob-carlborg]

Yes, but /usr/lib and /usr/local/lib are standard Posix library search paths.

[russel]

Indeed, but with Debian they only get populated by the default package if they support multiple versions, as with LLVM and Clang. Whilst Fedora also have a system for distinguishing variants, for LLVM and Clang they only have one. The issue here is to easily enable people to add to the LD_LIBRARY_PATH for the Dub build. From LDC build experience LLVM has a tool for setting all the paths, you just have to point to the version you want.

[jacob-carlborg]

The issue here is to easily enable people to add to the LD_LIBRARY_PATH for the Dub build

Wouldn't LD_LIBRARY_PATH=... dub work? Or do you have something else in mind?

From LDC build experience LLVM has a tool for setting all the paths, you just have to point to the version you want

I tried that, didn't work. It didn't read the environment variable for the config binary. But that should hopefully not be needed since in this cases there's only one file that it needs to find, libclang.so.

[russel]

I'll try the LD_LIBRARY_PATH thing now. Given then context this is probably the easiest way out.

[russel]

Sadly not :-(

|> find /usr -iname libclang.so find: ‘/usr/share/doc/google-chrome-stable’: Permission denied /usr/lib/llvm-3.7/lib/libclang.so 519 anglides:~ |> LD_LIBRARY_PATH=/usr/lib/llvm-3.7/lib dub run dstep -- --version Building package dstep in /home/users/russel/.dub/packages/dstep-0.2.2/ Performing "debug" build using dmd for x86_64. tango 1.0.3+2.068: target for configuration "static" is up to date. mambo 0.0.7: target for configuration "library" is up to date. dstack 0.0.5: target for configuration "library" is up to date. dstep 0.2.2: building configuration "default"... .dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed .dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed Linking... /usr/bin/ld: cannot find -lclang collect2: error: ld returned 1 exit status --- errorlevel 1 dmd failed with exit code 1.

[jacob-carlborg]

Do you have a symlink libclang.so pointing to libclang.so.1 or whatever that actual library is called?

[russel]

Yes. It is created when installing the libclang-.*-dev package:

|> ll /usr/lib/llvm-3.7/lib/libclang.so lrwxrwxrwx 1 root root 40 Dec 10 14:17 /usr/lib/llvm-3.7/lib/libclang.so -> ../../x86_64-linux-gnu/libclang-3.7.so.1

[jacob-carlborg]

Hmm, it looks liked Dub doesn't forward the environment variables. If I invoke DMD directly it works. But it still requires to use LD_LIBRARY_PATH at runtime.

[russel]

I suppose I could relent and just install the Debian default (which is currently 3.6).

[russel]

No, that doesn't work. From what I can see Debian install no LLVM related libraries directly in /usr/lib or /lib. LLVM related things are all siloed in /usr/lib/llvm--.

[jacob-carlborg]

:disappointed:

[russel]

It appears that Fedora ensure they put an entry for LLVM in a file in ld.so.conf.d so the LLVM libs are in the search path. From what I can tell, Debian do not do this. Fedora allows for multiple LLVMs, so they are siloed, but it is only the most recent and then any past ones applications need. Debian have many, many LLVMs available, but leave it entirely up to clang to sort of dynamic link paths.

[jacob-carlborg]

I'm available on IRC, perhaps easier to discuss there.

[russel]

Until Dub passes LD_LIBRARY_PATH I am not sure there is any solution to building DStep on Debian using Dub. Sad, but…

[jacob-carlborg]

I've tried a lots of different combinations of environment variables, but none of them seem to work. The only thing that works is to modify dub.json. Change the -L and -rpath flags in the lflags-posix attribute.

[russel]

This is seemingly to get worse :-( I have downgraded DMD to 2.069.2 on Fedora. libclang.so is in the path, but libbz2.so is not:

|> ldconfig -v | grep clang ldconfig: Can't stat /libx32: No such file or directory ldconfig: Path /usr/lib' given more than once ldconfig: Path/usr/lib64' given more than once ldconfig: Can't stat /usr/libx32: No such file or directory libclang.so -> libclang.so

|> ldconfig -v | grep bz2 ldconfig: Can't stat /libx32: No such file or directory ldconfig: Path /usr/lib' given more than once ldconfig: Path/usr/lib64' given more than once ldconfig: Can't stat /usr/libx32: No such file or directory libbz2.so.1 -> libbz2.so.1.0.6 libbz2.so.1 -> libbz2.so.1.0.6

|> dub run dstep -- --version Building package dstep in /home/russel/.dub/packages/dstep-0.2.2/ Performing "debug" build using dmd for x86_64. tango 1.0.3+2.068: building configuration "static"... tango.io.FileScan is deprecated. Please use tango.io.FilePath.toList or tango.io.vfs.* instead. tango.math.BigInt is deprecated. Please use std.bigint instead. mambo 0.0.7: building configuration "library"... .dub/packages/mambo-0.0.7/mambo/serialization/Serializer.d(718,30): Deprecation: typedef is removed .dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed .dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed dstack 0.0.5: building configuration "library"... .dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed .dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed dstep 0.2.2: building configuration "default"... .dub/packages/mambo-0.0.7/mambo/util/Traits.d(154,39): Deprecation: typedef is removed .dub/packages/mambo-0.0.7/mambo/util/Traits.d(182,30): Deprecation: typedef is removed Linking... /usr/bin/ld: cannot find -lclang /usr/bin/ld: cannot find -lbz2 collect2: error: ld returned 1 exit status

[jacob-carlborg]

I don't know what to tell you. libbz2 is a very standard/basic library. If it cannot find that it seems almost broken. Where does Fedora place its libraries?

[russel]

That was my fault, I forgot to install bzip2-devel. Silly me. Only the -lclang is a real problem, and I do not understand than on Fedora as libclang.so is in the search path.

[jacob-carlborg]

Creating a symlink in /usr/lib or /usr/local/lib should work. BTW, is the library the correct architecture?

[russel]

My current hypothesis is that dmd is driving the linker in such a way as it does not use the ld.so search path.

I will do that just to show it working, but I do not touch /usr/lib or /usr/local/lib manually, that is package territory.

All libraries are x86_64 here.

I manually hacked the final dmd command adding the path and:

dmd -of.dub/packages/dstep-0.2.2/.dub/build/default-debug-linux.posix-x86_64-dmd_2069-FD00298C00987FC76518B1B95E49DF2E/dstep .dub/packages/dstep-0.2.2/.dub/build/default-debug-linux.posix-x86_64-dmd_2069-FD00298C00987FC76518B1B95E49DF2E/dstep.o .dub/packages/dstack-0.0.5/libdstack.a .dub/packages/mambo-0.0.7/libmambo.a .dub/packages/tango-1.0.3_2.068/libtango.a -L--no-as-needed -L-rpath -L. -L-L. -L-L/usr/lib64/llvm -L-lclang -L-lbz2 -g

which works fine. So I think we can conclude it is dmd that is the culprit here.

[jacob-carlborg]

You can add -L-L/usr/lib64/llvm to dub.json to avoid manually invoking DMD. You can also append -v to the DMD command to see the linker command it uses.

Is the ld.so search path the same as LD_LIBRARY_PATH?

[russel]

dstep --help still shows 0.2.1.

LD_LIBRARY_PATH adds to the path that ld.so is the manager of. I now need to work out how dmd can invoke ld without ld.so path being in play. This cannot be.

The next real problem though is how to get commands built with Dub into $PATH without being very, very silly.

[jacob-carlborg]

dstep --help still shows 0.2.1.

Looks like I forgot to update that.

LD_LIBRARY_PATH adds to the path that ld.so is the manager of. I now need to work out how dmd can invoke ld without ld.so path being in play. This cannot be.

DMD invokes the C compiler to do the linking. You can see that by adding the -v flag when invoking DMD.

[russel]

I had a play with this and gcc is the problem, not dmd. Well not per se. It could yet be an option dmd puts out.

I have put out a query with someone who works on GCC on Fedora/RHEL.

[russel]

I was wrong, ld doesn't use the ldconfig path, only the runtime loader does. This means LLVM shared object location has to be manually specified on both Debian and Fedora. Having to manually edit dstep.json or dstep.sdl in the repository checkout cannot be the solution, just a temporary hack.

So gcc and dmd are working as they should. The problem thus rests with Dub and it needing to provide platform specific settings. Which actually it does, well in the SDL it does, not sure about JSON.

This means it is up to the original DStep source to have all the variants, which means collecting pull requests from volunteers like me. I'll see what I can do tomorrow on this.

[russel]

Just to note that 2.070-b2 fixes the problem noted earlier regarding 2.070-b1.