search

dlang / dub

Package and build management system for D
MIT License
674 stars 230 forks source link

Segmentation fault when searching for dependencies #1373

Closed jmafc closed 6 years ago

jmafc commented 6 years ago

System information

Bug Description

dub build causes segmentation fault when searching for dependencies

How to reproduce?

cat >dub.json <<EOF
{
  "name": "foo",
  "dependencies": {
    "anything": "~>0.1.1"
  }
}
EOF
dub build

Expected Behavior

It should fetch the dependencies and proceed to compilation and linking. Note: as suggested by somebody in the forum (general), the official release does not cause the segfault. The reported versions of the official release are confusing: ldc says it's 1.7.0 but dub reports 1.6.0.

Logs

$ dub -v build
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/jma/.dub/packages/local-packages.json
Note: Failed to determine version of package foo at .. Assuming ~master.
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/jma/.dub/packages/local-packages.json
  Version selection for dependency anything (anything) of foo is missing.
  Missing dependency anything ~>0.1.1 of foo
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at /home/jma/.dub/packages/local-packages.json
  Version selection for dependency anything (anything) of foo is missing.
  Missing dependency anything ~>0.1.1 of foo
Checking for missing dependencies.
Search for versions of anything (1 package suppliers)
Segmentation fault

Note: this is after removing ~/.dub but it also happens when that is present and has packages, even the one being sought for.

$ strace dub build
execve("/usr/bin/dub", ["dub", "build"], 0x7ffcf30bce98 /* 46 vars */) = 0
...
lstat("/home/jma/.dub/packages/", 0x7fff3e80dd80) = -1 ENOENT (No such file or directory)
lstat("/var/lib/dub/packages/", 0x7fff3e80dd80) = -1 ENOENT (No such file or directory)
lstat("/home/jma/.dub/packages/local-overrides.json", 0x7fff3e80dd60) = -1 ENOENT (No such file or directory)
lstat("/var/lib/dub/packages/local-overrides.json", 0x7fff3e80dd60) = -1 ENOENT (No such file or directory)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x60} ---
+++ killed by SIGSEGV +++
Segmentation fault
kirbyUK commented 6 years ago

Confirming this issue, I've made a bug report to the Debian maintainers as it seems to be a Debian-only issue, hopefully this is fixed soon.

ximion commented 6 years ago

Smells like a GDC 7 bug, fortunately it seems to work well with the awesome GDC 8 (which finally supports a very recent D version and standard library) Lucky for us, we can use GDC 8 in Debian already (alternatively rewriting the build script to work with LDC as well as GDC would have been an option as well).

So, I guess you could close this report, unless someone finds that this is still not working.

jmafc commented 6 years ago

@ximion I don't have any version of GDC installed, only LDC, although I have GCC 6 and 7. I did an apt update just update right now, but there's no new version of dub to be installed. I upgraded GCC 7 just in case, but the segfault still occurs. I'll do a more complete upgrade later today and report then.

ximion commented 6 years ago

@jmafc The issue depends on which version of GCC dub was compiled with. To update to a working version, you can fetch the updated package from Debian unstable, or wait 5 days and get it from testing via regular updates. (Since the fixed version was uploaded very recently, it might take up to 6 hours to even show up in unstable / on your mirror)

jmafc commented 6 years ago

@ximion Since I only use it sporadically and I have a working "official" version, I'll just wait for the regular update to testing.

ximion commented 6 years ago

It would still be helpful to test the new version though, to ensure the issue is really gone. You can get it from here: dub package (I have not found any issue while testing it when compiled with the new GDC)

jmafc commented 6 years ago

I installed the sid package:

Preparing to unpack dub_1.7.2-1_amd64.deb ...
Unpacking dub (1.7.2-1) over (1.7.0-2) ...
Setting up dub (1.7.2-1) ...

I then tried to use it to build a package that depends on derelict-pq.

$ cat dub.json 
{
  "name": "select",
  "description": "Application to test derelict-pq",
  "targetType": "executable",
  "mainSourceFile": "select.d",
  "dependencies": {
    "derelict-pq": "~>3.0.0-beta.3"
  }
}
$ dub --version
DUB version 1.7.2-1, built on Feb 16 2018
dub -v build
Using dub registry url 'https://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
...
  Version selection for dependency derelict-pq (derelict-pq) of select is missing.
  Missing dependency derelict-pq >=3.0.0-beta.3 <3.1.0-0 of select
Checking for missing dependencies.
Search for versions of derelict-pq (1 package suppliers)
Package derelict-pq not found in registry at https://code.dlang.org/ (fallback ["registry at http://code.dlang.org/", "registry at https://code-mirror.dlang.io/", "registry at https://code-mirror2.dlang.io/", "registry at https://dub-registry.herokuapp.com/"]): HTTP request returned status code 0 ()
Nothing found for derelict-pq
Root package select references unknown package derelict-pq

I also tried building a similar package with dpq2 "~>1.0.0-alpha.9" and had a similar result. So, no more segfault but something else is still wrong.

ximion commented 6 years ago

I have no idea why that happens - some help from the dub authors would be useful (and ideally we should close this bug report and open a new one for the new issue). Dub is linked and built with Curl, so it should have network support.

wilzbach commented 6 years ago

HTTP request returned status code 0 (

This means that the URL curl receives is invalid. I'm on the road over the weekend so I can't help much here, but I would start with logging the actual requested.

(There is CURLOPT_VERBOSE, but I don't remember by hard whether it works for libraries too.)

jmafc commented 6 years ago

Running dub --vverbose shows the following:

Search for versions of derelict-pq (1 package suppliers)
Downloading metadata for derelict-pq
Getting from https://code.dlang.org/packages/derelict-pq.json
Getting https://code.dlang.org/packages/derelict-pq.json...
Downloading metadata for derelict-pq
Getting from http://code.dlang.org/packages/derelict-pq.json
Getting http://code.dlang.org/packages/derelict-pq.json...
Package supplier getVersions failed. Trying next fallback.
Downloading metadata for derelict-pq
Getting from https://code-mirror.dlang.io/packages/derelict-pq.json
Getting https://code-mirror.dlang.io/packages/derelict-pq.json...
Package supplier getVersions failed. Trying next fallback.
Downloading metadata for derelict-pq
Getting from https://code-mirror2.dlang.io/packages/derelict-pq.json
Getting https://code-mirror2.dlang.io/packages/derelict-pq.json...
Package supplier getVersions failed. Trying next fallback.
Downloading metadata for derelict-pq
Getting from https://dub-registry.herokuapp.com/packages/derelict-pq.json
Getting https://dub-registry.herokuapp.com/packages/derelict-pq.json...
Package derelict-pq not found in registry at https://code.dlang.org/ (fallback ["registry at http://code.dlang.org/", "registry at https://code-mirror.dlang.io/", "registry at https://code-mirror2.dlang.io/", "registry at https://dub-registry.herokuapp.com/"]): HTTP request returned status code 0 ()
Full error: std.net.curl.HTTPStatusException@/usr/lib/gcc/x86_64-linux-gnu/8/include/d/std/net/curl.d(1050): HTTP request returned status code 0 ()
----------------
0x5576eda87d68 ???
    ???:0
...
0x7f761633ef29 __libc_start_main
    ???:0
0x5576ed9dcc19 ???
    ???:0
0xffffffffffffffff ???
    ???:0
Nothing found for derelict-pq
Configurations used for dependency resolution:
  derelict-pq (0, required): [invalid]
Root package select references unknown package derelict-pq
Full exception: object.Exception@source/dub/dependencyresolver.d(160): Root package select references unknown package derelict-pq

I've snipped most of the addresses because they have the same question-marks pattern. There's another similar dump at the end. I used curl from the command line for each of the URLs shown and got back the JSON data without errors.

jmafc commented 6 years ago

I have updated dub from the sid version to buster/testing and as expected it behaves as the previous one, i.e., it reports not being able to find any versions of derelict-pq. I also re-installed the "official" dub, which as reported earler was able to download the package onto my ~/.dub/packages and did a normal build. Now, deactivating the official dub and running the buster dub, the latter manages to build properly, even though it first reports not being able to find the package remotely. Earlier @ximion suggested closing this, because there is no longer a segfault, and opening another issue for the problem as it now stands. Please advise.

ximion commented 6 years ago

@ibuclaw Do you know something about GDC influencing behavior of curl functions? It seems like dub compiled with GDC 8-20180218 doesn't work properly due to that. @jmafc To keep issues tracked properly and not mix multiple separate conversations together, opening a new bug for a new problem is usually the way to go. Although I guess this one isn't the fault of dub, but rather GDC doing something odd.

ibuclaw commented 6 years ago

@ximion - Maybe its just the library version?

ximion commented 6 years ago

@ibuclaw That would be libcurl4-gnutls == 7.58.0 at the moment, which works fine otherwise. I wonder whether the issue is some linking weirdness... In any case, you can find a full buildlog at https://buildd.debian.org/status/fetch.php?pkg=dub&arch=i386&ver=1.7.2-1&stamp=1518788878&raw=0 (amd64 was built locally, so no log there at the moment)

ibuclaw commented 6 years ago

@ximion - I was thinking more the version of phobos that gdc built. Judging from the line of the exception, seems like it could be v2.076.1, which is the current version in master.

Any compiler bug reports would need something a little more reduced. Probably start with calling curl.get() with a HTTP connection object. If that fails, we can reduce from there.

ximion commented 6 years ago

At the moment I unfortunately don't have enough time for an excessive debugging session, while at the same time fixing this as soon as possible would be very nice, because otherwise Ubuntu 18.04 LTS will ship with a broken dub. I'm grateful for any help :-) (reporting a bug against Ubuntu would likely also be useful, it's more urgent there than it is for Debian)

ibuclaw commented 6 years ago

Alright, I thought I might check the CI - as I would have thought it would have picked up a problem if there was one.

However it seems that the net.curl module has been turned off for a while now... https://github.com/D-Programming-GDC/GDC/blob/c7cd7572b0ca7f7a6bb2a553e56db21182c072a9/libphobos/src/Makefile.am#L53-L57

wilzbach commented 6 years ago

Out of interest: have you tried the nonet build of dub? In other words the one without a dependency on curl, but vibe.d? With vibe-core you can even use it without the libevent dependency if that's a problem.

ximion commented 6 years ago

No, not yet - but an interesting idea! Unfortunately, Vibe.d in Debian is built with LDC, while Dub builds with GDC, so just linking to it is not possible. (doesn't prevent local builds against Vibe, of course)

FTR: Dub is built using GDC because when I added it I had no idea about how D works compiler-wise, and dub had a neat script to build with gdc, that needed just minimal adjustments to work for Debian. Also, gdc supports multiple architectures a bit better at the moment and GCC is much better supported in Debian than anything else. I actually considered building dub with LDC as a workaround to get something functional into Ubuntu, but doing that would just be as much pain as trying fixing the actual bug (due to the arch-specific removals this would inevitably cause, which will result in a time-delay - also Ubuntu is in feature freeze now).

ibuclaw commented 6 years ago

In both cases, dub was built using:

fakeroot debian/rules build

Test: dub 1.2.0 + curl 7.55.1 (artful)

$ cd issue1373
$ ../bin/dub build
Root package foo references unknown package anything

$ cd ../derelict1373
$ ../bin/dub build
Fetching derelict-pq 3.0.0-beta.3 (getting selected version)...
Fetching derelict-util 3.0.0-beta.2 (getting selected version)...
Performing "debug" build using dmd for x86_64.
derelict-util 3.0.0-beta.2: building configuration "library"...
derelict-pq 3.0.0-beta.3: building configuration "derelict-pq-dynamic"...
select ~master: building configuration "application"...
Error: module select is in file 'select.d' which cannot be read
import path[0] = ../../../../.dub/packages/derelict-pq-3.0.0-beta.3/derelict-pq/source/
import path[1] = ../../../../.dub/packages/derelict-util-3.0.0-beta.2/derelict-util/source/
import path[2] = /usr/include/dmd/phobos
import path[3] = /usr/include/dmd/druntime/import
dmd failed with exit code 1.

So far so good, ignoring the dmd build failure (I guess mine is too old?) the combination of dub 1.2.0 + curl 7.55 seems fine, it's downloading the metadata without problems.

Test: dub 1.7.2 + curl 7.55.1 (artful)

$ cd issue1373
$ ../bin/dub build
Root package foo references unknown package anything

$ cd ../derelict1373
$ ../bin/dub build
Fetching derelict-pq 3.0.0-beta.3 (getting selected version)...
Fetching derelict-util 3.0.0-beta.2 (getting selected version)...
Performing "debug" build using dmd for x86_64.
derelict-util 3.0.0-beta.2: building configuration "library"...
derelict-pq 3.0.0-beta.3: building configuration "derelict-pq-dynamic"...
select ~master: building configuration "application"...
Error: module select is in file 'select.d' which cannot be read
import path[0] = ../../../../.dub/packages/derelict-pq-3.0.0-beta.3/derelict-pq/source/
import path[1] = ../../../../.dub/packages/derelict-util-3.0.0-beta.2/derelict-util/source/
import path[2] = /usr/include/dmd/phobos
import path[3] = /usr/include/dmd/druntime/import
dmd failed with exit code 1.

Same result, so that should rule out dub being a problem. And I think the compiler is fine too, though this is with a vanilla gdc compiler I built myself.

I'll see if I can update curl and give the latest version a try.

ibuclaw commented 6 years ago

However when downloading the dub package and running it, I can confirm that it does indeed fail.

ibuclaw commented 6 years ago

Built gdc and phobos using commit as of 27/02/2018, and looks like I can reproduce it now, but the problem is gone in master.

So I'm left to believe that something in this commit diff fixed it, but I couldn't say what. There was one bug fix, but I would be surprised if that caused this issue.

https://github.com/D-Programming-GDC/GDC/compare/9aa4f3c4...f5f3889c

ibuclaw commented 6 years ago

Although now I seem to persistently get the problem. Probably a sign I should stop debugging, and rebuild my environment tomorrow.

wilzbach commented 6 years ago

dub had a neat script to build with gdc, that needed just minimal adjustments to work for Debian.

FYI: build.sh works with every compiler. The official releases even use DMD, because LDC still has issues with displaying the stack trace of uncaught exceptions.

ximion commented 6 years ago

FYI: build.sh works with every compiler.

Only if a DMD-compatible interface or wrapper script is available, which was not the case back then and still isn't for gdc today. (also, you need to know about that - when I started with D, I naively assumed every D compiler would use flags gcc uses and be compatible with each other, similarly to how clang works - I didn't know that wrappers even existed, or how the dmd interface looked like (due to me using software readily available in my distro back then instead of downloading things from dlang.org - dmd was still semi-proprietary back then[1]))

FTR, to compile dub we use a custom script at Debian, which includes the package revision number in dub's --version output and works around some Ubuntu-specific linker weirdness. See https://salsa.debian.org/d-team/dub/blob/master/debian/build_dub.sh (not relevant to this issue, but maybe I'll file a PR to update the upstream build-gdc.sh script at some point)

[1]: Today, nothing prevents dmd from being in Debian or any other distro except for the lack of maintainers.

ibuclaw commented 6 years ago

@ximion - By the way, any reason why you're using gcc trunk? Is version 8 set to be the default in bionic?

ximion commented 6 years ago

@ibuclaw See the original issue this bug was about: When building with gdc-7, dub outright crashes when you try to do anything with it. GDC 8 apparently has this issue fixed (but unfortunately gives us the curl one).

ibuclaw commented 6 years ago

@ximion - OK, after some proding round, I think I have the facepalm moment. :-)

Bug was that status code is always 0. Reason was because this regex was never matching.

void check(in char[] a)
{
    const m = match(a, regex(r"^HTTP.(\d+)\.(\d+) (\d+) (.*)$"));
    writeln(m.empty); // true
}

check("HTTP/1.1 404 Not Found");

This is because one of the char matchers in the engine returns false, but why? When matching \d, it should check the first number against the following CharMatcher

ascii = {filter = {0, 67043328, 0, 0}}

But when it comes to the comparison, we instead have a default initialized struct

ascii = {filter = {0, 0, 0, 0}}

Setting a breakpoint on the constructor, I can see that it is initialized correctly, but when I look at the return value, the value does indeed appear to get lost.

auto t  = getMatcher(set);
// t = {ascii = {filter = {0, 0, 0, 0}}, trie = {_table = {offsets = {0, 0}, sz = {0, 0}, storage = 0x0}}}

Inspecting the return of getMatcher, I see a suspicious line.

return (matcherCache[set] = CharMatcher(set));

So we must be returning the cached result of the AA index before it set. :scream: I bet if I were to rewrite this to:

matcherCache[set] = CharMatcher(set);
return matcherCache[set];

Then it would work correctly.

ibuclaw commented 6 years ago

Then it would work correctly.

Ah, no it doesn't, because set is a ref counted type, and so the key changes every single time a copy occurs. You'd have thought there'd be a unittest or testsuite entry for this.

ibuclaw commented 6 years ago

Compiler related regression came from: https://github.com/D-Programming-GDC/GDC/pull/571 - and I have a fix, but now I just need to track down what the original fix was intended for. :-)

ibuclaw commented 6 years ago

Fix is in gdc master @ximion.

The reduced cause is rather interesting though. ttps://run.dlang.io/is/ubKJT5 (hit run and see the range violation).

ximion commented 6 years ago

@ibuclaw You're amazing, thank you for debugging this! And actually quite good that this issue was caught early :-) Debian's GDC seems to be up to date already, so we can safely close that issue (and I will ensure dub is rebuilt with the fixed GDC version).

jmafc commented 6 years ago

Confirm fixed using dub 1.8.0-2, ldc 1.8.0 under Debian buster.