CyberShadow / Digger

A tool to build D and bisect old D versions
Other
57 stars 9 forks source link

"digger install" is broken when DMD was installed with Brew #74

Closed aliak00 closed 5 years ago

aliak00 commented 6 years ago

dmd --version: DMD64 D Compiler v2.081.2 dub run digger -- build v2.082.0:

// ...
digger: Saving to cache.
digger: Installing rdmd-ed0f6789ae11d7b57c05d632961e8d5de5d59d43-f0dc2f481814651476b02dc701db044e
digger: Copy: /Users/aliak/tmp/work/temp-cache/v3/rdmd-ed0f6789ae11d7b57c05d632961e8d5de5d59d43-f0dc2f481814651476b02dc701db044e/bin -> /Users/aliak/tmp/work/build/bin
digger: Clearing temporary cache
digger: Moving...
digger: Build successful.
dub run digger -- install
Building package digger in /Users/aliak/.dub/packages/digger-3.0.0-alpha-7/digger/
Performing "debug" build using dmd for x86_64.
digger 3.0.0-alpha-7: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running ../.dub/packages/digger-3.0.0-alpha-7/digger/digger install
digger: Found DMD executable: /usr/local/bin/dmd
std.format.FormatException@/usr/local/opt/dmd/include/dlang/dmd/std/format.d(6319): Orphan format arguments: args[1..2]
----------------
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:395 pure @safe void std.exception.bailOut!(std.format.FormatException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x84a5a82]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:395 pure @safe bool std.exception.enforce!(std.format.FormatException).enforce!(bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x84a59fe]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:395 pure @safe immutable(char)[] std.format.format!(char, immutable(char)[], immutable(char)[]).format(const(char[]), immutable(char)[], immutable(char)[]) [0x84bd401]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 immutable(char)[] install.findConfig(immutable(char)[]) [0x8567531]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 install.ComponentPaths install.parseConfig(immutable(char)[], custom.BuildInfo) [0x8567861]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 void install.install(bool, bool, immutable(char)[]) [0x8568acb]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int digger.Digger.install(ae.utils.funopt.OptionImpl!(0, bool, "Do not prompt", 'y', null, null).OptionImpl, ae.utils.funopt.OptionImpl!(0, bool, "Only print what would be done", 'n', null, null).OptionImpl, ae.utils.funopt.OptionImpl!(2, immutable(char)[], "Directory to install to. Default is to find one in PATH.", \x00, null, null).OptionImpl) [0x85654ae]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int ae.utils.funopt.funopt!(digger.Digger.install(ae.utils.funopt.OptionImpl!(0, bool, "Do not prompt", 'y', null, null).OptionImpl, ae.utils.funopt.OptionImpl!(0, bool, "Only print what would be done", 'n', null, null).OptionImpl, ae.utils.funopt.OptionImpl!(2, immutable(char)[], "Directory to install to. Default is to find one in PATH.", \x00, null, null).OptionImpl), ae.utils.funopt.FunOptConfig(null), ae.utils.funopt.funoptDispatch!(digger.Digger, ae.utils.funopt.FunOptConfig(null), digger.digger().usageFun(immutable(char)[])).funoptDispatch(immutable(char)[][]).fun(immutable(char)[], immutable(char)[][]).descUsageFun!("Install Digger's build result on top of an existing stable DMD installation").descUsageFun(immutable(char)[])).funopt(immutable(char)[][]) [0x85fefaf]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int ae.utils.funopt.funoptDispatch!(digger.Digger, ae.utils.funopt.FunOptConfig(null), digger.digger().usageFun(immutable(char)[])).funoptDispatch(immutable(char)[][]).fun(immutable(char)[], immutable(char)[][]) [0x85f4cad]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int ae.utils.funopt.funopt!(ae.utils.funopt.funoptDispatch!(digger.Digger, ae.utils.funopt.FunOptConfig(null), digger.digger().usageFun(immutable(char)[])).funoptDispatch(immutable(char)[][]).fun(immutable(char)[], immutable(char)[][]), ae.utils.funopt.FunOptConfig([6]), ae.utils.funopt.funoptDispatch!(digger.Digger, ae.utils.funopt.FunOptConfig(null), digger.digger().usageFun).funoptDispatch(immutable(char)[][]).myUsageFun(immutable(char)[])).funopt(immutable(char)[][]) [0x860a5e3]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int ae.utils.funopt.funoptDispatch!(digger.Digger, ae.utils.funopt.FunOptConfig(null), digger.digger().usageFun(immutable(char)[])).funoptDispatch(immutable(char)[][]) [0x85f468a]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int digger.digger() [0x8565aca]
../.dub/packages/digger-3.0.0-alpha-7/digger/bisect.d:204 int digger.__mixin21.run(immutable(char)[][]) [0x8565ee8]
../.dub/packages/ae-0.0.2177/ae/utils/main.d:45 _Dmain [0x849ec43]
Program exited with code 1
[12:43:47:18-09-07]/Users/aliak/tmp $ 

Something I'm missing?

Thanks!

CyberShadow commented 6 years ago

Looks like there was a bug with formatting an error message, fixed in 895d767e20213c010123569a77ff8af827e74616.

The error is "Can't find DMD configuration file %s corresponding to DMD located at %s", but I don't know why that happens for you. How did you install the DMD version in /usr/local/bin/dmd ?

aliak00 commented 6 years ago

Ah I see. I install DMD via brew. And brew usually symlinks to it's own local spot, in this case:

/usr/local/bin/dmd@ -> ../Cellar/dmd/2.081.2/bin/dmd

Does DMD have a configuration file somewhere?

CyberShadow commented 6 years ago

Yes, there should be a default configuration file included with D distributions. Does brew not install it?

aliak00 commented 6 years ago

Hmm. Find gives me these two:

/usr/local/etc/dmd.conf
/usr/local/Cellar/dmd/2.081.2/.bottle/etc/dmd.conf

So it seems brew puts it in user local etc.

CyberShadow commented 6 years ago

Does DMD actually find it there?

What do you get for dmd -v | grep Config ?

aliak00 commented 6 years ago

Seems so:

➜ dmd -v | grep Config
Config file: /usr/local/etc/dmd.conf
CyberShadow commented 6 years ago

Interesting.

I pushed a commit that should pull the config file location from the same place, can you try it?

aliak00 commented 6 years ago

Nahp. This will fix the install part:

if (result.status == 0)
  foreach (line; result.output.splitLines)
    if (line.findSkip("Config file: ")) // also sets line *at* position after arg
      if (pathOK(line.findSplitBefore(dmdConfigName)[0])) // pathOk adds dmdConfigName so don't get that
        return configPath;

So then install works:

[11:07:00:18-09-10]/Users/aliak/tmp $ dub run digger -- install
Building package digger in /Users/aliak/.dub/packages/digger-master/digger/
Performing "debug" build using dmd for x86_64.
digger ~master: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running ../.dub/packages/digger-master/digger/digger install
digger: Found DMD executable: /usr/local/bin/dmd
digger: Found DMD configuration: /usr/local/etc/dmd.conf
digger: Found Phobos static library (libphobos2.a): /usr/local/opt/dmd/lib
digger: Found Phobos source code (std/stdio.d): /usr/local/opt/dmd/include/dlang/dmd
digger: Found Druntime import files (object.d): /usr/local/opt/dmd/include/dlang/dmd
digger: Will install:
digger:  - Binaries to:           /usr/local/bin
digger:  - Libraries to:          /usr/local/opt/dmd/lib
digger:  - Phobos source code to: /usr/local/opt/dmd/include/dlang/dmd
digger:  - Druntime includes to:  /usr/local/opt/dmd/include/dlang/dmd
digger: This will be a new Digger installation.
digger: Backups and uninstall data will be saved in /usr/local/bin/.digger-install
digger: Preparing object list...
digger:  - dmd
digger:  - rdmd
digger:  - libphobos2-64.a
digger:  - object.di
digger:  - core
digger:  - std
digger:  - etc
digger: Testing write access and filesystem boundaries:
digger:  - /usr/local/bin
digger:  - /usr/local/opt/dmd/include/dlang/dmd
digger:  - /usr/local/opt/dmd/lib
digger: UID=501 GID=20 Mode=444
digger: Things to do:
digger:  - Install component dmd from /Users/aliak/tmp/work/result/bin/dmd to /usr/local/bin/dmd
digger:  - Install component rdmd from /Users/aliak/tmp/work/result/bin/rdmd to /usr/local/bin/rdmd
digger:  - Install component libphobos2-64.a from /Users/aliak/tmp/work/result/lib/libphobos2.a to /usr/local/opt/dmd/lib/libphobos2.a
digger:  - Install component object.di from /Users/aliak/tmp/work/result/import/object.d to /usr/local/opt/dmd/include/dlang/dmd/object.d
digger:  - Install component core from /Users/aliak/tmp/work/result/import/core to /usr/local/opt/dmd/include/dlang/dmd/core
digger:  - Install component std from /Users/aliak/tmp/work/result/import/std to /usr/local/opt/dmd/include/dlang/dmd/std
digger:  - Install component etc from /Users/aliak/tmp/work/result/import/etc to /usr/local/opt/dmd/include/dlang/dmd/etc
digger: You will be able to undo this action by running `digger uninstall`.
Proceed with installation? [Y/n] Y
digger: Saving uninstall information...
digger: Backing up original files...
digger:  - dmd
digger:  - rdmd
digger:  - libphobos2-64.a
digger:  - object.di
digger:  - core
digger:  - std
digger:  - etc
digger: Installing new files...
digger:  - dmd
digger:  - rdmd
digger:  - libphobos2-64.a
digger:  - object.di
digger:  - core
digger:  - std
digger:  - etc
digger: Applying attributes...
digger:  - dmd
digger: Warning: UID/GID mismatch for /usr/local/bin/dmd
digger:  - rdmd
digger: Warning: UID/GID mismatch for /usr/local/bin/rdmd
digger:  - libphobos2-64.a
digger:  - object.di
digger:  - core
digger:  - std
digger:  - etc
digger: Install OK.
digger: You can undo this action by running `digger uninstall`.

But running DMD does not work:

Failed to invoke the compiler dmd to determine the build platform: Error: cannot find source code for runtime library file 'object.d'
       dmd might not be correctly installed. Run 'dmd -man' for installation instructions.
       config file: not found
Specify path to file 'object.d' with -I switch
aliak00 commented 6 years ago

And uh... well... now I can't uninstall either 😱

dub run digger -- uninstall
Failed to invoke the compiler dmd to determine the build platform: Error: cannot find source code for runtime library file 'object.d'
       dmd might not be correctly installed. Run 'dmd -man' for installation instructions.
       config file: not found
Specify path to file 'object.d' with -I switch
aliak00 commented 6 years ago

Seems digger tramples on brew: reinstall also didn't work off the bat without some forced relinking:

[11:28:30:18-09-10]/Users/aliak/tmp $ brew reinstall dmd
==> Reinstalling dmd 
==> Downloading https://homebrew.bintray.com/bottles/dmd-2.081.2.high_sierra.bottle.tar.gz
Already downloaded: /Users/aliak/Library/Caches/Homebrew/downloads/a36692ff57f5366eadc203ca4fdcd8b3224e38ff3749a77289c48f2f5a740bda--dmd-2.081.2.high_sierra.bottle.tar.gz
==> Pouring dmd-2.081.2.high_sierra.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/dmd
Target /usr/local/bin/dmd
already exists. You may want to remove it:
  rm '/usr/local/bin/dmd'

To force the link and overwrite all conflicting files:
  brew link --overwrite dmd

To list all files that would be deleted:
  brew link --overwrite --dry-run dmd

Possible conflicting files are:
/usr/local/bin/dmd
/usr/local/bin/rdmd
==> Summary
🍺  /usr/local/Cellar/dmd/2.081.2: 598 files, 68.5MB
Error: Directory not empty @ dir_s_rmdir - /usr/local/Cellar/dmd/2.081.2.reinstall
[11:28:37:18-09-10]/Users/aliak/tmp $ brew link --overwrite --dry-run dmd
Would remove:
/usr/local/bin/dmd
/usr/local/bin/rdmd
[11:29:18:18-09-10]/Users/aliak/tmp $ brew link --overwrite dmd          
Linking /usr/local/Cellar/dmd/2.081.2... 15 symlinks created
[11:29:25:18-09-10]/Users/aliak/tmp $ dmd --version
DMD64 D Compiler v2.081.2
Copyright (C) 1999-2018 by The D Language Foundation, All Rights Reserved written by Walter Bright
[11:29:28:18-09-10]/Users/aliak/tmp $ 
CyberShadow commented 6 years ago

Okay, looks like there's still a few things missing:

  1. We need to follow symlinks before we overwrite any files.
  2. The version installed by brew somehow knows to look for the dmd configuration file in a different location. This may be caused by 1 or needing to pass the SYSCONFDIR variable to the DMD build.
CyberShadow commented 6 years ago

And uh... well... now I can't uninstall either scream

Ah, that's coming from dub because the D compiler is broken. Invoking the Digger binary might have worked.

CyberShadow commented 6 years ago

Do you have SYSCONFDIR in your environment? (env | grep ^SYSCONFDIR)

CyberShadow commented 6 years ago

SYSCONFDIR is set in the Brew formula:

https://github.com/Homebrew/homebrew-core/blob/master/Formula/dmd.rb#L51

Unfortunately, it doesn't look like Digger can do much about this. Brew is building a custom version, and Digger doesn't know ahead of time if the binary it builds is meant to replace Brew's build or one installed normally.

aliak00 commented 6 years ago

Seems it does!

You mean digger can't do an install or digger can't trample on Brew's? If the latter I think it's ok because you can brew force it back to working.

When Digger does a build can it set the conf file path itself for the DMD it builds?

CyberShadow commented 6 years ago

When Digger does a build can it set the conf file path itself for the DMD it builds?

Hmm... there is a way. Try digger -c 'build.components.common.makeArgs=["SYSCONFDIR=/usr/local/etc"]' build

aliak00 commented 6 years ago

Hmm... there is a way. Try

:( did not work

Ran Digger/digger -c 'build.components.common.makeArgs=["SYSCONFDIR=/usr/local/etc"]' build v2.082.0

And then:

./work/result/bin/dmd --help | grep Config
Config file: /Users/aliak/tmp/work/result/bin/dmd.conf

Is it possible when digger builds dmd to force /Users/aliak/tmp/work/result/bin/dmd.conf (or whoever digger is building) as the path to the conf file?

So then when you install with digger the dmd that was built with it would know where the conf file is?

CyberShadow commented 6 years ago

That's expected, DMD looks in its directory before the system directory for a configuration file. Check what happens if you remove dmd.conf from that location.

aliak00 commented 6 years ago

Ok yes you're right!

So, during digger build step can it do the same for where the specific build's dmd.conf was put? (i.e. in work/result/bin/dmd.conf)

Then it'll work no matter where you install it right?

CyberShadow commented 6 years ago

Sorry, I don't understand the question.

DMD looks for dmd.conf in its binary's directory, among other places, so putting a dmd.conf in /usr/local/bin (or whatever directory /usr/local/bin/dmd points to, if it's a symlink) would work, but it would be the wrong place to put it.

aliak00 commented 6 years ago

I don't mean moving dmd.conf, I just tried this: Digger/digger -c 'build.components.common.makeArgs=["SYSCONFDIR=/Users/aliak/tmp/work/result/bin"]' build v2.082.0

And then Digger/digger install

Which put the new dmd in /usr/local/bin`

And then:

$ dmd -h | grep Config
Config file: /Users/aliak/tmp/work/result/bin/dmd.conf

So it "sees" the conf file it was built with

aliak00 commented 6 years ago

So since digger build knows where it'l' put the binary in, I'm asking if it can do the 'build.components.common.makeArgs=["SYSCONFDIR=/Users/aliak/tmp/work/result/bin"] magic during build

CyberShadow commented 6 years ago

No, that's not a good idea - it would break the compiler installed in the system directories if you were to build another D version with Digger, or move Digger's work directory, or try using the system compiler from a different user account...

The installed version should be independent and not rely on any files from DIgger's work directory.

aliak00 commented 6 years ago

But it's not independent. It relies on a conf file being in the same dir. Which is apparently not a given.

dmd -v | grep Config
Config file: (null)

I don't see how does it break the system compiler. It only tells the new dmd where to find its own conf file. Uninstalling should put the old compiler back which will know where it's own SYSCONFDIR.

Or what am I missing here?

CyberShadow commented 6 years ago

But it's not independent. It relies on a conf file being in the same dir.

That's different. It's like saying that it's not independent because it relies on /lib/ld.so or libc. Point is that the compiler binary is installed systemwide and has the same lifetime and visibility as other systemwide-installed programs, but its configuration file does not.

As mentioned, it will cause many problems. I will not support it.

aliak00 commented 6 years ago

Huh? No it's not. libc is not part of dmd. It's a dependency. It's not the same. As mentioned, not sure I understand the problems but anyway I guess that's that.

But do note that brew is probably the most common way to install command line software on osx so it'd be nice if there's some way to make it work, and digger seems really cool too so it'd be nice to have it just work on osx as well.

On Thu, Sep 13, 2018 at 11:53 AM Vladimir Panteleev < notifications@github.com> wrote:

But it's not independent. It relies on a conf file being in the same dir.

That's different. It's like saying that it's not independent because it relies on /lib/ld.so or libc. Point is that the compiler binary is installed systemwide and has the same lifetime and visibility as other systemwide-installed programs, but its configuration file does not.

As mentioned, it will cause many problems. I will not support it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CyberShadow/Digger/issues/74#issuecomment-420950789, or mute the thread https://github.com/notifications/unsubscribe-auth/AChBzILxqnSAtpwAaP3u9LCSm_tuA3oFks5uaisWgaJpZM4WemzR .

CyberShadow commented 6 years ago

so it'd be nice if there's some way to make it work

digger -c 'build.components.common.makeArgs=["SYSCONFDIR=/usr/local/etc"]' build is the way.