With Xcode 9, `make check` for `rsync` spews failures

chdiza commented 7 years ago

[x] Confirmed this is a problem with brew installing one, specific Homebrew/homebrew-core formula (not cask or tap) and not every time you run brew? If it's a general brew problem please file this issue at https://github.com/Homebrew/brew/issues/new. If it's a brew cask problem please file this issue at https://github.com/Homebrew/caskroom/homebrew-cask/new. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
[x] Ran brew update and retried your prior step?
[x] Ran brew doctor, fixed all issues and retried your prior step?
[ ] Ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
[x] If brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

This report isn't really about rsync so much as it is about a potential problem that can be illustrated with rsync.

Having just bumped to Xcode 9, I like to run make check and make test in my formulas to catch any funny business. I caught such funny business---a bunch of errors that weren't there when using Xcode 8.3.3---and it's reproducible simply by adding make check before make install in the official HB rsync formula and then doing brew install --build-from-source rsync.

Normally this is quality control for upstream, but here's the thing. All the make check tests pass when rsync is built outside of HB. This makes me worry that something about HB is not playing nicely with Xcode 9. (No such problem exists for Xcode 8.3.3).

Here's my brew config (in which you can see two separate bugs, one the xcrun: error gibberish which might even share a common cause with what I'm reporting) the other the Unable to find any JVMs):

HOMEBREW_VERSION: 1.3.2-82-gb93fa96
ORIGIN: https://github.com/Homebrew/brew
HEAD: b93fa96ec8b79bbe97253bc4a493c82ce1bb6d50
Last commit: 18 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: ed1b3b19ba97efb344cb2f2886ce808140f04655
Core tap last commit: 3 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local/hb
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: quad-core 64-bit arrandale
Homebrew Ruby: 2.0.0-p648
xcrun: error: unable to find utility "gcc-4.0", not a developer tool or in PATH
xcrun: error: unable to find utility "gcc-4.2", not a developer tool or in PATH
Clang: 9.0 build 900
Git: 2.13.5 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Perl: /usr/bin/perl
Python: /usr/bin/python
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
Unable to find any JVMs matching version "(null)".
Java: N/A
macOS: 10.12.6-x86_64
Xcode: 9.0
CLT: 9.0.0.0.1.1504363082
X11: N/A

brew doctor says "ready to brew"

And finally here is the log for the make check stage, and you can see a bunch of weirdos in there, such as:

dyld: lazy symbol binding failed: Symbol not found: _utimensat
  Referenced from: /Users/chdiza/.tmp/tmpdir/rsync-20170920-87566-kqs8qp/rsync-3.1.2/rsync
  Expected in: /usr/lib/libSystem.B.dylib

ilovezfs commented 7 years ago

utimensat is like clock_gettime in terms of the bugs it causes. The problem should be specific to Xcode 9 + 10.12 and should not affect Xcode 9 + 10.13.

chdiza commented 7 years ago

FWIW, basically the same thing happens with wget.

ilovezfs commented 7 years ago

Yes, it will be pretty common. The sooner you upgrade to 10.13, the less of it you will see :)

chdiza commented 7 years ago

I'm scared of the immature APFS.

Anyway, the clock_gettime stuff, IIRC, was a problem for both HB and outside of HB. The current stuff seems to only happen inside HB. This makes me wonder whether this is HB's problem rather than just the typical breaking Xcode. That's why I haven't done my yearly "Every flippin' time a major Xcode version comes out, they break stuff for no reason other than that they just like shuffling stuff around" complaint. :smile:

ilovezfs commented 7 years ago

Anyway, the clock_gettime stuff, IIRC, was a problem for both HB and outside of HB. The current stuff seems to only happen inside HB.

No it's the same situation.

ilovezfs commented 7 years ago

I'm scared of the immature APFS.

You can run startosinstall --converttoapfs NO to avoid it if it's just too scary :)

chdiza commented 7 years ago

You can run startosinstall --converttoapfs NO to avoid it if it's just too scary :)

I sure hope this is true when it's released for real. Apple's own APFS FAQ says that SSD users cannot opt out of APFS.

ilovezfs commented 7 years ago

It's true in the GM, at least.

FWIW, I would suggest just trusting APFS. It's pretty cool :)

chdiza commented 7 years ago

No it's the same situation.

Well then, right on schedule: "Every flippin' time a major Xcode version comes out, they break stuff for no reason other than that they just like shuffling stuff around".

Why they allow Xcode 9 on Sierra is an utter mystery, if Sierra lacks the relevant functions.

ilovezfs commented 7 years ago

So it is, and so it shall be now that there's a unitary SDK.

chdiza commented 7 years ago

FWIW, I would suggest just trusting APFS. It's pretty cool :)

In theory. It's too new, and my life is too rsync-heavy to risk bugs that result from devs not having enough time to play with APFS.

ilovezfs commented 7 years ago

Then startosinstall --converttoapfs NO it is!

chdiza commented 7 years ago

So it is, and so it shall be now that there's a unitary SDK.

Yup, they just like shuffling stuff around, without regard for the consequences.

It's still mysterious to me why, in the particular case that starts this issue thread, the problem doesn't arise outside HB.

ilovezfs commented 7 years ago

It's still mysterious to me why, in the particular case that starts this issue thread, the problem doesn't arise outside HB.

It's because we set SDKROOT to the Xcode.app SDKROOT, so that CI catches issues that would only affect people who have Xcode but don't have the CLT.

chdiza commented 7 years ago

FWIW, the make check errors don't happen if installed with --env=std.

so that CI catches issues that would only affect people who have Xcode but don't have the CLT.

Wouldn't catching those be less important that avoiding all the utimensat (or clock_gettime, or whatever they break next year) bugs to begin with?

ilovezfs commented 7 years ago

Wouldn't catching those be less important that avoiding all the utimensat (or clock_gettime, or whatever they break next year) bugs to begin with?

We want things to break before we ship them, not after.

chdiza commented 7 years ago

Add GNU patch to the list.

ilovezfs commented 7 years ago

:tada:

chdiza commented 7 years ago

FWIW, in all the problem cases I've caught today---rsync, gpatch, and wget---things only go bad during make check/make test.

ilovezfs commented 7 years ago

Do they go :boom: if you run brew test on them? That's where we'd expect it to blow up.

chdiza commented 7 years ago

I have never run brew test on them. I'll try and report back.

Add vim to the list. EDIT: the vim problem occurs even outside HB.

chdiza commented 7 years ago

rsync passes brew test, but when I try to actually use it to do something, I get:

sending incremental file list
dyld: lazy symbol binding failed: Symbol not found: _utimensat
  Referenced from: /usr/local/bin/rsync
  Expected in: /usr/lib/libSystem.B.dylib

chdiza commented 7 years ago

wget passes brew test.

chdiza commented 7 years ago

At least in the case of rsync, everything is fixed by using xcode-select -s to point to the CLT location instead of Xcode.app.

ilovezfs commented 7 years ago

At least in the case of rsync, everything is fixed by using xcode-select -s to point to the CLT location instead of Xcode.app.

This will cause build failures elsewhere.

ilovezfs commented 7 years ago

@chdiza this should stanch the bleeding https://github.com/Homebrew/brew/pull/3182

chdiza commented 7 years ago

This will cause build failures elsewhere.

Yeah, I know, I just mentioned it as a one-off workaround.

chdiza commented 7 years ago

@chdiza this should stanch the bleeding Homebrew/brew#3182

Thanks. That seemed to work on a quick attempt with rsync. I'll try some others later when I'm at the relevant machine.

ilovezfs commented 7 years ago

I'll try some others later when I'm at the relevant machine.

Thanks!

chdiza commented 7 years ago

OK, I've run my automated install of my usual 61 packages, many of which are from custom formulas with make {check,test} inserted. All of them built correctly and passed their make {check/test}s, with the exception of curl (and that one---test 1148---I think has nothing to do with Xcode 9 and was failing before anyway).

I'm on the lookout for runtime failures and will report them here if I find them.

ilovezfs commented 6 years ago

@chdiza please let us know if you run into any more of these issues.

Homebrew / homebrew-core

With Xcode 9, `make check` for `rsync` spews failures #18335