Closed NathanDunfield closed 5 years ago
Description changed:
---
+++
@@ -2,4 +2,4 @@
https://groups.google.com/forum/#!topic/sage-devel/FloU3fAgQLM
-for a discussion and some potential solutions.
+for a discussion and some potential solutions. Currently, it seems unknown if this NTL builds on 10.10 but it is known to do so on 10.11.
Description changed:
---
+++
@@ -2,4 +2,21 @@
https://groups.google.com/forum/#!topic/sage-devel/FloU3fAgQLM
-for a discussion and some potential solutions. Currently, it seems unknown if this NTL builds on 10.10 but it is known to do so on 10.11.
+for a discussion and some potential solutions. Currently, it seems unknown if this NTL builds on 10.10 but it is known to do so on 10.11.
+
+---
+
+From the above mentioned thread:
+
+ *If you pass `gcc` the flag `-Wa,-q`, then it will use clang's assembler rather than the ancient version of the GNU assembler that comes with these versions of OS X. See*
+
+ [http://stackoverflow.com/questions/9840207](http://stackoverflow.com/questions/9840207)
+
+ ''I can confirm that adding this flag allowed me to compile all of NTL on these machines. Specifically, changing line 78 of `build/pkgs/ntl/spkg-install` to
+
+ `SHAREDFLAGS="-fno-common -Wa,-q"`
+
+ *caused NTL to build smoothly.*
+
+ *Nathan*
+
Changed keywords from none to MacOS AVX no such instruction
Is this something we should report upstream?
(And is the work-around using clang's assembler instead possible on all affected MacOS X systems? In fact, it's better than dropping -march=native
or adding -mno-avx
and/or -mno-avx2
in case the default assembler doesn't support AVX.)
We could ask Victor for his opinion, but it probably works out of the box on an OS X system. The -march=native
is automatically tested by ntl's tuning system unless NATIVE=off
is passed to the configuration. I actually inserted a bit of code in spkg-install
to disable native for binary. The test could be extended to OS X.
# If SAGE_FAT_BINARY is enabled we don't want ntl to be built with cpu specific
# instructions such as avx and fma.
if [ "$SAGE_FAT_BINARY" = "yes" ]; then
DISABLE_NATIVE="NATIVE=off"
else
DISABLE_NATIVE="NATIVE=on"
fi
But I would go for adding the -Wa,-q
in the already existing appropriate section. I would think it would work for all versions of OS X.
As mentioned on #20563, we could also configure Sage's GCC (which is built and used on MacOS X in any case?) to use clang's assembler on MacOS X by default.
Not using AVX (when building from source) on capable CPUs is perhaps too much punishment, or do you think they get what they deserve?
Replying to @nexttime:
As mentioned on #20563, we could also configure Sage's GCC (which is built and used on MacOS X in any case?) to use clang's assembler on MacOS X by default.
That would actually be the best solution in my opinion and would mean no changes are needed in other packages. ntl and SnapPy are probably just the tip of the iceberg (linbox and co will want to do that kind of stuff too once we finish their upgrade ticket). Stuff will start to break. If a single fix in gcc can save us trouble in the future we should go for it.
I agree that we shouldn't use clang's assembler just for NTL --- I compiled SnapPy
successfully with -Wa,-q
on 10.8 against the binary distro of Sage (which used gcc's assembler) but got segfaults when I ran it possibly related to stack alignment issues. Going whole hog and using clang's assembler for all of Sage makes sense, assuming it actually works ;-).
Well, only one way to do that, let's try it. How do we do that.
Replying to @kiwifb:
Well, only one way to do that, let's try it. How do we do that.
https://gcc.gnu.org/install/configure.html#with-as ?
An even easier way would be to install a wrapper script as
in $SAGE_LOCAL/bin/
on MacOS X to call the "right" one, no idea whether its location differs on different systems. (Or does clang's assembler have a unique alias?)
I am not in favor of the wrapper unless unless we cannot do otherwise. I am more concerned about whether clang needs options to tell it to act as an assembler.
Replying to @kiwifb:
I am not in favor of the wrapper unless unless we cannot do otherwise. I am more concerned about whether clang needs options to tell it to act as an assembler.
Well, more precisely, it's not clang's assembler, but LLVM's, hence presumably llvm-as
.
I don't seem to have llvm-as
on my OS X system, but based on the script at https://gist.github.com/xianyi/2957847, it looks like clang -c -x assembler
is how you invoke it.
We may have to go the wrapper route unfortunately
checking whether a default linker was specified... no
configure: error: cannot execute: clang -c -x assembler: check --with-as or env. var. DEFAULT_ASSEMBLER
make[2]: *** [configure-stage1-gcc] Error 1
I am not sure what configure expect, config.log
doesn't have more info.
Replying to @NathanDunfield:
I don't seem to have
llvm-as
on my OS X system,
Sure? It's usually not in $PATH
, but in some subfolder of clang/LLVM, on Linux for example in /usr/local/clang-X.Y/bin/`.
but based on the script at https://gist.github.com/xianyi/2957847, it looks like
clang -c -x assembler
is how you invoke it.
Hmmm, the -x assembler
should only be necessary if the input comes from a pipe, or the filename doesn't end with .s
, .S
, .asm
or the like.
Well same results from gcc's stage 1 configure script with or without -x assembler
.
clang -print-prog-name=llvm-as
perhaps? (Dumps the full pathname at least in case it is non-standard; maybe as
instead of llvm-as
.)
That's a GCC(-compatible) option btw.
Hum...
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ clang --print-prog-name=as
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ clang --print-prog-name=llvm-as
llvm-as
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ gcc --print-prog-name=as
as
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ which as
/usr/bin/as
And
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ as --help
OVERVIEW: clang LLVM compiler
USAGE: clang [options] <inputs>
so the as
on the path is already clang
.
Replying to @kiwifb:
And
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ as --help OVERVIEW: clang LLVM compiler USAGE: clang [options] <inputs>
so the
as
on the path is alreadyclang
.
What version of OS X is this? On 10.8 and 10.9, the as
command doesn't even accept the --help
flag and as -v
gives (on 10.9)
Apple Inc version cctools-862, GNU assembler version 1.38
I think Apple switched to a clang-based as
in either 10.10 or 10.11, which is why this version of NTL has been confirmed to work on 10.11 but fails on 10.8 and 10.9.
Replying to @NathanDunfield:
I think Apple switched to a clang-based
as
in either 10.10 or 10.11, which is why this version of NTL has been confirmed to work on 10.11 but fails on 10.8 and 10.9.
So, in fact, Sage has been using clang's as
on 10.11 without a hitch or anyone even noticing, which suggests that we should be able to switch to it on 10.8, 10.9, and (maybe) 10.10 to solve this and related tickets.
It is 10.11.5.
(sage-sh) fbissey@Mirage:sage-7.2.beta5$ as -v
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1as -triple x86_64-apple-macosx10.9.0 -filetype obj -main-file-name - -target-cpu core2 -dwarf-version=2 -fdebug-compilation-dir /Users/fbissey/build/sage-7.2.beta5 -dwarf-debug-producer Apple LLVM version 7.3.0 (clang-703.0.29) -mrelocation-model pic -o a.out -
W.r.t. script solutions:
Putting one of the following into $SAGE_ROOT/local/bin/as
(and of course chmod +x
it) should work.
#!/bin/sh
exec /usr/bin/as -q "$@"
#!/bin/sh
exec `clang -print-prog-name=as` "$@"
I'm not sure whether the first at all works if /usr/bin/as
is already LLVM's as
(e.g. on 10.11; François?), and one could hardcode the path to as
in the second (perhaps unless it is shipped with a binary distribution of Sage).
P.S.:
At least both versions avoid using -x assembler
etc. and checking whether filenames were given or the input comes from a pipe.
Testing stuff on 10.11.5....
It's taking more time to do it right than I had expected. Especially when you realize you messed up after a full build on this little laptop...
Testing on 10.9...
Replying to @NathanDunfield:
Testing on 10.9...
Still working on this, but a quick comment: The second script in comment [comment:21] actually invokes the default gcc-based as
on 10.9, not clang's. The following gets clang's as
but I'm still seeing if this fixes the problem at hand:
#!/usr/bin/env bash
clang -print-prog-name=as "$@"
Replying to @NathanDunfield:
Replying to @NathanDunfield:
Testing on 10.9...
Still working on this, but a quick comment: The second script in comment [comment:21] actually invokes the default gcc-based
as
on 10.9, not clang's. The following gets clang'sas
but I'm still seeing if this fixes the problem at hand:#!/usr/bin/env bash clang -print-prog-name=as "$@"
Does clang -print-prog-name=as
give a full pathname on your system? (If no specific assembler or other program was configured during the build, the "input" just gets echoed verbatim, no matter whether such a program exists at all, and even the return value is zero.)
Does llvm-as
exist on your system (not necessarily along $PATH
)?
Does
clang -print-prog-name=as
give a full pathname on your system?
Yes, it gives /Library/Developer/CommandLineTools/usr/bin/as
which is the gcc as
. On a different 10.9 machine which has Xcode installed, not just the command line tools, I get:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/as
which is again the gcc as
.
Does
llvm-as
exist on your system (not necessarily along$PATH
)?
No, or at least not as far as I can tell. In particular, there no llvm-as
(or any renamed version of such) in /Library/Developer/CommandLineTools/usr/bin/
which where it should be if it exists.
On 10.10 and newer, the version of as
at /Library/Developer/CommandLineTools/usr/bin/
becomes clang's assembler.
I realized my replacement script doesn't work either --- I guess we have do something with the -x assembler
flag
or similar?
We could check whether clang -print-prog-name=as
still gives Apple's version of GAS, and if so, use as -q
. (I wonder what the latter calls in that case; is /usr/bin/as
a script in that case?)
P.S.: I didn't know it is at all possible to install the command line tools without installing Xcode. Does Sage build without having Xcode installed?
Replying to @nexttime:
We could check whether
clang -print-prog-name=as
still gives Apple's version of GAS, and if so, useas -q
.
I'll try as -q
on 10.9 and report back.
(I wonder what the latter calls in that case; is
/usr/bin/as
a script in that case?
On 10.9 /usr/bin/as
is a tiny 14K executable, so I guess it must function basically as a shell script.
P.S.: I didn't know it is at all possible to install the command line tools without installing Xcode...
This has been possible for a while. On clean 10.9 machine if you type "clang" in Terminal.app, then a dialog pops up to ask if you want to install the command line tools, though there is also a button to install the whole of Xcode.
Does Sage build without having Xcode installed?
I'm about to find out ;-). It should, I think. At the very least, it can build everything up to NTL.
FWIW, on Linux at least, llvm-as
(or whatever it is called on the system) is quite small, too (~16 KB), since most functionality is contained in an LLVM shared library it uses.
Ok, with Sage 7.3.beta5, which includes this version of NTL, the following worked to build Sage on 10.9:
Get source.
Create $SAGE_ROOT/local/bin and put a script as
consisting of
#!/usr/bin/env bash
exec /usr/bin/as -q "$@"
make
as usual. I'm running the doctests now, and will report back when those are done. It looks good so far, though.
Replying to @NathanDunfield:
Ok, with Sage 7.3.beta5, which includes this version of NTL, the following worked to build Sage on 10.9:
Get source.
Create $SAGE_ROOT/local/bin and put a script
as
consisting of#!/usr/bin/env bash exec /usr/bin/as -q "$@"
- Type
make
as usual.
Yep, we could create such a script in Sage's configure
when necessary (i.e., on MacOS X, if the default as
is still GAS).
I still don't know whether as -q ...
works on more recent versions (where it shouldn't be necessary). While we could check that in configure
for source builds, that doesn't help with binary dists. If it doesn't break anything on 10.10 and later, we could simply include the script in (all) binary distributions of Sage for MacOS X.
Not sure whether we have to require Apple's "command line tools" and Xcode then (at least for builds from source and "developing" with bdists).
All of this is IMHO still orthogonal to fixing NTL's -march=native
check (which apparently simply tests whether the compiler accepts the option, but nothing beyond that -- I haven't looked into the code though).
P.S.: Where "developing with bdists" includes installing optional Sage packages that involves compilation of C/C++ files, cf. #20563.
I still don't know whether
as -q ...
works on more recent versions (where it shouldn't be necessary). While we could check that inconfigure
for source builds, that doesn't help with binary dists. If it doesn't break anything on 10.10 and later, we could simply include the script in (all) binary distributions of Sage for MacOS X.
I checked that on 10.10 and 10.11 that as
accepts -q
without complaining. I haven't yet tried building Sage this way, though.
Not sure whether we have to require Apple's "command line tools" and Xcode then (at least for builds from source and "developing" with bdists).
Definitely just the command-line tools --- I didn't have Xcode installed on the 10.9 system referred to above.
All of this is IMHO still orthogonal to fixing NTL's
-march=native
check (which apparently simply tests whether the compiler accepts the option, but nothing beyond that -- I haven't looked into the code though).
I think it's reasonable on NTL's part to assume that if the compiler accepts the option then it generates working code --- I would argue that the issue here on OS X is a mismatch or if you prefer a misconfiguration between the compiler and the assembler.
I haven't tested yet, but I think as -q
might also resolve #20563, so two birds with one stone and all that.
Replying to @NathanDunfield:
All of this is IMHO still orthogonal to fixing NTL's
-march=native
check (which apparently simply tests whether the compiler accepts the option, but nothing beyond that -- I haven't looked into the code though).I think it's reasonable on NTL's part to assume that if the compiler accepts the option then it generates working code.
:-) It does generate "working" code in the sense that the assembly output is valid (even in the case of #20563 btw.).
But the test should check whether the whole toolchain works, and -- unless cross-compiling, where -march=native
wouldn't make sense anyway -- whether such a program can be run on the machine.
Besides hardware bugs, there've also frequently been issues with VMs. (And note that for example AVX requires operating system support, not just an AVX-capable CPU.)
I haven't tested yet, but I think
as -q
might also resolve #20563, so two birds with one stone and all that.
The (newer) LLVM assembler certainly complies with Intel's AVX specification... ;-)
Replying to @NathanDunfield:
Ok, with Sage 7.3.beta5, which includes this version of NTL, the following worked to build Sage on 10.9: [...] I'm running the doctests now, and will report back when those are done.
All tests pass! I did skip the long ones, will run those now. Tomorrow, I will test on 10.11 as well.
I actually already done so on 10.11. build succeeded and sage starts. Haven't run doctests though but I would think it is all ok.
Replying to @NathanDunfield:
Ok, with Sage 7.3.beta5, which includes this version of NTL, the following worked to build Sage on 10.9: [...] I'm running the doctests now, and will report back when those are done.
All tests pass! I did skip the long ones, will run those now.
There was one failure in the long doctests, but probably presumably not related to the change at hand:
sage -t --long --warn-long 24.0 src/sage/parallel/map_reduce.py
**********************************************************************
File "src/sage/parallel/map_reduce.py", line 1082, in sage.parallel.map_reduce.RESetMapReduce.start_workers
Failed example:
all(w.is_alive() for w in S._workers)
Expected:
True
Got:
False
**********************************************************************
So assuming we want to add this as
script, how would we go about it? I'm completely ignorant about Sage's bootstrap/config process...
This now hits us on CentOS (6.8, with apparently ancient binutils) as well:
Host system:
Linux pmlapsag01 2.6.32-642.1.1.el6.x86_64 #1 SMP Tue May 31 21:57:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/home/sageadm/sage/local/libexec/gcc/x86_64-unknown-linux-gnu/4.9.3/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../src/configure --prefix=/home/sageadm/sage/local --with-local-prefix=/home/sageadm/sage/local --with-gmp=/home/sageadm/sage/local --with-mpfr=/home/sageadm/sage/local --with-mpc=/home/sageadm/sage/local --with-system-zlib --disable-multilib --disable-nls --enable-languages=c,c++,fortran --disable-libitm
Thread model: posix
gcc version 4.9.3 (GCC)
****************************************************
Applying patches to NTL.
....
config.status: executing libtool commands
Configuring NTL.
***** checking for libtool *****
libtool (GNU libtool) 2.4.2
Written by Gordon Matzigkeit < ... >, 1996
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***** libtool OK *****
*** checking -march=native flag
CXXAUTOFLAGS=" -march=native"
*** -march=native works
CXXAUTOFLAGS=" -march=native"
[ntl-9.8.1.p0] Tuning and building NTL.
[ntl-9.8.1.p0] make[3]: Entering directory `/home/sageadm/sage/local/var/tmp/sage/build/ntl-9.8.1.p0/src/ntl/src'
[ntl-9.8.1.p0] make setup1
[ntl-9..8.1.p0] make[4]: Entering directory `/home/sageadm/sage/local/var/tmp/sage/build/ntl-9.8.1.p0/src/ntl/src'
[ntl-9.8.1.p0] g++ -I../include -I. -march=native -O2 -g -c MakeDescAux.c
[ntl-9.8.1.p0] g++ -I../include -I. -march=native -O2 -g -L/home/sageadm/sage/local/lib -Wl,-rpath,/home/sageadm/sage/local/lib -o MakeDesc MakeDesc.c MakeDescAux.o -lm
[ntl-9.8.1.p0] /tmp/ccPRNw4z.s: Assembler messages:
[ntl-9.8.1.p0] /tmp/ccPRNw4z.s:873: Error: no such instruction: `shlx %r12,%rax,%rax'
[ntl-9.8.1.p0] /tmp/ccPRNw4z.s:932: Error: no such instruction: `shlx %r12,%rax,%rax'
...
[ntl-9.8.1.p0] /tmp/ccPRNw4z.s:4374: Error: no such instruction: `shlx %rax,%r15,%r15'
[ntl-9.8.1.p0] /tmp/ccPRNw4z.s:4791: Error: no such instruction: `shlx %r13,%rdx,%rdx'
[ntl-9.8.1.p0] make[4]: *** [setup1] Error 1
(From sage-devel.)
Replying to @nexttime:
All of this is IMHO still orthogonal to fixing NTL's
-march=native
check (which apparently simply tests whether the compiler accepts the option, but nothing beyond that -- I haven't looked into the code though).
Because this now hits us on older Linux distros as well (where Sage's GCC gets built), I think we should fix the NTL package (or any package which tries to use -march=native
) quickly on another ticket (making it a blocker for Sage 7.3).
Replying to @nexttime:
Because this now hits us on older Linux distros as well (where Sage's GCC gets built), I think we should fix the NTL package (or any package which tries to use
-march=native
) quickly on another ticket (making it a blocker for Sage 7.3).
I think it would be fine to use the current ticket to fix NTL on both Linux and OS X, assuming it's essentially the same fix on both platforms. Messing with clang's asm on OS X can go under #20563.
Replying to @NathanDunfield:
Replying to @nexttime:
Because this now hits us on older Linux distros as well (where Sage's GCC gets built), I think we should fix the NTL package (or any package which tries to use
-march=native
) quickly on another ticket (making it a blocker for Sage 7.3).I think it would be fine to use the current ticket to fix NTL on both Linux and OS X, assuming it's essentially the same fix on both platforms. Messing with clang's asm on OS X can go under #20563.
From the ticket's title, yes, but the comments here mainly deal with clang
and MacOS X specifics, so I'd prefer to leave it open for a solution to the latter on this ticket (rather than #20563).
Replying to @nexttime:
From the ticket's title, yes, but the comments here mainly deal with
clang
and MacOS X specifics, so I'd prefer to leave it open for a solution to the latter on this ticket (rather than #20563).
Ok, that makes sense. If you put me in the "cc" for the new ticket I can test on OS X.
Hmmm, still haven't pushed a branch to #21064 (still playing a little with improvements)...
I wonder what as -q -v </dev/null
gives on older versions of MacOS X where the default as
is still Apple's GAS.
Does it print its own version, or does it invoke LLVM's as
, and if so, what does the latter give?
(At least older versions of LLVM's assembler on Linux, probably not on MacOS X, don't know/accept -v
, but -version
as well as --version
.)
And does as -v -q </dev/null
(options swapped) behave the same?
I wonder what
as -q -v </dev/null
gives on older versions of MacOS X where the defaultas
is still Apple's GAS.Does it print its own version, or does it invoke LLVM's
as
, and if so, what does the latter give?
It's LLVM's as
that's invoked (seen here on 10.9):
% as -v -q < /dev/null
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1as -triple x86_64-apple-macosx10.9.0 -filetype obj -main-file-name - -target-cpu core2 -fdebug-compilation-dir /Users/dunfield -dwarf-debug-producer Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn) -o a.out -
And does
as -v -q </dev/null
(options swapped) behave the same?
Yes, it does. The -version
flag is not accepted in either order, but --version
and produces:
% as -q --version < /dev/null
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix
P.S.: A funny interpretation of -q
on Apple's side by the way; to the original GNU assembler, -q
means "quieten some warnings" (not including errors, I guess)... XD
FYI (in case you don't already know it):
This nice list maps Xcode versions to (Apple) clang versions at least.
We could use it to get the Xcode version from clang --version
, and in conjunction with data from wikipedia, could check whether it is (one of) the most recent version(s) available for MacOS X 10.N (aka Darwin N+4 IIRC).
(I haven't found any information on the assembler versions / when Apple's GAS vanished, though.)
According to that, Travis' friend (with as
being GNU's) was using something between Xcode 6.3 and 6.4 (inclusive; cctools-870) on "El Capitan", while for that, 7.x (and 8.0 beta) are available since about ten months (or e.g. four months for 7.3).
The problem is that the flag "-march=native" is now given to the compiler, causing GCC to generate assembly code that is not accepted by the ancient version of "as" that ships with 10.8 and 10.9. See
https://groups.google.com/forum/#!topic/sage-devel/FloU3fAgQLM
for a discussion and some potential solutions. Currently, it seems unknown if this NTL builds on 10.10 but it is known to do so on 10.11.
From the above mentioned thread:
If you pass
gcc
the flag-Wa,-q
, then it will use clang's assembler rather than the ancient version of the GNU assembler that comes with these versions of OS X. Seehttp://stackoverflow.com/questions/9840207
''I can confirm that adding this flag allowed me to compile all of NTL on these machines. Specifically, changing line 78 of
build/pkgs/ntl/spkg-install
tocaused NTL to build smoothly.
Nathan
CC: @kcrisman @unhyperbolic @kiwifb
Component: algebra
Keywords: MacOS AVX no such instruction
Issue created by migration from https://trac.sagemath.org/ticket/20779