Update GMP-ECM to version 7.0

[zimmermann6]

GMP-ECM 7.0.4 is available at https://gforge.inria.fr/frs/download.php/file/36224/ecm-7.0.4.tar.gz.

---

(This was referring to version 7.0:)

Some issues were reported for the Debian package:

https://buildd.debian.org/status/package.php?p=gmp-ecm, in particular on 32-bit PowerPC and FreeBSD. Is there any such machine on the Sage build farm?

---

Upgrade GMP-ECM to 7.0.4

• I haven't repackaged the upstream tarball (.tar.gz) [yet]. Not sure whether we could/should also delete some stuff not needed for Sage (cf. SPKG.txt); at least there's no spkg-src script.

• I haven't touched spkg-install yet. We may address potential issues with SAGE_FAT_BINARY (mentioned both there and in SPKG.txt), as well as known problems with -march=native -O3 on bdver[1-4] (AMD FX-???? !Bulldozer/Piledriver, AMD Opteron 6[23]xx, etc.) where we should add -mno-xop because of bugs in (AFAIK) every GCC version that supports these processors, silently leading to broken code ("just" testsuite failures IIRC). Also, we currently copy Sage's config.guess (because it was newer at some point), which presumably is no longer necessary (but shouldn't hurt either, provided we keep Sage's up-to-date and GMP-ECM won't rely on its own).

• The doctests in sage/libs/libecm.pyx may need some changes, especially with respect to verbose mode tests. (But I'll presumably work on them elsewhere anyway, independently of this upgrade.)

• We definitely also have to fix the pexpect interface to ecm in src/sage/interfaces/ecm.py, as the command line interface has changed in GMP-ECM 7.x.

CC: @kiwifb

Component: packages: standard

Author: Leif Leonhardy, Jeroen Demeyer, François Bissey, Antonio Rojas

Branch: 946c580

Reviewer: Jean-Pierre Flori

Issue created by migration from https://trac.sagemath.org/ticket/20385

[kiwifb]

comment:44

OS X doesn't like this thing much

/bin/sh ./libtool  --tag=CC   --mode=link gcc  -g -march=native -g -O3  -fPIC  -version-info 1:0:0 -g -Wl,-znoexecstack  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -Wl,-rpath,/Users/fbissey/build/sage-7.2.beta5/local/lib   -o libecm.la -rpath /Users/fbissey/build/sage-7.2.beta5/local/lib libecm_la-ecm.lo libecm_la-ecm2.lo libecm_la-pm1.lo libecm_la-pp1.lo libecm_la-getprime_r.lo libecm_la-listz.lo libecm_la-lucas.lo libecm_la-stage2.lo libecm_la-mpmod.lo libecm_la-mul_lo.lo libecm_la-polyeval.lo libecm_la-median.lo libecm_la-schoen_strass.lo libecm_la-ks-multiply.lo libecm_la-rho.lo libecm_la-bestd.lo libecm_la-auxlib.lo libecm_la-random.lo libecm_la-factor.lo libecm_la-sp.lo libecm_la-spv.lo libecm_la-spm.lo libecm_la-mpzspm.lo libecm_la-mpzspv.lo libecm_la-ntt_gfp.lo libecm_la-ecm_ntt.lo libecm_la-pm1fs2.lo libecm_la-sets_long.lo libecm_la-auxarith.lo libecm_la-batch.lo libecm_la-parametrizations.lo libecm_la-cudawrapper.lo aprtcle/libecm_la-mpz_aprcl.lo   ./x86_64/libmulredc.la  /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm -lm -lm -lm -lm  

*** Warning: Linking the shared library libecm.la against the
*** static library /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a is not portable!
libtool: link: (cd .libs/libecm.lax/libmulredc.a && ar x "/Users/fbissey/build/sage-7.2.beta5/local/var/tmp/sage/build/ecm-7.0.3/src/./x86_64/.libs/libmulredc.a")
libtool: link: ar cru .libs/libecm.a /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libecm.a(libecm_la-cudawrapper.o) has no symbols
libtool: link: ranlib .libs/libecm.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libecm.a(libecm_la-cudawrapper.o) has no symbols
libtool: link: rm -fr .libs/libecm.lax
libtool: link: ( cd ".libs" && rm -f "libecm.la" && ln -s "../libecm.la" "libecm.la" )
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-auxi.o -MD -MP -MF .deps/ecm-auxi.Tpo -c -o ecm-auxi.o `test -f 'auxi.c' || echo './'`auxi.c
mv -f .deps/ecm-auxi.Tpo .deps/ecm-auxi.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-b1_ainc.o -MD -MP -MF .deps/ecm-b1_ainc.Tpo -c -o ecm-b1_ainc.o `test -f 'b1_ainc.c' || echo './'`b1_ainc.c
mv -f .deps/ecm-b1_ainc.Tpo .deps/ecm-b1_ainc.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-candi.o -MD -MP -MF .deps/ecm-candi.Tpo -c -o ecm-candi.o `test -f 'candi.c' || echo './'`candi.c
mv -f .deps/ecm-candi.Tpo .deps/ecm-candi.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-eval.o -MD -MP -MF .deps/ecm-eval.Tpo -c -o ecm-eval.o `test -f 'eval.c' || echo './'`eval.c
mv -f .deps/ecm-eval.Tpo .deps/ecm-eval.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-main.o -MD -MP -MF .deps/ecm-main.Tpo -c -o ecm-main.o `test -f 'main.c' || echo './'`main.c
mv -f .deps/ecm-main.Tpo .deps/ecm-main.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-resume.o -MD -MP -MF .deps/ecm-resume.Tpo -c -o ecm-resume.o `test -f 'resume.c' || echo './'`resume.c
mv -f .deps/ecm-resume.Tpo .deps/ecm-resume.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-getprime_r.o -MD -MP -MF .deps/ecm-getprime_r.Tpo -c -o ecm-getprime_r.o `test -f 'getprime_r.c' || echo './'`getprime_r.c
mv -f .deps/ecm-getprime_r.Tpo .deps/ecm-getprime_r.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT aprtcle/ecm-mpz_aprcl.o -MD -MP -MF aprtcle/.deps/ecm-mpz_aprcl.Tpo -c -o aprtcle/ecm-mpz_aprcl.o `test -f 'aprtcle/mpz_aprcl.c' || echo './'`aprtcle/mpz_aprcl.c
mv -f aprtcle/.deps/ecm-mpz_aprcl.Tpo aprtcle/.deps/ecm-mpz_aprcl.Po
gcc -DHAVE_CONFIG_H -I.  -DOUTSIDE_LIBECM -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include  -g -march=native -g -O3  -fPIC -MT ecm-memusage.o -MD -MP -MF .deps/ecm-memusage.Tpo -c -o ecm-memusage.o `test -f 'memusage.c' || echo './'`memusage.c
mv -f .deps/ecm-memusage.Tpo .deps/ecm-memusage.Po
/bin/sh ./libtool  --tag=CC   --mode=link gcc  -g -march=native -g -O3  -fPIC  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -Wl,-rpath,/Users/fbissey/build/sage-7.2.beta5/local/lib   -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o libecm.la /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm -lm -lm -lm -lm  
libtool: link: gcc -g -march=native -g -O3 -fPIC -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib -o ecm ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o  -L/Users/fbissey/build/sage-7.2.beta5/local/lib ./.libs/libecm.a /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm
ld: warning: ignoring file ./.libs/libecm.a, file was built for archive which is not the architecture being linked (x86_64): ./.libs/libecm.a
Undefined symbols for architecture x86_64:
  "___ecm_cputime", referenced from:
      _main in ecm-main.o
  "_ecm_clear", referenced from:
      _main in ecm-main.o
  "_ecm_factor", referenced from:
      _main in ecm-main.o
  "_ecm_init", referenced from:
      _main in ecm-main.o
  "_ecm_version", referenced from:
      _main in ecm-main.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [ecm] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Error building GMP-ECM.

It is not obvious that znoexecstack is at fault but I cannot rule it out.

I am otherwise in the camp that says dynamic linking is actually the lesser trouble :)

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:45

P.S.: Actually libgmp.a should get linked into the Python extension module as well, but that's currently not the case.

[kiwifb]

comment:46

Interesting znoexecstack is not a problem building static - it is probably just ignored but build shared with ECM_CONFIGURE="--enable-shared --disable-static

libtool: link: gcc -dynamiclib -Wl,-undefined -Wl,dynamic_lookup -o .libs/libecm.1.dylib  .libs/libecm_la-ecm.o .libs/libecm_la-ecm2.o .libs/libecm_la-pm1.o .libs/libecm_la-pp1.o .libs/libecm_la-getprime_r.o .libs/libecm_la-listz.o .libs/libecm_la-lucas.o .libs/libecm_la-stage2.o .libs/libecm_la-mpmod.o .libs/libecm_la-mul_lo.o .libs/libecm_la-polyeval.o .libs/libecm_la-median.o .libs/libecm_la-schoen_strass.o .libs/libecm_la-ks-multiply.o .libs/libecm_la-rho.o .libs/libecm_la-bestd.o .libs/libecm_la-auxlib.o .libs/libecm_la-random.o .libs/libecm_la-factor.o .libs/libecm_la-sp.o .libs/libecm_la-spv.o .libs/libecm_la-spm.o .libs/libecm_la-mpzspm.o .libs/libecm_la-mpzspv.o .libs/libecm_la-ntt_gfp.o .libs/libecm_la-ecm_ntt.o .libs/libecm_la-pm1fs2.o .libs/libecm_la-sets_long.o .libs/libecm_la-auxarith.o .libs/libecm_la-batch.o .libs/libecm_la-parametrizations.o .libs/libecm_la-cudawrapper.o aprtcle/.libs/libecm_la-mpz_aprcl.o   -Wl,-force_load,./x86_64/.libs/libmulredc.a  -L/Users/fbissey/build/sage-7.2.beta5/local/lib -lgmp -lm  -g -march=native -g -O3 -g -Wl,-znoexecstack -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib   -install_name  /Users/fbissey/build/sage-7.2.beta5/local/lib/libecm.1.dylib -compatibility_version 2 -current_version 2.0 -Wl,-single_module
ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

And that definitely kills the linking. So we will need to remove it from upstream sources.

Yes once you link static stuff you need to provide all the bits (dependencies) since static libraries are dumb objects.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:47

Presumably unrelated, but 7.2.beta5?

What "architecture" in Apple speak do the object files in libecm.a have?

[kiwifb]

comment:48

Replying to @nexttime:

Presumably unrelated, but 7.2.beta5?

That's just the tarball I initially downloaded, but I pulled the latest develop and then your branch. I fully rebuild not so long ago.

What "architecture" in Apple speak do the object files in libecm.a have?

Good question, I'll see if I can figure it out.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:49

Replying to @kiwifb:

Interesting znoexecstack is not a problem building static - it is probably just ignored

It's not used at all when building static stuff, or more precisely, only when building libecm.la.

but build shared with ECM_CONFIGURE="--enable-shared --disable-static

ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

And that definitely kills the linking.

I told you so... ;-)

So we will need to remove it from upstream sources.

Well, patch a comma into it. (It's in Makefile.am and hence also in what's built from it.)

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:50

Replying to @nexttime:

Replying to @kiwifb:

ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

Well, patch a comma into it. (It's in Makefile.am and hence also in what's built from it.)

That is, in the source tarball only one instance in the top-level Makefile.in.

I wouldn't even use patch for that... ;-)

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:51

But I'm really in favour of submitting something upstream that in configure adds -Wa,--noexecstack to CCASFLAGS when supported (or maybe -Wl,-z,noexecstack to some *LDFLAGS).

[kiwifb]

comment:52

Replying to @nexttime:

Replying to @nexttime:

Replying to @kiwifb:

ld: unknown option: -znoexecstack
collect2: error: ld returned 1 exit status

Well, patch a comma into it. (It's in Makefile.am and hence also in what's built from it.)

That is, in the source tarball only one instance in the top-level Makefile.in.

I wouldn't even use patch for that... ;-)

May be you wouldn't. But in terms of sage, I would do a spkg-src. You can only be that dirty in a big collaborative project of that kind for so long. That kind of stuff just doesn't scale and hurt you in the long term.

In my opinion the whole spkg-install is overdone. Just let the build system do its job unless you want to produce a binary.

+1 for a report upstream.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:53

It's just always a headache to patch autotools files, and spkg-src is IMHO not a good idea unless you pull some svn version (as opposed to a released tarball), or really remove loads of unnecessary files in order to keep the tarball small.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:54

(Sage's "build system" doesn't yet support upstream tarball URLs though, and the "design" of for example checksums.ini is ... -- can't say that here.)

[kiwifb]

comment:55

No comment. I am the sage-on-gentoo maintainer - anything I say on the matter is kind of political.

Back to our little problem on OS X. It looks like it is something done in spkg-install, going to a sage shell and the build location I just do

make distclean
./configure # by default only build static
make

is successful and

(sage-sh) fbissey@Mirage:src$ make check
Making check in x86_64
make[1]: Nothing to be done for `check'.
make  ecm \
      test.pp1 test.pm1 test.ecm 
make[2]: `ecm' is up to date.
make[2]: Nothing to be done for `test.pp1'.
make[2]: Nothing to be done for `test.pm1'.
make[2]: Nothing to be done for `test.ecm'.
make  check-TESTS
PASS: test.pp1
PASS: test.pm1
PASS: test.ecm
============================================================================
Testsuite summary for ecm 7.0.3
============================================================================
# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

so it is definitely the configuration/environment from spkg-install, the configuration picked up sage's gcc as the compiler and sage's mpir as gmp without a hitch and assembly code has been used.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

6741ee0

GMP-ECM: #20385: Add temporary patch for '-Wl,znoexecstack' [sic] (Makefile.{am,in})

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0affd29 to 6741ee0

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:57

#20385:  Fix currently hardcoded '-Wl,znoexecstack' because
there's a space missing.  (E.g. Apple's linker bails out on
'-znoexecstack', while GNU ld just treats it as if it was
'-z noexecstack', the correct form.)

This is an ugly, temporary fix until we submit something
better upstream.

First of all, 'configure' should check whether this is / which
options are supported and add them based on that, and it's
presumably better to just put '-Wa,--noexecstack' into
CCASFLAGS when the assembler supports that option.

diff -Naur ecm-7.0.3.orig/Makefile.am ecm-7.0.3/Makefile.am
--- ecm-7.0.3/Makefile.am   2016-06-27 23:22:08.000000000 -0700
+++ ecm-7.0.3/Makefile.am   2016-08-03 16:02:21.901175462 -0700
@@ -50,7 +50,7 @@
 # www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html
 # If any interfaces have been added, removed, or changed since the last
 # update, increment current, and set revision to 0.
-libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g -Wl,-znoexecstack
+libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g -Wl,-z,noexecstack
 libecm_la_LIBADD = $(MULREDCLIBRARY)
 if WANT_GPU 
   libecm_la_SOURCES += cudakernel.cu
diff -Naur ecm-7.0.3.orig/Makefile.in ecm-7.0.3/Makefile.in
--- ecm-7.0.3/Makefile.in   2016-07-04 01:02:58.000000000 -0700
+++ ecm-7.0.3/Makefile.in   2016-08-03 16:03:13.181175501 -0700
@@ -715,7 +715,7 @@
 # If any interfaces have been added, removed, or changed since the last
 # update, increment current, and set revision to 0.
 libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g \
-   -Wl,-znoexecstack $(am__append_5)
+   -Wl,-z,noexecstack $(am__append_5)
 libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4) $(GMPLIB)
 @WANT_GPU_TRUE@ecm_LDFLAGS = $(CUDARPATH)
 @WITH_GWNUM_TRUE@ecm_LDFLAGS = $(AM_LDFLAGS) -Wl,gwdata.ld

---

New commits:

6741ee0

GMP-ECM: #20385: Add temporary patch for '-Wl,znoexecstack' [sic] (Makefile.{am,in})

[kiwifb]

comment:58

With that I have

ld: unknown option: -z

now when enabling shared libraries on OS X. At least that's consistent with the man page.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:59

Replying to @kiwifb:

Back to our little problem on OS X. It looks like it is something done in spkg-install, going to a sage shell and the build location I just do

make distclean
./configure # by default only build static
make

is successful [...] so it is definitely the configuration/environment from spkg-install, the configuration picked up sage's gcc as the compiler and sage's mpir as gmp without a hitch and assembly code has been used.

Well, post the part of the log where spkg-install prints all the relevant environment variables and configure options passed.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:60

Replying to @kiwifb:

With that I have

ld: unknown option: -z

now when enabling shared libraries on OS X. At least that's consistent with the man page.

What does gcc -c -Wa,--noexecstack foo.c give? (foo.c at your like, e.g. void foo() {}.)

(And probably also gcc -v -o foo ... without -c.)

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:61

Replying to @nexttime:

(And probably also gcc -v -o foo ... without -c.)

Ahem, forget that.

[kiwifb]

comment:62

clang: error: unsupported option '--noexecstack'

Not in the manual either.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:63

And/or attach the log of the last failed build here.

[kiwifb]

ecm log - bare spkg-install run

[kiwifb]

Attachment: ecm.log

Attachment: config.log

config.log matching ecm.log

[kiwifb]

comment:64

Done. So just plain ./sage -p ecm logs. No funky things in ECM_CONFIGURE.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:65

• Attachment ecm.log​ added ecm log - bare spkg-install run

Interestingly, ld there didn't bail out on -z noexecstack, for whatever reason. (Did it at all get invoked?)

Otherwise I'm stuck at libtool weirdness and what Apple's ld considers "not x86_64 architecture".

Also, while we have

checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no

only -fPIC is used (which we add in spkg-install).

And libtool of course^TM keeps building .o and .lo files, and both foo.a and foo.la.

[kiwifb]

comment:66

Found the bit that cause problem --with-gmp=${SAGE_LOCAL} without it configure tells me

configure: Configuration:
configure: Build for host type x86_64-apple-darwin15.6.0
configure: CC=gcc, CFLAGS=-g -O2
configure: Linking GMP with -lgmp

and the build succeeds. With it

configure: Configuration:
configure: Build for host type x86_64-apple-darwin15.6.0
configure: CC=gcc, CFLAGS=-g -O2
configure: Linking GMP with /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a

and the build fails....

[kiwifb]

comment:67

Rebuilding develop branch from scratch to make sure mpir is as sane as it can be.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:68

Replying to @kiwifb:

Found the bit that cause problem --with-gmp=${SAGE_LOCAL} without it configure tells me

configure: Configuration:
configure: Build for host type x86_64-apple-darwin15.6.0
configure: CC=gcc, CFLAGS=-g -O2
configure: Linking GMP with -lgmp

I thought of that, but you told me "the configuration picked up sage's gcc as the compiler and sage's mpir as gmp without a hitch" and I was wondering whether that's due to Sage GCC's configuration.

But before my internet connection went down the drain 8/ I was about to post:

---

And then I'd need a log from your successful "manual" build (make distclean && ./configure && make) from within a Sage subshell to compare with.

OTOH the only (major) difference -- regarding the environment when configure is invoked -- should be -fPIC in CFLAGS, as ECM picks up MPIR's when CFLAGS are empty / not set (which is the case in a Sage subshell unless you've set them yourself).

Not sure how libtool reacts to that, or whether there's anything in configure that's influenced by that.

---

Now clearly the difference seems to (also) be that you don't have a static libgmp installed system-wide.

[kiwifb]

comment:69

I don't even have a dynamic one.

The environment passes -L${SAGE_LOCAL}/lib and -Wl,-rpath=${SAGE_LOCAL}/lib, further the configure script identifies gmp as being in fact mpir in both case. So the -lgmp picked can only be from ${SAGE_LOCAL}/lib.

Even further when outside of the sage shell configure bails out because it cannot find any gmp. So the only gmp available is in sage's install.

On another front I separately tried to add -fPIC to CFLAGS for configuration and that wasn't a problem. Just adding -fPIC lead to a successful compilation.

libgmp.a or the process of using libgmp.a wholesale is causing trouble. Using libgmp.a that way tries to fold the gmp archive inside libecm.a (so you don't need to link with libgmp.a as it is already inside), apparently it causes trouble on OS X. Possibly something in mpir rather than ecm in my opinion.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:70

Replying to @kiwifb:

libgmp.a or the process of using libgmp.a wholesale is causing trouble. Using libgmp.a that way tries to fold the gmp archive inside libecm.a (so you don't need to link with libgmp.a as it is already inside), apparently it causes trouble on OS X. Possibly something in mpir rather than ecm in my opinion.

It shouldn't. You cannot "fold" a static library into another, you can include object files from one into another, or when you build a shared library, include object files into the former (as needed -- symbols get resolved and the object files from which they originate get included, all ending up in the same single text segment etc.).

Wondering whether we build MPIR's/GMP's static library also with PIC. (Btw., ECM also has --with-pic.)

My guess is rather that it's ECM's fault.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:71

Replying to @nexttime:

Wondering whether we build MPIR's/GMP's static library also with PIC.

Looks like it (MPIR, not explicitly we) does (-fPIC -DPIC).

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:72

Note that in

libtool: link: gcc -g -march=native -g -O3 -fPIC
    -Wl,-rpath -Wl,/Users/fbissey/build/sage-7.2.beta5/local/lib 
    -o ecm
    ecm-auxi.o ecm-b1_ainc.o ecm-candi.o ecm-eval.o ecm-main.o ecm-resume.o
    ecm-getprime_r.o aprtcle/ecm-mpz_aprcl.o ecm-memusage.o
    -L/Users/fbissey/build/sage-7.2.beta5/local/lib ./.libs/libecm.a
    /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a -lm
ld: warning: ignoring file ./.libs/libecm.a, file was built for archive which is not the architecture being linked (x86_64): ./.libs/libecm.a

the linker only complains about libecm.a, but not libgmp.a.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:73

But you're right, ECM actually adds libgmp.a to libecm.a:

libtool: link: ar cru .libs/libecm.a /Users/fbissey/build/sage-7.2.beta5/local/lib/libgmp.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o 

and that's presumably the entry Apple's ld complains about (and instead of ignoring the entry, it drops the whole archive, libecm.a).

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:74

I guess this is the culprit (in Makefile.am, line 61):

libecm_la_LIBADD += $(GMPLIB)

(In Makefile.in, on line 719.)

New tentative patch:

#20385:  Don't put $(GMPLIB) (=libgmp.a) into libecm.a, as Darwin's
linker bails out on nested archives.  Probably a weird idea anyway,
or some libtool bug, as it's presumably supposed to add *the contents*
(i.e., the object files) of libgmp.a into libecm.a.

diff -Naur ecm-7.0.3.orig/Makefile.am ecm-7.0.3/Makefile.am
--- ecm-7.0.3/Makefile.am   2016-06-27 23:22:08.000000000 -0700
+++ ecm-7.0.3/Makefile.am   2016-08-03 23:20:24.352113089 -0700
@@ -58,7 +58,7 @@
   libecm_la_LDFLAGS += $(CUDALDFLAGS)
   ecm_LDFLAGS = $(CUDARPATH)
 endif
-libecm_la_LIBADD += $(GMPLIB)
+# libecm_la_LIBADD += $(GMPLIB)

 bin_PROGRAMS = ecm
 noinst_PROGRAMS = tune ecmfactor bench_mulredc aprcl
diff -Naur ecm-7.0.3.orig/Makefile.in ecm-7.0.3/Makefile.in
--- ecm-7.0.3/Makefile.in   2016-07-04 01:02:58.000000000 -0700
+++ ecm-7.0.3/Makefile.in   2016-08-03 23:19:34.952113147 -0700
@@ -716,7 +716,7 @@
 # update, increment current, and set revision to 0.
 libecm_la_LDFLAGS = $(LIBECM_LDFLAGS) -version-info 1:0:0 -g \
    -Wl,-znoexecstack $(am__append_5)
-libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4) $(GMPLIB)
+libecm_la_LIBADD = $(MULREDCLIBRARY) $(am__append_4)
 @WANT_GPU_TRUE@ecm_LDFLAGS = $(CUDARPATH)
 @WITH_GWNUM_TRUE@ecm_LDFLAGS = $(AM_LDFLAGS) -Wl,gwdata.ld
 # Most binaries want to link libecm.la, and the ones which don't will

Note that this new patch gets applied before the other one, so I had to change the latter as well since they unfortunately overlap. (But the first I added will get dropped later anyway, if Darwin's linker doesn't understand -z ... and also errors out on it.)

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:75

I'll push the changes in a couple of minutes...

[kiwifb]

comment:76

Yes, removing that bit looks right. After all it is already added to LDFLAGS by configure. For the record I successfully installed ecm-7.0.3 on OS X with the following change

diff --git a/build/pkgs/ecm/spkg-install b/build/pkgs/ecm/spkg-install
index fc49b3d..7c1e42a 100755
--- a/build/pkgs/ecm/spkg-install
+++ b/build/pkgs/ecm/spkg-install
@@ -237,7 +237,6 @@ else
 fi
 echo "  --prefix=\"$SAGE_LOCAL\""
 echo "  --libdir=\"$SAGE_LOCAL/lib\""
-echo "  --with-gmp=\"$SAGE_LOCAL\""
 for opt in $ECM_CONFIGURE; do
     echo "  $opt"
 done
@@ -249,7 +248,6 @@ fi
 echo

 ./configure --prefix="$SAGE_LOCAL" --libdir="$SAGE_LOCAL/lib" \
-            --with-gmp="$SAGE_LOCAL" \
             $ECM_CONFIGURE
 if [ $? -ne 0 ]; then
     echo >&2 "Error configuring GMP-ECM."

Just doing that removes libgmp.a from the ar call

libtool: link: (cd .libs/libecm.lax/libmulredc.a && ar x "/Users/fbissey/build/sage-7.2.beta5/local/var/tmp/sage/build/ecm-7.0.3/src/./x86_64/.libs/libmulredc.a")
libtool: link: ar cru .libs/libecm.a  libecm_la-ecm.o libecm_la-ecm2.o libecm_la-pm1.o libecm_la-pp1.o libecm_la-getprime_r.o libecm_la-listz.o libecm_la-lucas.o libecm_la-stage2.o libecm_la-mpmod.o libecm_la-mul_lo.o libecm_la-polyeval.o libecm_la-median.o libecm_la-schoen_strass.o libecm_la-ks-multiply.o libecm_la-rho.o libecm_la-bestd.o libecm_la-auxlib.o libecm_la-random.o libecm_la-factor.o libecm_la-sp.o libecm_la-spv.o libecm_la-spm.o libecm_la-mpzspm.o libecm_la-mpzspv.o libecm_la-ntt_gfp.o libecm_la-ecm_ntt.o libecm_la-pm1fs2.o libecm_la-sets_long.o libecm_la-auxarith.o libecm_la-batch.o libecm_la-parametrizations.o libecm_la-cudawrapper.o aprtcle/libecm_la-mpz_aprcl.o  .libs/libecm.lax/libmulredc.a/mulredc1.o .libs/libecm.lax/libmulredc.a/mulredc10.o .libs/libecm.lax/libmulredc.a/mulredc11.o .libs/libecm.lax/libmulredc.a/mulredc12.o .libs/libecm.lax/libmulredc.a/mulredc13.o .libs/libecm.lax/libmulredc.a/mulredc14.o .libs/libecm.lax/libmulredc.a/mulredc15.o .libs/libecm.lax/libmulredc.a/mulredc16.o .libs/libecm.lax/libmulredc.a/mulredc17.o .libs/libecm.lax/libmulredc.a/mulredc18.o .libs/libecm.lax/libmulredc.a/mulredc19.o .libs/libecm.lax/libmulredc.a/mulredc1_10.o .libs/libecm.lax/libmulredc.a/mulredc1_11.o .libs/libecm.lax/libmulredc.a/mulredc1_12.o .libs/libecm.lax/libmulredc.a/mulredc1_13.o .libs/libecm.lax/libmulredc.a/mulredc1_14.o .libs/libecm.lax/libmulredc.a/mulredc1_15.o .libs/libecm.lax/libmulredc.a/mulredc1_16.o .libs/libecm.lax/libmulredc.a/mulredc1_17.o .libs/libecm.lax/libmulredc.a/mulredc1_18.o .libs/libecm.lax/libmulredc.a/mulredc1_19.o .libs/libecm.lax/libmulredc.a/mulredc1_2.o .libs/libecm.lax/libmulredc.a/mulredc1_20.o .libs/libecm.lax/libmulredc.a/mulredc1_3.o .libs/libecm.lax/libmulredc.a/mulredc1_4.o .libs/libecm.lax/libmulredc.a/mulredc1_5.o .libs/libecm.lax/libmulredc.a/mulredc1_6.o .libs/libecm.lax/libmulredc.a/mulredc1_7.o .libs/libecm.lax/libmulredc.a/mulredc1_8.o .libs/libecm.lax/libmulredc.a/mulredc1_9.o .libs/libecm.lax/libmulredc.a/mulredc2.o .libs/libecm.lax/libmulredc.a/mulredc20.o .libs/libecm.lax/libmulredc.a/mulredc3.o .libs/libecm.lax/libmulredc.a/mulredc4.o .libs/libecm.lax/libmulredc.a/mulredc5.o .libs/libecm.lax/libmulredc.a/mulredc6.o .libs/libecm.lax/libmulredc.a/mulredc7.o .libs/libecm.lax/libmulredc.a/mulredc8.o .libs/libecm.lax/libmulredc.a/mulredc9.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .libs/libecm.a(libecm_la-cudawrapper.o) has no symbols
libtool: link: ranlib .libs/libecm.a

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e115141

GMP-ECM: #20385: Add another patch avoiding nested archives and rebase the previous on it.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 6741ee0 to e115141

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:78

Pushed the new patch(es), and thereby deleted my comment I hadn't posted yet -- no idea what I had written... >8( Thanks, HTTPS!

... I seem to remember that I had speculated that ..._LIBADD = $(GMPLIB) would presumably have worked if $(GMPLIB) was libgmp.la, as libtool indeed unpacks $(MULREDCLIBRARY) and adds its contents to the libecm archive.

---

Still have to see what harm my patch does to a shared libecm; it mayget underlinked now.

[kiwifb]

comment:79

Replying to @nexttime:

Pushed the new patch(es), and thereby deleted my comment I hadn't posted yet -- no idea what I had written... >8( Thanks, HTTPS!

... I seem to remember that I had speculated that ..._LIBADD = $(GMPLIB) would presumably have worked if $(GMPLIB) was libgmp.la, as libtool indeed unpacks $(MULREDCLIBRARY) and adds its contents to the libecm archive.

---

Still have to see what harm my patch does to a shared libecm; it mayget underlinked now.

I wouldn't think so. Using libgmp.la means that it has to be present on the system. Relying on an installed, external to your project, .la file is extremely fragile. In any case I don't appear to have any lib{gmp,mpir}.la installed on OS X so it would fail - miserably.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:80

Replying to @nexttime:

Still have to see what harm my patch does to a shared libecm; it may get underlinked now.

Exactly that happens.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:81

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:82

Replying to @nexttime:

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

Rather libecm_la_LDADD += $(GMPLIB) I think.

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:83

Replying to @nexttime:

Replying to @nexttime:

Still have to see what harm my patch does to a shared libecm; it may get underlinked now.

Exactly that happens.

But AFAICS that wouldn't be a regression w.r.t. GMP-ECM 6.4.4 at least...

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:84

Replying to @nexttime:

Replying to @nexttime:

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

Rather libecm_la_LDADD += $(GMPLIB) I think.

François, could you test this (adding the latter to Makefile.am)?

I don't have automake 1.15 installed...

[kiwifb]

comment:85

Just looked at ecm's configure.ac. I am not impressed but I guess this is geared towards static linking. However the proper way tp link statically is not add $path_to/libname.a, if you have gcc you pass -static.

I can test adding that stuff, give me a few minutes (i may suffer from interruptions from a 5 year old that needs to go to bed).

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:86

Replying to @kiwifb:

Just looked at ecm's configure.ac. I am not impressed but I guess this is geared towards static linking. However the proper way to link statically is not add $path_to/libname.a, if you have gcc you pass -static.

But that apparently doesn't work on Darwin, as in your log we had:

checking if gcc static flag -static works... no

Not sure what libtool magic offers.

Passing foo.a is certainly safer and more portable.

[kiwifb]

comment:87

One to bed. libtool as an -all-static mode for linking program that use only static libraries, not sure how it applies on OS X. But you cannot do that with libraries. You don't link static libraries, you just just ar+ranlib or equivalent. Folding one library in another is probably frowned on by libtool.

[kiwifb]

comment:88

Replying to @nexttime:

Replying to @nexttime:

Replying to @nexttime:

libecm_la_LDFLAGS += $(GMPLIB) to avoid underlinking?

Rather libecm_la_LDADD += $(GMPLIB) I think.

François, could you test this (adding the latter to Makefile.am)?

I don't have automake 1.15 installed...

Didn't really like it

***** automake *****
***** PWD: /scratch2/portage/sci-mathematics/gmp-ecm-7.0.3-r1/work/ecm-7.0.3
***** automake --add-missing --copy --force-missing

configure.ac:161: installing './compile'
configure.ac:8: installing './missing'
Makefile.am:61: error: libecm_la_LDADD must be set with '=' before using '+='
Makefile.am:61: error: use 'libecm_la_LIBADD', not 'libecm_la_LDADD'
Makefile.am: installing './depcomp'
Makefile.am:61: warning: variable 'libecm_la_LDADD' is defined but no program or
Makefile.am:61: library has 'libecm_la' as canonical name (possible typo)

Using = instead of += doesn't really help the second message

***** automake *****
***** PWD: /scratch2/portage/sci-mathematics/gmp-ecm-7.0.3-r1/work/ecm-7.0.3
***** automake --add-missing --copy --force-missing

configure.ac:161: installing './compile'
configure.ac:8: installing './missing'
Makefile.am:61: error: use 'libecm_la_LIBADD', not 'libecm_la_LDADD'
Makefile.am: installing './depcomp'
Makefile.am:61: warning: variable 'libecm_la_LDADD' is defined but no program or
Makefile.am:61: library has 'libecm_la' as canonical name (possible typo)

[83660e46-0051-498b-a8c1-f7a7bd232b5a]

comment:89

While I've now installed automake 1.15, it removes

runstatedir = @runstatedir@

from Makefile.in (without me having made any changes to Makefile.am)...