numpy fails to build on OpenSolaris or Solaris 10 64-bit

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

Build environment

Sun Ultra 27 3.33 GHz Intel W3580 Xeon. Quad core. 8 threads. 12 GB RAM
OpenSolaris 2009.06 snv_111b X86
Sage 4.3.1 (with a few packages hacked to work on 64-bit)
gcc 4.3.4 configured with Sun linker and GNU assembler from binutils version 2.20.
64-bit build. SAGE64 was set to yes, plus various other tricks to get -m64 into packages.
The problem

numpy is failing to build. It looks like there is a mix of 32 and 64-bit objects.

numpy-1.3.0.p2/src/setup.py
numpy-1.3.0.p2/src/MANIFEST.in
numpy-1.3.0.p2/.hgtags
Finished extraction
****************************************************
Host system
uname -a:
SunOS hawk 5.11 snv_111b i86pc i386 i86pc
****************************************************
****************************************************
CC Version
gcc -v
Using built-in specs.
Target: i386-pc-solaris2.11
Configured with: ../gcc-4.3.4/configure --prefix=/usr/local/gcc-4.3.4-GNU-assembler-Sun-linker --with-as=/usr/local/binutils-2.20/bin/as --with-ld=/usr/ccs/bin/ld --with-gmp=/usr/local --with-mpfr=/usr/local
Thread model: posix
gcc version 4.3.4 (GCC) 
****************************************************
Running from numpy source directory.
F2PY Version 2
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /export/home/drkirkby/sage-4.3.1/local/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /export/home/drkirkby/sage-4.3.1/local/lib
  NOT AVAILABLE

atlas_blas_info:
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/export/home/drkirkby/sage-4.3.1/local/lib']
    language = c
    include_dirs = ['/export/home/drkirkby/sage-4.3.1/local/include']

/export/home/drkirkby/sage-4.3.1/spkg/build/numpy-1.3.0.p2/src/numpy/distutils/command/config.py:361: DeprecationWarning: 
+++++++++++++++++++++++++++++++++++++++++++++++++
Usage of get_output is deprecated: please do not 
use it anymore, and avoid configuration checks 
involving running executable on the target machine.
+++++++++++++++++++++++++++++++++++++++++++++++++

  DeprecationWarning)
customize Sage_FCompiler_1
customize Sage_FCompiler_1
customize Sage_FCompiler_1 using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
gcc _configtest.o -L/export/home/drkirkby/sage-4.3.1/local/lib -lf77blas -lcblas -latlas -o _configtest
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libf77blas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libcblas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libatlas.so: wrong ELF class: ELFCLASS64
ld: fatal: file processing errors. No output written to _configtest
collect2: ld returned 1 exit status
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libf77blas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libcblas.so: wrong ELF class: ELFCLASS64
ld: fatal: file /export/home/drkirkby/sage-4.3.1/local/lib/libatlas.so: wrong ELF class: ELFCLASS64
ld: fatal: file processing errors. No output written to _configtest
collect2: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
Status: 255
Output: 
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/export/home/drkirkby/sage-4.3.1/local/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 2)]
    include_dirs = ['/export/home/drkirkby/sage-4.3.1/local/include']

lapack_opt_info:
lapack_mkl_info:

CC: @jaapspies

Component: porting: Solaris

Author: David Kirkby

Reviewer: John Palmieri

Merged: sage-4.5.3.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/8086

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

Description changed:

--- 
+++ 
@@ -1,7 +1,7 @@
 ## Build environment
 * Sun Ultra 27 3.33 GHz Intel W3580 Xeon. Quad core. 8 threads. 12 GB RAM
 * OpenSolaris 2009.06 snv_111b X86
-* Sage 4.3.1.alpha1 (with a few packages hacked to work on 64-bit)
+* Sage 4.3.1 (with a few packages hacked to work on 64-bit)
 * gcc 4.3.4 configured with Sun linker and GNU assembler from binutils version 2.20.
 * 64-bit build. SAGE64 was set to yes, plus various other tricks to get -m64 into packages. 
 ## The problem

jaapspies commented 14 years ago

comment:2

I do have a working numpy on your machine.

There is a numpy-1.3.0.p3.spkg in trac waiting to be merged.

Here is a pre version of mine: http://boxen.math.washington.edu/home/jsp/ports/numpy-1.3.0.p3.spkg

Works on Open Solaris.

Jaap

jaapspies commented 14 years ago

Author: Jaap Spies

jaapspies commented 14 years ago

comment:3

Added patch file and updated spkg:

http://boxen.math.washington.edu/home/jsp/ports/numpy-1.3.0.p3.spkg

Jaap

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

comment:4

Hi Jaap,

I'm having a problem with this. I got a message about "could not fork". I even rebooted the machine once, thinking something was wrong, but everything else seems to work.

Is this bit of code in the patch right? It looks wrong to me:

echo "#! /usr/bin/env bash" > gcc 
echo "`which gcc` -m64 " | sed 's/$/\$@/' >> fake_gcc

Should the first line not write to the file 'fake_gcc' rather than 'gcc'? If so, since it gets copied to $SAGE_LOCAL/bin/gcc, why not write it directly there?

i.e.

echo "#! /usr/bin/env bash" > $SAGE_LOCAL/bin/gcc
echo "`which gcc` -m64 " | sed 's/$/\$@/' >> $SAGE_LOCAL/bin/gcc

Also, why use 'sed'? If I understand your intentions correctly,

echo "`which gcc` -m64 \$@"

would achieve the same, but without invoking 'sed'.

I would also add that 'which' is not a POSIX command, whereas 'command -v' gives you the same information, but in a standards complient way. This can be important, especially on Solaris 10, as the return code of 'which' is undefined on there.

drkirkby@hawk:~$ echo "`command -v gcc` -m64 \$@" 
/usr/local/gcc-4.4.4/bin/gcc -m64 $@

is more portable.

However, I can't seem to get it to work. I'm not sure if I understand exactly what you are aiming to do here. Why do we need a fake gcc?

Dave

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

Changed author from Jaap Spies to David Kirkby

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

comment:5

Jaap,

I can't get your patch to work, so have reimplemented this. I've completely removed the file fake_gcc from Numpy, and instead create the fake file dynamically with the correct path.

I've also reported the issue to the Numpy bug database.

http://projects.scipy.org/numpy/ticket/1525

which should have been done years ago when the problem was first realised in OS X.

Here's a package waiting for review, which I believe does fix the problem.

http://boxen.math.washington.edu/home/kirkby/patches/numpy-1.3.0.p4.spkg

Dave

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

Description changed:

--- 
+++ 
@@ -94,4 +94,3 @@
 lapack_opt_info:
 lapack_mkl_info:

-

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

comment:8

Great news

I've found a much simpler solution to this, which totally avoids using any fake gcc files. Just compile with

CC="gcc -m64" export CC

and it works! I will attach a simpler patch in 15 minutes. I'm marking as "needs work" for now, as I need to do something else just now.

Dave

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

Attachment: 8086.patch.gz

Mercurial patch which solves the Numpy build issues without any sort of hack. I think this would work on OS X too, but I'm not changing the OS X code now.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

comment:9

Here's the updated package. I believe this is a decent fix (at last).

http://boxen.math.washington.edu/home/kirkby/patches/numpy-1.3.0.p4.spkg

I've reported this upstream

http://projects.scipy.org/numpy/ticket/1525

Dave

jhpalmieri commented 14 years ago

comment:10

The changes look good. The spkg builds on a number of different machines, both 32-bit and 64-bit, and doctests pass.

By the way, with the old spkg, numpy didn't build properly on 64-bit Solaris (either sparc -- t2 -- or x86 -- fulvia), so it's not exclusively an OpenSolaris issue. With the new spkg, it works on fulvia. I haven't gotten t2 (with SAGE64='yes') to build the prerequisite packages yet. I'm working on that, and if there are issues building numpy, I'll revert the positive review.

(Tested on t2 (32-bit), mark2 (32-bit), Mac OS X 10.6 (64-bit), sage.math (both with SAGE64='yes' and with SAGE64 unset), taurus (both with SAGE64='yes' and SAGE64 unset), cicero (32-bit), fulvia (both settings for SAGE64), and iras (64-bit machine? but with SAGE64 unset).)

jhpalmieri commented 14 years ago

Reviewer: John Palmieri

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago

comment:11

Note to the release manager

All changes are committed in the repository of the .spkg file. No library changes need to be made. Just drop in

http://boxen.math.washington.edu/home/kirkby/patches/numpy-1.3.0.p4.spkg

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago

Merged: sage-4.5.3.alpha0

sagemath / sage