sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.33k stars 453 forks source link

Error building PIL on RHEL Server 5.5 #9864

Closed e14f4152-4982-4ace-8c95-73a0599b109b closed 13 years ago

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago

Minh Nguyen reported this about building a trial "final" 4.5.3 (essentially the same as 4.5.3.rc0) on rosemary.math, which is an Intel(R) Xeon(R) CPU X7460 @ 2.66GHz system running RedHat Enterprise Linux (RHEL) Server 5.5

building '_imaging' extension
gcc -pthread -shared build/temp.linux-x86_64-2.6/_imaging.o build/temp.linux-x86
_64-2.6/decode.o build/temp.linux-x86_64-2.6/encode.o build/temp.linux-x86_64-2.
6/map.o build/temp.linux-x86_64-2.6/display.o build/temp.linux-x86_64-2.6/outlin
e.o build/temp.linux-x86_64-2.6/path.o build/temp.linux-x86_64-2.6/libImaging/Ac
cess.o build/temp.linux-x86_64-2.6/libImaging/Antialias.o build/temp.linux-x86_6
4-2.6/libImaging/Bands.o build/temp.linux-x86_64-2.6/libImaging/BitDecode.o buil
d/temp.linux-x86_64-2.6/libImaging/Blend.o build/temp.linux-x86_64-2.6/libImagin
g/Chops.o build/temp.linux-x86_64-2.6/libImaging/Convert.o build/temp.linux-x86_
64-2.6/libImaging/ConvertYCbCr.o build/temp.linux-x86_64-2.6/libImaging/Copy.o b
uild/temp.linux-x86_64-2.6/libImaging/Crc32.o build/temp.linux-x86_64-2.6/libIma
ging/Crop.o build/temp.linux-x86_64-2.6/libImaging/Dib.o build/temp.linux-x86_64
-2.6/libImaging/Draw.o build/temp.linux-x86_64-2.6/libImaging/Effects.o build/te
mp.linux-x86_64-2.6/libImaging/EpsEncode.o build/temp.linux-x86_64-2.6/libImagin
g/File.o build/temp.linux-x86_64-2.6/libImaging/Fill.o build/temp.linux-x86_64-2
.6/libImaging/Filter.o build/temp.linux-x86_64-2.6/libImaging/FliDecode.o build/
temp.linux-x86_64-2.6/libImaging/Geometry.o build/temp.linux-x86_64-2.6/libImagi
ng/GetBBox.o build/temp.linux-x86_64-2.6/libImaging/GifDecode.o build/temp.linux
-x86_64-2.6/libImaging/GifEncode.o build/temp.linux-x86_64-2.6/libImaging/HexDec
ode.o build/temp.linux-x86_64-2.6/libImaging/Histo.o build/temp.linux-x86_64-2.6
/libImaging/JpegDecode.o build/temp.linux-x86_64-2.6/libImaging/JpegEncode.o bui
ld/temp.linux-x86_64-2.6/libImaging/LzwDecode.o build/temp.linux-x86_64-2.6/libI
maging/Matrix.o build/temp.linux-x86_64-2.6/libImaging/ModeFilter.o build/temp.l
inux-x86_64-2.6/libImaging/MspDecode.o build/temp.linux-x86_64-2.6/libImaging/Ne
gative.o build/temp.linux-x86_64-2.6/libImaging/Offset.o build/temp.linux-x86_64
-2.6/libImaging/Pack.o build/temp.linux-x86_64-2.6/libImaging/PackDecode.o build
/temp.linux-x86_64-2.6/libImaging/Palette.o build/temp.linux-x86_64-2.6/libImagi
ng/Paste.o build/temp.linux-x86_64-2.6/libImaging/Quant.o build/temp.linux-x86_6
4-2.6/libImaging/QuantHash.o build/temp.linux-x86_64-2.6/libImaging/QuantHeap.o 
build/temp.linux-x86_64-2.6/libImaging/PcdDecode.o build/temp.linux-x86_64-2.6/l
ibImaging/PcxDecode.o build/temp.linux-x86_64-2.6/libImaging/PcxEncode.o build/t
emp.linux-x86_64-2.6/libImaging/Point.o build/temp.linux-x86_64-2.6/libImaging/R
ankFilter.o build/temp.linux-x86_64-2.6/libImaging/RawDecode.o build/temp.linux-
x86_64-2.6/libImaging/RawEncode.o build/temp.linux-x86_64-2.6/libImaging/Storage
.o build/temp.linux-x86_64-2.6/libImaging/SunRleDecode.o build/temp.linux-x86_64
-2.6/libImaging/TgaRleDecode.o build/temp.linux-x86_64-2.6/libImaging/Unpack.o b
uild/temp.linux-x86_64-2.6/libImaging/UnpackYCC.o build/temp.linux-x86_64-2.6/li
bImaging/XbmDecode.o build/temp.linux-x86_64-2.6/libImaging/XbmEncode.o build/te
mp.linux-x86_64-2.6/libImaging/ZipDecode.o build/temp.linux-x86_64-2.6/libImagin
g/ZipEncode.o -L/usr/local/lib -L/home/wstein/mvngu/sage-4.5.3/local/lib -L/usr/
lib -L/home/wstein/mvngu/sage-4.5.3/local/lib -ljpeg -lz -lpython2.6 -o build/li
b.linux-x86_64-2.6/_imaging.so
gcc -O3 -g -fPIC -I. -I/home/wstein/mvngu/sage-4.5.3/local/include -I/home/wstei
n/mvngu/sage-4.5.3/local/include  -DHAVE_CONFIG_H -c omList.c
/usr/bin/ld: skipping incompatible /usr/lib/libjpeg.so when searching for -ljpeg
/usr/bin/ld: /usr/local/lib/libpython2.6.a(abstract.o): relocation R_X86_64_32 a
gainst `a local symbol' can not be used when making a shared object; recompile w
ith -fPIC
/usr/local/lib/libpython2.6.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
Error building PIL: 'Error installing PIL'

real    0m10.654s
user    0m8.426s
sys     0m2.141s
sage: An error occurred while installing pil-1.1.6.p2

The full build log is here.

William Stein also reported this on sage-devel.

Related: #7344, #10059.

New package at http://spkg-upload.googlecode.com/files/pil-1.1.6.p3.spkg

CC: @sagetrac-mvngu @TimDumol

Component: build

Keywords: pil spkg

Author: Leif Leonhardy, Mitesh Patel

Reviewer: Jeroen Demeyer

Merged: sage-4.6.1.alpha3

Issue created by migration from https://trac.sagemath.org/ticket/9864

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago

Description changed:

--- 
+++ 
@@ -60,3 +60,5 @@
 sage: An error occurred while installing pil-1.1.6.p2

The full build log is here. + +William Stein also reported this on sage-devel.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:2

I would triple-check that all objects are built with position independent code (PIC), using the -fPIC option. There are three problems with PIC.

Because of this, sometimes people don't build PIC code. The GNU manual implies you should, but is not very clear about it. The Sun linker manual is absolutely clear - shared libraries should be built with PIC.

I've also found that some libraries that should be position independent, do not appear to do so according to the link editor. See #9840

I know of 2 possible issues that can arise.

but there are others.

I'm aware of three libraries which show problems related to this on Solaris or OpenSolaris in 64-bit mode, but two of them might be an issue on all platforms.

If the system has the elfdump command, you might try this:

$ elfdump -d ./build/libecl.so |  grep TEXTREL

At least on Solaris, that should produce no output. If it does, it indicates a problem with the library.

Dave

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:3

libecl.so was just an example - do this on all libraries.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:4

I just noticed something else - this is using the wrong python:

/usr/local/lib/libpython2.6.a: could not read symbols: Bad value

I've seen this problem before - see #9209. In fact, perhaps this should be marked as a duplicate of #9209.

Dave

aghitza commented 14 years ago
comment:5

Here is another data point: I just built 4.5.3 from scratch on a machine running RHEL Server 5.5, and had no problems whatsoever; also longtests passed. Here are the specs:

 [aghitza@soleil sage-4.5.3]$ uname -a
 Linux soleil.ms.unimelb.edu.au 2.6.18-194.11.3.el5 #1 SMP Mon Aug 23 15:51:38 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

 [aghitza@soleil sage-4.5.3]$ cat /etc/issue
 Red Hat Enterprise Linux Server release 5.5 (Tikanga)
 Kernel \r on an \m

 [aghitza@soleil sage-4.5.3]$ gcc -v
 Using built-in specs.
 Target: x86_64-redhat-linux
 Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
 --infodir=/usr/share/info --enable-shared --enable-threads=posix 
 --enable-checking=release --with-system-zlib --enable-__cxa_atexit 
 --disable-libunwind-exceptions --enable-libgcj-multifile 
 --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk 
 --disable-dssi --enable-plugin 
 --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre 
 --with-cpu=generic --host=x86_64-redhat-linux
 Thread model: posix
 gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)

 [aghitza@soleil ~]$ which python
 /usr/bin/python

 [aghitza@soleil ~]$ python
 Python 2.4.3 (#1, Jun 11 2009, 14:09:37) 
 [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> 

Note in particular that the machine does have Python installed independently of Sage, but that did not seem to cause problems.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:6

Replying to @aghitza:

Here is another data point: I just built 4.5.3 from scratch on a machine running RHEL Server 5.5, and had no problems whatsoever; also longtests passed. Here are the specs:

 [aghitza@soleil sage-4.5.3]$ uname -a
 Linux soleil.ms.unimelb.edu.au 2.6.18-194.11.3.el5 #1 SMP Mon Aug 23 15:51:38 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

 [aghitza@soleil sage-4.5.3]$ cat /etc/issue
 Red Hat Enterprise Linux Server release 5.5 (Tikanga)
 Kernel \r on an \m

 [aghitza@soleil sage-4.5.3]$ gcc -v
 Using built-in specs.
 Target: x86_64-redhat-linux
 Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
 --infodir=/usr/share/info --enable-shared --enable-threads=posix 
 --enable-checking=release --with-system-zlib --enable-__cxa_atexit 
 --disable-libunwind-exceptions --enable-libgcj-multifile 
 --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk 
 --disable-dssi --enable-plugin 
 --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre 
 --with-cpu=generic --host=x86_64-redhat-linux
 Thread model: posix
 gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)

 [aghitza@soleil ~]$ which python
 /usr/bin/python

 [aghitza@soleil ~]$ python
 Python 2.4.3 (#1, Jun 11 2009, 14:09:37) 
 [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> 

Note in particular that the machine does have Python installed independently of Sage, but that did not seem to cause problems.

Note however that

all had the directory /usr/local/lib/libpython2.6.a involved.

I suspect there might be some packages which look under /usr/local/lib, before whatever is specific in Sage.

Dave

aghitza commented 14 years ago
comment:7

And I can confirm that /usr/local/lib doesn't have any Python libraries in it on the machine I tested.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:8

Another, somewhat related issue is #9551. In that case, Sage tried to write to /usr/lib/python2.4/site-packages/ on t2.math.washington.edu despite I'd tried to build Sage under my home directory.

I think any assumption that having the first python in the path being $SAGE_LOCAL/bin/python will mean Sage will never try to do anything with python outside $SAGE_ROOT is rather flawed. That assumption would seem to be valid 99% of the time, but there are the odd exceptions. I wish I could pin down what causes these rare exceptions.

Another assumption that is sometimes flawed, is that having a library in $SAGE_LOCAL/lib will mean the same library anywhere else will always be ignored and not present any conflicts with the library in Sage. The fact the iconv library can't be installed reliably on Fedora and OpenSuse (see #8567) is one example of that.

One possible hack to solve this PIL issue might be to change any occurrence of /usr/local in any PIL file to /some/stupid/junk/path. If that does not work, we could consider changing any occurances of /usr/local in the Python package to /some/stupid/junk/path.

Dave

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:9

Should this be a blocker for releasing 4.5.3? It's a pretty serious issue when copies of python in /usr/local are sometimes screwing up building Sage on two supported platforms (Solaris 10 and Redhat Linux).

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:11

I've changed the milestone to 4.6, as I'll release 4.5.3 later today.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:12
... -L/usr/local/lib -L/home/wstein/mvngu/sage-4.5.3/local/lib -L/usr/lib ...

is obviously wrong.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:13

Replying to @nexttime:

... -L/usr/local/lib -L/home/wstein/mvngu/sage-4.5.3/local/lib -L/usr/lib ...

is obviously wrong.

Yes.

There's some things in patches/setup.py which look odd to me, though since I don't know what SAGE_BINARY_BUILD is, I can't say for sure.

       #
        # add standard directories

        if not SAGE_BINARY_BUILD:
            add_directory(library_dirs, "/usr/local/lib")
            add_directory(include_dirs, "/usr/local/include")

            add_directory(library_dirs, "/usr/lib")
            add_directory(include_dirs, "/usr/include")

        #

I think they should be removed from the pil package.

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:14

From the Installation Guide:

SAGE_BINARY_BUILD - used by the pil package. If set to “yes”, then force Sage to use the versions of libjpeg, libtiff and libpng from $SAGE_ROOT/local/lib. Otherwise, allow the use of the system’s versions of these libraries.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:15

I'm pretty sure we can simply delete the first occurrence of

            add_directory(library_dirs, "/usr/local/lib")
            # FIXME: check /opt/stuff directories here?

Who knows if at all the indentation is correct? The comment below apparently refers to the (closed) elif-Darwin branch. /usr/local/lib is added again later (in the code, in the snippet Dave gave above), after SAGE_LOCAL/lib and /usr/lib, but since it's already in the path, the Sage directory ends up in the middle.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:16

Btw, s/2008/2009/ in the changelog for p1 and p2. ;-)

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:17

Replying to @sagetrac-drkirkby:

  • Libraries built without PIC run a bit more slowly.

I guess you meant "with PIC" (-fpic or -fPIC). (In general that depends on the architecture, and how the code is used.)

I'm aware of three libraries which show problems related to this on Solaris or OpenSolaris in 64-bit mode, but two of them might be an issue on all platforms.

  • PolyBori

Did you report that, too? (Ticket?)

If the system has the elfdump command, you might try this:

$ elfdump -d ./build/libecl.so |  grep TEXTREL

There's also nm, binutils' objdump, and readelf.

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:18

Replying to @nexttime:

Replying to @sagetrac-drkirkby:

  • Libraries built without PIC run a bit more slowly.

I guess you meant "with PIC" (-fpic or -fPIC). (In general that depends on the architecture, and how the code is used.)

Yes, I did.

I'm aware of three libraries which show problems related to this on Solaris or OpenSolaris in 64-bit mode, but two of them might be an issue on all platforms.

  • PolyBori

Did you report that, too? (Ticket?)

Yes, I created a ticket yesterday. Alexander Dreyer is already working on it - see #9872

Dave

e14f4152-4982-4ace-8c95-73a0599b109b commented 14 years ago
comment:19

Georg Weber has suggested on sage-release that we set SAGE_BINARY_BUILD='yes' by default. Thoughts?

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 14 years ago
comment:20

Replying to @qed777:

Georg Weber has suggested on sage-release that we set SAGE_BINARY_BUILD='yes' by default. Thoughts?

Defiantly not.

The problem here is PIL looking for things in /usr/local. This is happening during the build process. I don't see how this is going to help this problem. The reason for looking for things in /usr/local needs to be resolved. Adding options like

$ export SAGE_INCLUDE_pil_greatest-ever-library=/usr/local/greatest-ever-library

would be OK by me. Forcing SAGE_BINARY_BUILD=yes is not the answer IMHO.

Dave

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 14 years ago
comment:21

Replying to @qed777:

Georg Weber has suggested on sage-release that we set SAGE_BINARY_BUILD='yes' by default. Thoughts?

As far as I understand this, one has to (manually) copy the build system's respective shared libraries into $SAGE_LOCAL/lib (before building PIL) anyhow to make this effective. (Probably some headers into $SAGE_LOCAL/include, too.) So enabling SAGE_BINARY_BUILD by default (where?) doesn't make much sense to me.

9343d2e0-59ba-4406-bd4f-c78e4cf1230e commented 13 years ago
comment:22

Replying to @nexttime:

Replying to @qed777:

Georg Weber has suggested on sage-release that we set SAGE_BINARY_BUILD='yes' by default. Thoughts?

As far as I understand this, one has to (manually) copy the build system's respective shared libraries into $SAGE_LOCAL/lib (before building PIL) anyhow to make this effective. (Probably some headers into $SAGE_LOCAL/include, too.) So enabling SAGE_BINARY_BUILD by default (where?) doesn't make much sense to me.

Maybe, as with SAGE_PIL_NOTK, sage should set SAGE_BINARY_BUILD='yes' if building PIL fails.

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago
comment:23

Are there any comments about the proposal by Niles?

6bf4345a-8fdd-4a0e-b9a5-03107e609570 commented 13 years ago
comment:24

Niles' suggestion seems optimal, however I do not have enough time at the moment (this week and the next) to think hard about it.

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago

Description changed:

--- 
+++ 
@@ -62,3 +62,5 @@
 The full build log is [here](http://wiki.sagemath.org/devel/BuildFarm/sage-4.5.3?action=AttachFile&do=get&target=rosemary.math-build.log.bz2).

 William Stein also reported this on [sage-devel](http://groups.google.com/group/sage-devel/browse_thread/thread/3487f96fda36b6f0/ec8cd2dc85f64f9a#ec8cd2dc85f64f9a).
+
+Related: #7344, #10059.
e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago
comment:25

For what it's worth, in Tachyon's spkg-install, there's an example of retrying an spkg installation after an initial failure:

if [ $UNAME = "Linux" ]; then
    GCCVERSION=`gcc -dumpversion`
    case $GCCVERSION in
      4.2.4* | 4.3*)
        export GCCFIX="-fno-crossjumping -fno-reorder-blocks";;
      *);;
    esac
    make linux-thr
    if [ $? -ne 0 ]; then
        echo "Maybe your system is 64-bit; trying again."
        if [ `uname -m` = "ia64" ]; then
          echo "ia64"
          make linux-ia64-thr
        else
          echo "64-bit arch"
          make linux-64-thr
        fi
    fi
    finished
fi

(If it's possible to avoid doing this with Tachyon, we can open a new ticket for it.)

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago
comment:26

I'm changing the milestone to 4.6.1. Of course, the next release manager is welcome to change the priority.

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago
comment:27

I've added a RHEL 5.5 builder on Jon Hanke's machine rosemary. The zeroth build failed on PIL.

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago
comment:28

Replying to @nexttime:

I'm pretty sure we can simply delete the first occurrence of

            add_directory(library_dirs, "/usr/local/lib")
            # FIXME: check /opt/stuff directories here?

If I comment out this add_directory line, PIL builds successfully on rosemary. The doctests that didn't pass previously -- in plot/plot3d/base.pyx -- now pass.

Does anyone see a reason to keep this line for any platform? Has it not caused problems on other systems, because the systems haven't had a /usr/local/lib/libpython2.6.a?

Who knows if at all the indentation is correct? The comment below apparently refers to the (closed) elif-Darwin branch. /usr/local/lib is added again later (in the code, in the snippet Dave gave above), after SAGE_LOCAL/lib and /usr/lib, but since it's already in the path, the Sage directory ends up in the middle.

Indeed, with the original line, I get

['/usr/local/lib', '/home/buildbot/build/sage/rosemary-1/rosemary_full/build/sage-4.6.1.alpha2/local/lib', '/usr/lib']  # library_dirs
['libImaging', '/home/buildbot/build/sage/rosemary-1/rosemary_full/build/sage-4.6.1.alpha2/local/include', '/usr/local/include', '/usr/include']  # include_dirs

but without it, I get


['/home/buildbot/build/sage/rosemary-1/rosemary_full/build/sage-4.6.1.alpha2/local/lib', '/usr/local/lib', '/usr/lib']  # library_dirs
['libImaging', '/home/buildbot/build/sage/rosemary-1/rosemary_full/build/sage-4.6.1.alpha2/local/include', '/usr/local/include', '/usr/include']  # include_dirs
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:29

Replying to @qed777:

Replying to @nexttime:

I'm pretty sure we can simply delete the first occurrence of

            add_directory(library_dirs, "/usr/local/lib")
            # FIXME: check /opt/stuff directories here?

If I comment out this add_directory line, PIL builds successfully on rosemary. The doctests that didn't pass previously -- in plot/plot3d/base.pyx -- now pass.

Does anyone see a reason to keep this line for any platform? Has it not caused problems on other systems, because the systems haven't had a /usr/local/lib/libpython2.6.a?

Yes please, can you create a patch such that we can close this ticket after month of deep thinking?

If new failures occur, we can open another ticket (I'm hopefully not cc'ed on ;-) ).


I'm wondering if I should open a ticket for changing SAGE_PIL_NOTK to SAGE_PIL_NO_TK...

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago

Fix library_dirs order. Use patch. SPKG patch.

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago

Author: Leif Leonhardy, Mitesh Patel

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago
comment:30

Attachment: trac_9864-PIL_library_path.patch.gz

I've added a spkg link to the description and attached an SPKG patch. The main changes:

The package installs successfully and the long doctests pass for me on bsd (OSX 10.6-32), eno (Fedora 13-64), hawk (OpenSolaris 06.2009-32), redhawk (Ubuntu 10-64), rosemary (RHEL 5.5-64), and sage.math (Ubuntu 8-64).

e14f4152-4982-4ace-8c95-73a0599b109b commented 13 years ago

Description changed:

--- 
+++ 
@@ -64,3 +64,7 @@
 William Stein also reported this on [sage-devel](http://groups.google.com/group/sage-devel/browse_thread/thread/3487f96fda36b6f0/ec8cd2dc85f64f9a#ec8cd2dc85f64f9a).

 Related: #7344, #10059.
+
+New package at
+
+ http://sage.math.washington.edu/home/mpatel/trac/9864/pil-1.1.6.p3.spkg
jdemeyer commented 13 years ago

Work Issues: make patches #9419-compliant

jdemeyer commented 13 years ago
comment:31

Thanks for the work on this ticket, I will do some more testing

About using patch: for sake of consistency, please follow the following instructions from [#9419 comment:5]:

1) create a directory pil-1.1.6.p2
2) cd pil-1.1.6.p2
3) put upstream source inpil-1.1.6.p2/src
For every ISSUE which needs to be patched, do the following:
    4) cp -pr src src.patched
    5) edit src.patched to fix ISSUE.
    6) diff -ur src src.patched >patches/ISSUE.patch
    7) rm -r src.patched

To apply the patches in spkg-install:

1) cd src
2) patch -p1 <../patches/ISSUE.patch
83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:32

Replying to @jdemeyer:

About using patch: for sake of consistency, please follow the following instructions from [#9419 comment:5]...

Well, #9419 is not even in "needs review" state. I'm not sure if your method is the only desirable one; it at least lacks how we should document patches.

IMHO Mitesh's way is as appropiate here.

jdemeyer commented 13 years ago
comment:33

New package works fine on 64-bit Gentoo Linux, 32-bit Gentoo Linux, 32-bit OS X PPC. So from a functionality point of view, this package gets a positive review from me. I don't think I'm able to review the actual code though.

fbdfc054-b23c-4e3a-8828-e5f2a5392232 commented 13 years ago
comment:34

I'm having issues with PIL 1.1.6p2 (the release version) not finding its JPEG libraries. These are located in /usr/lib64 on CentOS 5.5 / OpenSuse, however the default library path only considers /usr/lib. Adding /usr/lib64 as a library path fixes the problem in my stand-alone PIL build. I'm not quite sure how to patch the setup.py in SAGE to add this path by default.

The following part of the log shows the problem (or the lack of a jpeg_decoder function in PIL.Image.core)


PIL 1.1.6 BUILD SUMMARY


version 1.1.6 platform linux2 2.6.4 (r264:75706, Oct 4 2010, 14:47:23) [GCC 4.3.4]


TKINTER support not available JPEG support not available --- ZLIB (PNG/ZIP) support ok --- FREETYPE2 support ok


83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:35

Replying to @sagetrac-Koen:

I'm having issues with PIL 1.1.6p2 (the release version) not finding its JPEG libraries. These are located in /usr/lib64 on CentOS 5.5 / OpenSuse, however the default library path only considers /usr/lib. Adding /usr/lib64 as a library path fixes the problem in my stand-alone PIL build. I'm not quite sure how to patch the setup.py in SAGE to add this path by default.

Well, we should just check if /usr/lib64 (and /usr/local/lib64) exist, and if so, add these instead of /usr/lib etc., unless realpath("/usr/lib64") == realpath("/usr/lib"). (We may also check we're really on a 64-bit system, too, though the presence of /usr/lib64 should normally indicate that.)

Note that on other (64-bit) Linuces (e.g. Debian), /usr/lib64 is a symbolic link to /usr/lib, while on Fedora etc. /usr/lib is a synonym of /usr/lib32 (like on 32-bit systems).

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:36

Just noticed our src/ isn't vanilla upstream:

~/Sage/spkgs$ diff -ur upstream/Imaging-1.1.6/ pil-1.1.6.p3/src/
Only in pil-1.1.6.p3/src/: ff
diff -ur upstream/Imaging-1.1.6/setup.py pil-1.1.6.p3/src/setup.py
--- upstream/Imaging-1.1.6/setup.py 2006-12-03 12:37:29.000000000 +0100
+++ pil-1.1.6.p3/src/setup.py   2010-11-26 10:39:04.000000000 +0100
@@ -90,6 +90,9 @@
 except ImportError:
     _tkinter = None

+# Force None, so don't build tk -- this helps on some platforms.
+_tkinter = None
+
 def add_directory(path, dir, where=None):
     if dir and os.path.isdir(dir) and dir not in path:
         if where is None:

(p3's ff is some left-over diff file.)

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 13 years ago
comment:37

Replying to @nexttime:

Replying to @sagetrac-Koen:

I'm having issues with PIL 1.1.6p2 (the release version) not finding its JPEG libraries. These are located in /usr/lib64 on CentOS 5.5 / OpenSuse, however the default library path only considers /usr/lib. Adding /usr/lib64 as a library path fixes the problem in my stand-alone PIL build. I'm not quite sure how to patch the setup.py in SAGE to add this path by default.

Well, we should just check if /usr/lib64 (and /usr/local/lib64) exist, and if so, add these instead of /usr/lib etc., unless realpath("/usr/lib64") == realpath("/usr/lib"). (We may also check we're really on a 64-bit system, too, though the presence of /usr/lib64 should normally indicate that.

I'm having issues with PIL 1.1.6p2 (the release version) not finding its JPEG libraries. These are located in /usr/lib64 on CentOS 5.5 / OpenSuse, however the default library path only considers /usr/lib. Adding /usr/lib64 as a library path fixes the problem in my stand-alone PIL build. I'm not quite sure how to patch the setup.py in SAGE to add this path by default.

Is it essential to have the JPEG libraries? If not, then they should be excluded when SAGE_FAT_BINARY=yes, otherwise we risk breakages if people install a Sage binary on a system without these libraries.

I'm puzzled why it should be necessary to so add /usr/lib64. Has someone mis-configured the system?

On every system that I know, that is able to build both 32-bit and 64-bit libraries, the runtime lnker will always search for the 64-bit ones when building 64-bit code. On Solaris the 64-bit libraries are in /usr/lib/sparcv9 or /usr/lib/amd64, depending on the CPU type. But the run time linker knows that, and will search for them:One never needs to put /usr/lib/sparcv9 or /usr/lib/amd64 in any sort of PATH (LD_LIBRARY_PATH etc).

-bash-3.00$ crle    

Default configuration file (/var/ld/ld.config) not found
  Default Library Path (ELF):   /lib:/usr/lib  (system default)
  Trusted Directories (ELF):    /lib/secure:/usr/lib/secure  (system default)
-bash-3.00$ crle -64

Default configuration file (/var/ld/64/ld.config) not found
  Default Library Path (ELF):   /lib/64:/usr/lib/64  (system default)
  Trusted Directories (ELF):    /lib/secure/64:/usr/lib/secure/64  (system default)
-bash-3.00$ 

It's a complely different matter in directories like /usr/local/lib/sparcv9 since the run time linker does not to search there.

I know this is true on HP-UX too.

On AIX, the libraries are in an archives containing both 32-bit and 64-bit libraries in the same archive. But again, the run-time linker knows how to find them.

Clearly if the system is mis-configured, we should not work around the problem.

I think we should ascertain if someone has mis-configured the system before proceeding to patch Sage

Dave

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:38

Replying to @sagetrac-drkirkby:

I'm puzzled why it should be necessary to so add /usr/lib64. Has someone mis-configured the system? ;

On every system that I know, that is able to build both 32-bit and 64-bit libraries, the runtime lnker will always search for the 64-bit ones when building 64-bit code. [...]

Perhaps a look at setup.py answers some of your questions... ;-)

(And no, the Linux system having 64-bit libraries [only] in /usr/lib64 is not "misconfigured", see above. We had to add a similar patch to PARI's graphics detection for the same reason btw.)

bac7d3ea-3f1b-4826-8464-f0b53d5e12d2 commented 13 years ago
comment:39

Replying to @nexttime:

(And no, the Linux system having 64-bit libraries [only] in /usr/lib64 is not "misconfigured",

I was not saying having the 64-bit libraries only in /usr/lib64 was mis-configured - I would expect that. What I question is why the 64-bit libraries are not found. I would have thought the operating system would have been configured so linking them would find them, without one needing to specify a path. Clearly if the libraries are in a non-standard place, then I would expect it, but not when they are in the standard

fbdfc054-b23c-4e3a-8828-e5f2a5392232 commented 13 years ago
comment:40

It's because the PIL setup.py is trying to be smart: it creates its own list of search paths for includes & libs, and when it cannot find jpeglib.h in the includes or libjpeg.so in the libraries search paths, it will disable its own JPEG support. However, it does not check a common place for 64-bit Linuxes (which are not Debian-based), so right now it is too eager in disabling JPEG support. Note that SciPy uses PIL to read JPEG images, so not having it working is quite annoying.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:41

Replying to @sagetrac-Koen:

It's because the PIL setup.py is trying to be smart: it creates its own list of search paths for includes & libs, and when it cannot find jpeglib.h in the includes or libjpeg.so in the libraries search paths, it will disable its own JPEG support. However, it does not check a common place for 64-bit Linuxes (which are not Debian-based), so right now it is too eager in disabling JPEG support. Note that SciPy uses PIL to read JPEG images, so not having it working is quite annoying.

Yep. If we don't fix that (here), you could still create symbolic links in SAGE_LOCAL/{include,lib}; then PIL should find them as well.

(If we build binary distributions (with e.g. JPEG support enabled), we have to manually copy the system's respective libraries and headers to SAGE_LOCAL/... anyway.)

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:42

Replying to @nexttime:

If we don't fix that (here), you could still create symbolic links in SAGE_LOCAL/{include,lib}; then PIL should find them as well.

P.S.: PIL 1.1.7 adds a find_include_file() function, but still won't find a library in /usr/lib64, so I think we should fix that, not sure if on this ticket though.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:43

I've uploaded a slightly modified p3 spkg:

http://spkg-upload.googlecode.com/files/pil-1.1.6.p3.spkg

md5sum: 5453cd4356f1cfa9e1c097b0f7dcc609 pil-1.1.6.p3.spkg

(src/ is now vanilla -> patch rebased; some clean-up, additions to SPKG.txt.)

Does not fix the .../lib64 issue; this should IMHO go onto another ticket.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago

SPKG patch, based on Mitesh's p3. Upstream now vanilla, some clean-up.

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago

Changed work issues from make patches #9419-compliant to none

83660e46-0051-498b-a8c1-f7a7bd232b5a commented 13 years ago
comment:44

Attachment: trac_9864-remove_duplicate_usr_local_lib_dir-v2-spkg.patch.gz