search

rasterio / rasterio-wheels

MIT License
11 stars 16 forks source link

libgdal is too big #26

Closed sbrunner closed 5 years ago

sbrunner commented 5 years ago

Actual status

$ ls -lh libgdal-a3e05b74.so.20.5.1
-rwxr-xr-x 1 root root 97M Jul  2 13:24 libgdal-a3e05b74.so.20.5.1
$ file libgdal-a3e05b74.so.20.5.1
libgdal-a3e05b74.so.20.5.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=8a8032a095e4f6d37c963305af0c7ab2963a079c, not stripped

Expected status

Operating system

For example: Docker image: ubuntu:18.04

Rasterio version and provenance

1.0.24 manylinux1 wheel installed from PyPI using pip 9.0.1. Filename: rasterio-1.0.24-cp37-cp37m-manylinux1_x86_64.whl

sgillies commented 5 years ago

Good suggestion. I've mainly been considering the size of the .whl file, which isn't so much bigger than other big scientific python packages, and following the lead of the multibuild project. I see in https://github.com/matthew-brett/multibuild/commit/453308deaf6e045ef437d413f8791ec297fcc60d that multibuild now strips binaries, so I'll look into doing this in the same way.

sgillies commented 5 years ago

I've updated multibuild and now, in the linux wheels, all libs are stripped except gdal:

(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libz-a147dcb0.so.1.2.3 
libz-a147dcb0.so.1.2.3: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libwebp-8ccd29fd.so.7.0.2 
libwebp-8ccd29fd.so.7.0.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d1967607555e25e2e12f93042c76c196e018cf56, stripped
(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libgeos_c-595de9d4.so.1.10.2 
libgeos_c-595de9d4.so.1.10.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=5bdf817a7f68b8466a8e7e5fdf96e014103048e3, stripped
(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libcurl-835381e7.so.4.4.0 
libcurl-835381e7.so.4.4.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=55cba01443fde8c37fc3194c2f798e389ea67512, stripped
(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libp
libpng16-898afbbd.so.16.35.0  libproj-bd876d1a.so.12.0.0    
(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libproj-bd876d1a.so.12.0.0 
libproj-bd876d1a.so.12.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a4980c02c17b6661396cbba0dc283c9a1802fc7b, stripped
(frs-wheels) sean@PF-17LYEH:~/projects/rasterio-wheels/rasterio/.libs$ file libgdal-fe382901.so.20.5.2 
libgdal-fe382901.so.20.5.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=1ed565a2e1735031831fbfbf78fff5fc1cb0c934, not stripped

I think libtool may be messing with me, and I've found some hints for getting around it.

sgillies commented 5 years ago

Found the magic words to make a smaller GDAL lib: --with-hide-internal-symbols. On top of this, I'm running strip --strip-unneeded on the shared library before running auditwheel on it.

sgillies commented 5 years ago

Down to 55 MB for libgdal. I wish I knew what the Ubuntu packagers were doing to strip it down further.

  4116192  2019-08-06 18:09   rasterio/.libs/libhdf5-c22fb506.so.101.1.0
   376744  2019-08-06 18:09   rasterio/.libs/libopenjp2-8f6da918.so.2.3.0
    37712  2019-08-06 18:09   rasterio/.libs/libaec-2147abcd.so.0.0.4
   182032  2019-08-06 18:09   rasterio/.libs/libhdf5_hl-d68fbc5b.so.100.1.0
    13240  2019-08-06 18:09   rasterio/.libs/libsz-1c7dd0cf.so.2.0.1
    60792  2019-08-06 18:09   rasterio/.libs/libjson-c-5f02f62c.so.2.0.2
   580440  2019-08-06 18:09   rasterio/.libs/libproj-bd876d1a.so.12.0.0
   224232  2019-08-06 18:09   rasterio/.libs/libexpat-09c47d4c.so.1.6.8
 55410984  2019-08-06 18:09   rasterio/.libs/libgdal-8b68ca59.so.20.5.2
  4883424  2019-08-06 18:09   rasterio/.libs/libgeos-3-cd838e67.6.2.so
  1876336  2019-08-06 18:09   rasterio/.libs/libnetcdf-e64b17ac.so.11.0.4
   431816  2019-08-06 18:09   rasterio/.libs/libgeos_c-595de9d4.so.1.10.2
   279824  2019-08-06 18:09   rasterio/.libs/libpng16-898afbbd.so.16.35.0
  3501408  2019-08-06 18:09   rasterio/.libs/libcurl-835381e7.so.4.4.0
   344704  2019-08-06 18:09   rasterio/.libs/libjpeg-3fe7dfc0.so.9.3.0
   776912  2019-08-06 18:09   rasterio/.libs/libwebp-8ccd29fd.so.7.0.2
   203512  2019-08-06 18:09   rasterio/.libs/libnghttp2-11cb20b8.so.14.17.1
  1267504  2019-08-06 18:09   rasterio/.libs/libsqlite3-fdd57a2d.so.0.8.6
    87848  2019-08-06 18:09   rasterio/.libs/libz-a147dcb0.so.1.2.3
sbrunner commented 5 years ago

Thanks :-)