Large package size

[mrocklin]

When I install the same conda environment that includes openblas using both conda-forge and defaults I find that the conda-forge environment has a larger binary for openblas

conda create -n openblas-size-cf -c conda-forge openblas
conda create -n openblas-size-defaults -c defaults openblas

$ du -hs openblas-size-cf/
96M openblas-size-cf/
$ du -hs openblas-size-defaults/
60M openblas-size

[isuruf]

Can you check conda-forge/label/gcc7 as well?

[ocefpaf]

Can you check conda-forge/label/gcc7 as well?

I got 104M from the gcc7 label :-(

> conda create --name openblas-size-cf --channel conda-forge/label/gcc7 openblas

Solving environment: done

## Package Plan ##

  environment location: /home/user/miniconda3/envs/openblas-size-cf

  added / updated specs: 
    - openblas

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    openblas-0.3.3             |    h9ac9557_1001        15.8 MB  conda-forge/label/gcc7
    libgfortran-ng-7.3.0       |       hdf63c60_0         1.3 MB  defaults
    libgcc-ng-7.3.0            |       hdf63c60_0         6.1 MB  conda-forge/label/gcc7
    ------------------------------------------------------------
                                           Total:        23.2 MB

The following NEW packages will be INSTALLED:

    libgcc-ng:      7.3.0-hdf63c60_0    conda-forge/label/gcc7
    libgfortran-ng: 7.3.0-hdf63c60_0    defaults              
    openblas:       0.3.3-h9ac9557_1001 conda-forge/label/gcc7

cd ~/miniconda3/envs
du -hs openblas-size-cf/

104M    openblas-size-cf

[ocefpaf]

@isuruf for gdal and its dependencies I used -O2 -Wl,-S to create smaller binaries. Would that be a good fit for openblas?

[ocefpaf]

BTW, here is the size difference for gdal:

> du -hs gdal-size-*
927M    gdal-size-cf
1.5G    gdal-size-defaults

As you can see the -S is quite effective in reducing the size. I know that @jjhelmus has more info on the consequences of using -S and the loss of information/debug symbols but I'm ignorant on the topic, just copied-n-pasted what worked :grimacing:

[jakirkham]

Thoughts on this @mingwandroid?

[mingwandroid]

-s for OpenBLAS should be ok I reckon since most of the performance critical stuff is in assembly AFAIK. See also https://github.com/cython/cython/issues/2102#issuecomment-404321524

[jjhelmus]

-Wl,-S strips debugging symbols which can make stack traces hard to follow. This is not great if you are trying to remotely debug an issue but it is much better than -Wl,-s which strips all debugging information, even function names.

For example a stack trace of segmentation fault when compiled with -Wl,-S would look something like this:

(gdb) bt
#0  0x0000000000400560 in crasher ()
#1  0x0000000000400595 in main ()

with -Wl,-s you loose the function names:

(gdb) bt
#0  0x0000000000400560 in ?? ()
#1  0x0000000000400595 in ?? ()

Another argument to try is -Os which instructs the compiler to optimize for size rather than performance. As @mingwandroid mentioned this may not have much effect on performance of OpenBLAS because the critical portions are written in assembly.

[ocefpaf]

@jjhelmus thanks for explaining it to me again :smile: (I promise to save this comment and come back here instead of asking it for the nth time :grimacing:)

I guess that, in conda-forge, we must decided between the ability to debug vs smaller packages. I believe that we can sacrifice debugging b/c we are not really promising "support." With that said the levels to choose from are -S, -s, and -0s (the last one seems to be OK only for openblas though if I understood @jjhelmus comment correctly).

I believe that -S, no debug symbols but keeping the function names, is OK. I'll make a few tests locally to see how much we can reduce our packages.

PS: I'm adding a topic of discussion for our next meeting about this.

[mingwandroid]

I believe you want to figure out why ADs is so much smaller than CFs first.

[jakirkham]

How does AD ship the static library (if at all)?

Edit: Asking as the static library in c-f alone is ~50MB on Linux. As this shouldn't be in a users production environment, it's a pretty easy win to split this out without compromising on any debug information.

[jjhelmus]

AD ships the static libraries in a separate package, libopenblas-static.

[jcrist]

AD ships the static libraries in a separate package, libopenblas-static.

I created a new issue for this at #69. This is something that has come up in our docker images, and would be useful for us to fix.

[ocefpaf]

Splitting the static lib was done in #70. As pointed in https://github.com/conda-forge/openblas-feedstock/issues/54#issuecomment-435148824 we can reduce the size of the packages further with some compilation flags, but that is a more general discussion and does not belong on this issue. (I'll follow up on that in our meeting later.)