Closed erotavlas closed 4 years ago
Does PyInstaller look at the modules and guess the dependencies based on the imports? PyTorch isn't a dependency of spaCy itself, but spaCy does ship a few functions that import torch
within the function body, so you can use them with PyTorch if you have it installed. There's also a try
/except
block that imports torch
if it's installed for extra functionality. So maybe PyInstaller thinks that spaCy requires torch
, even though it doesn't?
Yes if its listed as an import somewhere, then pyinstaller will automatically bundle it into the package (it scans a script and finds all its imports, and imports within those etc.).
So I guess that import torch
statement explains why its being included.
I found there is an option --exclude-module
that I can try to ignore torch.
What about the mkl package? Were there any changes regarding this?
I rebuilt the package and the exclude worked - without torch the package size reduces to around 500MB which is still large. I think the remaining issue is the presence of the 'mkl' package. Any ideas about that?
EDIT: I created a new environment but instead of installing spacy through anaconda I used pip. First I installed numpy via pip, then I installed spacy version 2.1.8 using pip. After running puinstaller my package size was down to just over 70 MB and the mkl packages are no longer there.
The anaconda version of spacy seems to be including the mkl instead of openblas.
Glad it worked – thanks for updating!
And yes, numpy
installed via Anaconda seems to default to MKL instead of OpenBLAS. I think you can configure that during installation. Or alternatively, installing from pip works as well.
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
I noticed when creating a pyinstaller executable from my spacy python scripts, that the size of the resulting package is very large mostly due to the inclusion of something called 'mkl' (whose 'dll's take up over 500MB of the package. When I created my original pyinstaller package using spacy version 2.1.4 those were not present. Although there may be additional packages which are contributing to the file size increase.
EDIT: I found that a folder called 'torch' takes up the most space (over 50 %) and is being included although I don't see any package relating to pytorch installed in the environment
This was definitely not present before when I was first using pyinstaller to create spacy executables sometime between spacy versions 2.0.16 and 2.1.4
Here is a breakdown of what is taking up the most space in the pyintaller output folder
I went from something around 40MB to 1.8GB which is pretty significant.
I found another reference to this in the following regarding mkl in particular https://github.com/conda-forge/numpy-feedstock/issues/84
I'm using Anaconda environment with spacy installed using
conda install spacy==2.1.8
below are the packages installed to my environment
via
conda install spacy==2.1.8
My pyinstaller command for 2.1.8 is
pyinstaller ner.py --hidden-import cymem.cymem --hidden-import thinc.linalg --hidden-import murmurhash.mrmr --hidden-import cytoolz.utils --hidden-import cytoolz._signatures --hidden-import spacy.strings --hidden-import spacy.morphology --hidden-import spacy.lexeme --hidden-import spacy.tokens --hidden-import spacy.gold --hidden-import spacy.tokens.underscore --hidden-import spacy.parts_of_speech --hidden-import dill --hidden-import spacy.tokens.printers --hidden-import spacy.tokens._retokenize --hidden-import spacy.syntax --hidden-import spacy.syntax.stateclass --hidden-import spacy.syntax.transition_system --hidden-import spacy.syntax.nonproj --hidden-import spacy.syntax.nn_parser --hidden-import spacy.syntax.arc_eager --hidden-import thinc.extra.search --hidden-import spacy.syntax._beam_utils --hidden-import spacy.syntax.ner --hidden-import thinc.neural._classes.difference --hidden-import spacy.vocab --hidden-import spacy.lemmatizer --hidden-import spacy._ml --hidden-import spacy.lang.en --hidden-import srsly.msgpack.util --hidden-import preshed.maps --hidden-import thinc.neural._aligned_alloc --hidden-import blis --hidden-import blis.py --hidden-import spacy.matcher._schemas --hidden-import spacy._align --hidden-import spacy.syntax._parser_model --hidden-import spacy.kb
EDIT: I am unable to recreate my original package (size of 42MB) I think because every time I try to install spacy (earlier version such as 2.1.4) it still installs the latest of most other packages,, and somehow torch gets included somehow as well.