Closed ctrado18 closed 3 years ago
ctrado18
commented
7 years ago Do you need anymore information? The infos here are a bit complex to understand and how to process those:
https://pythonhosted.org/PyInstaller/when-things-go-wrong.html
So I think the problems are easy to solve. But there are no easy information how to include those missing modules.
ghost
commented
7 years ago Does the exe run? And if not what is the traceback?
ctrado18
commented
7 years ago Thanks. No! Running exe gives alway (including those warning messages above)
Traceback (most recent call last):
File "test.py", line 6, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\__init__.py", line 4, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\util.py", line 10, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\compat.py", line 5, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\ftfy\__init__.py", line 12, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\ftfy\fixes.py", line 12, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\ftfy\chardata.py", line 145, in <module>
File "site-packages\pkg_resources\__init__.py", line 1233, in resource_string
File "site-packages\pkg_resources\__init__.py", line 1475, in get_resource_string
File "site-packages\pkg_resources\__init__.py", line 1548, in _get
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 479, in get_data
with open(path, 'rb') as fp:
IOError: [Errno 2] No such file or directory: u'C:\\Users\\Namec\\Desktop\\RASA_N~1\\dist\\test\\ftfy\\char_classes.dat'
[11172] Failed to execute script testFurthermore: I have created now an virtualenv plus the latest dev pyinstaller. It also gives with
pyinstaller --onefile test.py
the early pywin32 import warning although I installed it!!
ghost
commented
7 years ago Use onedir to debug and place char_clases.dat in the same folder as the exe.
ctrado18
commented
7 years ago Thanks. I have created the folder ftfy like mentioned in the traceback not only copied the file char.dat. Now since this folder exists with this data it works and now comes "easy" error:
Traceback (most recent call last):
File "test.py", line 6, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\__init__.py", line 5, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\deprecated.py", line 8, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\cli\__init__.py", line 5, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\cli\train.py", line 8, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\scorer.py", line 4, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 687, in load_module
module = imp.load_module(fullname, fp, filename, ext_tuple)
File ".env/lib/python2.7/site-packages/cymem/cymem.pxd", line 1, in init spacy.gold (spacy/gold.cpp:23496)
ImportError: No module named cymem.cymem
[10564] Failed to execute script test
I think those are the common Import Errors. But I really dont understand how to proceed! The doc site from here are a bit to advanced with no examples where and in what files I have to add the proper paths to those modules etc...
ctrado18
commented
7 years ago If I do:
pyinstaller --onedir --hidden-import=cymen.cymen test.py
The same error occurs that cymen.cymen is not found...
I looked into scorer.py where:
from .gold import tags_to_entities
Then in gold.pxd
from cymem.cymem cimport Pool
So should I import cymen.cymen or Pool?
Is the process different because it hase to do with .pxd files and cython?
ctrado18
commented
7 years ago Now It works with only one hidden import. cymem.cymem works. No it wants module map:
Traceback (most recent call last):
File "test.py", line 6, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\__init__.py", line 5, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\deprecated.py", line 8, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\cli\__init__.py", line 5, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\cli\train.py", line 8, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 396, in load_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\scorer.py", line 4, in <module>
File "c:\python\env\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 687, in load_module
module = imp.load_module(fullname, fp, filename, ext_tuple)
File ".env/lib/python2.7/site-packages/preshed/maps.pxd", line 36, in init spacy.gold (spacy/gold.cpp:23499)
ImportError: No module named maps
[10092] Failed to execute script test
So I did
pyinstaller --onedir --hidden-import=cymen.cymen --hidden-import=maps test.py
But still same Traceback!
Where is this hidden import, because in
File ".env/lib/python2.7/site-packages/preshed/maps.pxd", line 36, in init spacy.gold (spacy/gold.cpp:23499)
there is no imported map or something else...
ctrado18
commented
7 years ago @xoviat So I find out the this has to do with cython. Now im taggling to find out where I have to put the cython module in the .spec file.
I have now tried:
# -*- mode: python -*-
block_cipher = None
bin=[('C:\\Python\\env\\Lib\\site-packages\\spacy\\gold.pxd','spacy'),('C:\\Python\\env\\Lib\\site-packages\\preshed\\maps.pxd','cy')]
a = Analysis(['test.py'],
pathex=['C:\\Users\\name\\Desktop\\test'],
binaries=bin,
datas=[],
hiddenimports=['cymem.cymem'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
exclude_binaries=True,
name='test',
debug=False,
strip=False,
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
name='test')
But still it cant find maps.pxd...
It might have something to do with TOC. But I cannot find an simple example where a cython module is includes within a TOC and how to do that. Eg sould I write it within binaries=['maps.pxd']:
or in the Analysis part something like TOC(name, pathe, EXTENSION)
Sorry, but the docs are quite not understandable for me. If ther would be a simple example then it should be clear.:-)
ghost
commented
7 years ago maps.pxd is a data file. Include the full path in datas
ctrado18
commented
7 years ago Thank you! I hope you can furrther help me further. So you mean in above the analysis class (and only?)
datas=bin?
This yields same error...
Furthermore soemthing in very strange in:
File ".env/lib/python2.7/site-packages/preshed/maps.pxd", line 36, in init spacy.gold (spacy/gold.cpp:23499)
What is this path ".env/lib/"? This does not exist on my system! Im am using a virtual environment... With this I also detect something that it outputs that distutils is retatgeting to non-virtualenv
Maybe the disutils hook redirects to the wrong path for maps.pxd?
@xoviat @
ctrado18
commented
7 years ago I found that adding maps.pxd to data was not successful. But instead importing it directy in my script gives success. So I need to import every module manually by :
from preshed import maps e.g.
So I have to it for many modules like string.pxd in the spacy package:
import spacy (is this problematic?) from spacy import strings Actually I have to include evergy pxd file in the spacy package like this.
Can I circumvent this by adding a path manually somewehere in the spec file?
htgoebel
commented
6 years ago Adding the path does not help. But you can write a hook (resp. two hooks: one for preshed, one for spacy) to search for all these modules and add them as hidden imports. We'd appreciate if you publish the hooks then :-)
Please have a look at https://pyinstaller.readthedocs.io/en/latest/hooks.html, esp. https://pyinstaller.readthedocs.io/en/latest/hooks.html#useful-items-in-pyinstaller-utils-hooks
I'm not sure if collect_submodules() works as expected. If not use get_package_paths() and glob.glob() to collect the files (you need to convert them to submodule-names, though).
timtutt
commented
6 years ago Was there ever any resolution on this or hooks created? I'm running into the same issue actually and I'm not quite sure how to create the hooks necessary.
htgoebel
commented
6 years ago @timtutt See https://github.com/pyinstaller/pyinstaller/issues/2806#issuecomment-349637492 – still somebody needs to implement the solution
wandonye
commented
6 years ago What learned from this thread is that whenever I see some module not found, I can import it in my .py which is fed to pyinstaller. So I'm doing this iteratively for spacy:
Here is what I have now. It is still growing:
from cymem import cymem
from thinc import linalg
from thinc.neural import ops
from murmurhash import mrmr
import dill
from cytoolz import utils, _signatures
from spacy import strings, morphology, vocab, lexeme, parts_of_speech, pipeline, language, syntax
from spacy.tokens import underscore, printers, _retokenize
from spacy.syntax import stateclass, transition_system
I tried creating hook-space.py, with
from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files('spacy')Doesn't work. I don't understand the glob.glob method. Where to put it? what to glob?
Will keep growing the import list. Stay in tune, or let me know if there is an easier way out.
timtutt
commented
6 years ago @wandonye - I abandoned this method in favor of Pex which is amazing. Has it's own issues, but much easier to use.
wandonye
commented
6 years ago @timtutt Thanks for the info. I will take a look at Pex. With my approach above, I have the complete list for spacy now, but still need to incorporate the model data.
farlee2121
commented
6 years ago I made it to the bottom of the rabbit hole. Expanding on wandonye's comment , the final import list is
from cymem import cymem
from thinc import linalg, linear, extra
from thinc.neural import ops
from thinc.linear import avgtron, features, sparse, serialize
from thinc.extra import eg, mb, search
from murmurhash import mrmr
import dill
from cytoolz import utils, _signatures
from spacy import strings, morphology, vocab, lexeme, parts_of_speech, pipeline, language, syntax, cfile, serialize
from spacy.tokens import printers
from spacy.syntax import stateclass, transition_system, iterators, _parse_features, beam_parser
from spacy.serialize import bits, packer, huffmanYou will also need to copy ftfy/char_classes.dat and the language model files (usually en_core_web_sm).
I'm currently working on the hook file, which I'd be happy to contribute
farlee2121
commented
6 years ago I'm currently working on a hook file for spacy. The hidden imports weren't a problem, but i'm very confused by data files.
According to this documentation, I should just be able to add a datas tuple to the hook file, like so
datas = [("ftfy/char_classes.dat", "ftfy")]
I've tried many variants of this (and pkgutil.get_data) and looked all around the documentation to no avail. It doesn't seem to do anything.
Update (2018-08-01): I've gotten datas to work from my .spec with a relative path to my virtualenv. However, the same scheme silently fails (or does nothing) from the hook file. A relative path won't work independent of dev environment either.
I've also tried splitting out a separate hook file for ftfy, with no effect.
asarmohd
commented
5 years ago @farlee2121 : Please share if you have the hook file ready
htgoebel
commented
5 years ago @farlee2121 Goof to hear you are working on a hook.
Not sure I understoodd what the excat problem is, but two points:
collect_data_files is what you want.farlee2121
commented
5 years ago Shoot. I don't currently have access to the project where I did that work. I'll see if I can get a copy of the hook file.
Legorooj
commented
4 years ago What's the status on this folks?
farlee2121
commented
4 years ago The company never replied to me and I think they went out of business. I'll try one more time, but I don't have high hopes
Legorooj
commented
4 years ago @farlee2121 why was your code there in the first place?
farlee2121
commented
4 years ago @Legorooj Because I wrote the code for that company while working for that company
Legorooj
commented
4 years ago @farlee2121 thought so. Can you remember any of the code?
farlee2121
commented
4 years ago It's not really that kind of code. I basically just kept adding dependencies to the list until the generated executable stopped complaining
Legorooj
commented
4 years ago @farlee2121 ah that method. Well let us know if you have any luck with the company - maybe you know one of the workers there? They might respond faster.
Hey,
I hope you can guide me working my exe. :-) I am new to so I need support what I have to done corresponding top the many missing modules!
Pyinstaller 3.3.dev0+d118e6e00 python 2.7.13 Using Anaconda 4.3.23 Windows-7-6.1 SP1 UPX is not avialble
My spec:
Alos follwoing Warning comes out of console:
Internal error: early pywin32 import was introducedAlso I copy the warning message file with the missing modules. I cannot understand why tehre are so many. So I should add all these modules in the hidden import option?