Closed yousiku closed 5 years ago
Can you share the output of pip freeze
?
Edit: Sorry, I just realize this worked fine without pyinstaller. Did you look at the spec file to see if there is anything odd in there? I'm not super experienced with pyinstaller, but I can take a look if you want.
The error that you are getting indicates that when the system tries to import the gevent handling code there was something missing, so check for missing dependencies.
here is my spec file:
# -*- mode: python -*-
block_cipher = None
a = Analysis(['ems\\ems_station_service.py'],
pathex=['ems'],
binaries=None,
datas=[
('ems/core/rule_parser/*.py', 'core/rule_parser/'),
('ems/data/ems.conf', 'data'),
('ems/data/pid', 'data/pid'),
('ems/data/images', 'data/images'),
],
hiddenimports=[
'netifaces',
'gevent.ssl',
'gevent.builtins',
'crc16',
'ctypes',
],
hookspath=None,
runtime_hooks=None,
excludes=None,
win_no_prefer_redirects=None,
win_private_assemblies=None,
cipher=block_cipher)
a.binaries + [('my_lib.dll', 'ems\\data\\lib', 'BINARY')]
import platform
if platform.system().find("Windows")>= 0:
a.datas = [i for i in a.datas if i[0].find('Include') < 0]
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
exclude_binaries=True,
name='ems_station_service',
debug=False,
strip=None,
icon='ems\\data\\images\\logo.ico',
upx=True,
console=True )
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=None,
upx=True,
name='ems_station_service')
I have no idea about this problem... I am honored that you are willing take a look.
I think it is a fairly common problem that pyinstaller missing some dependencies when those dependencies are optional (i.e. not explicitly listed as dependencies for other projects).
Take a look at how to explicitly include packages as hidden imports: https://pythonhosted.org/PyInstaller/when-things-go-wrong.html#listing-hidden-imports. I think you should add gevent and gevent-websocket as hidden imports. Let me know if that works.
It works!!! I add 'engineio.async_gevent' to hiddenimports in spec file. very, very grateful for your help!! lol hahahahaha
I have the same problem, howevere I am not using gevent and not uwsgi, since I am using Python 3. I am using eventlet. Did someone manage to solve this issue with eventlet?
@Popkultur You need to ensure engineio.async_eventlet
is included in the installer package.
@Popkultur Would you mind sharing your pip freeze
, python 3.x version ? I am trying to bundle the wsgi-example - the latency test - which either complains about wrong async_mode
or fails to import name greenio
. I am using python 3.4.2
and win32com build 220 64bit on a clean Win7sp1 Here is my latency.spec
:
# -*- mode: python -*-
result = Analysis(
['latency.py'],
hiddenimports=['engineio.async_eventlet'],
hookspath=None,
runtime_hooks=None
)
def Datafiles(*filenames, **kw):
[...]
templates = Datafiles('templates/latency.html', strip_path=False)
static = Datafiles('static/style.css', strip_path=False)
pyz = PYZ(result.pure)
exe = EXE(
pyz,
result.scripts,
result.binaries,
result.zipfiles,
result.datas,
templates,
static,
name='latency.exe',
debug=True,
strip=None,
upx=False,
console=True
)
Any help is greatly appreciated.
@espretto did you add eventlet to your list of hidden imports?
Yes, I tried adding 'eventlet
' to the above hiddenimports
but to no avail. Here is what I tried so far for python versions 2.7.13
, 3.3.5
, 3.4.2
and 3.5.3
on Win7sp1:
$ mkvirtualenv latency
(latency)$ pip install eventlet python-socketio Flask pyinstaller
(latency)$ pyinstaller --onefile latency.spec
(latency)$ .\dist\latency.exe
which gives me this:
Traceback (most recent call last):
File "latency.py", line 9, in <module>
sio = socketio.Server(async_mode=async_mode)
File "site-packages\socketio\server.py", line 82, in __init__
File "site-packages\engineio\server.py", line 114, in __init__
ValueError: Invalid async_mode specified
installing pyinstaller
apparently pulls in the correct version of pywin32
itself. python latency.py
runs just fine in every case.
update
apparently engineio
uses the importlib
module to dynamically import whichever module corresponds to the chosen async_mode
. hardcoding that part now leaves me at these lines:
Traceback (most recent call last):
File "latency.py", line 9, in <module>
sio = socketio.Server(async_mode=async_mode)
File "site-packages\socketio\server.py", line 82, in __init__
File "site-packages\engineio\server.py", line 103, in __init__
File "c:\users\test\envs\latency-2.7.13\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.__dict__)
File "site-packages\engineio\async_eventlet.py", line 4, in <module>
File "c:\users\test\envs\latency-2.7.13\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.__dict__)
File "site-packages\eventlet\__init__.py", line 10, in <module>
File "c:\users\test\envs\latency-2.7.13\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
exec(bytecode, module.__dict__)
File "site-packages\eventlet\convenience.py", line 3, in <module>
ImportError: cannot import name greenio
greenio
as in eventlet.greenio
, no the standalone pip module with the same name.
This is my working SPEC:
# -*- mode: python -*-
block_cipher = None
added_files = [ ( 'db/*.*', 'db' ),
( 'templates/*.*', 'templates' ),
( 'RECORD', 'RECORD' ),
( 'uploads', 'uploads' )
]
a = Analysis(['main.py'],
pathex=['C:\Kamopticon'],
binaries=[],
datas=added_files,
hiddenimports=['engineio.async_eventlet'],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries + [('msvcp120.dll', 'C:\\Windows\\System32\\msvcp120.dll', 'BINARY'),
('msvcr120.dll', 'C:\\Windows\\System32\\msvcr120.dll', 'BINARY')],
a.zipfiles,
a.datas,
Tree('static/', 'static'),
name='main',
debug=False,
strip=False,
upx=True,
console=True )
And I used it within https://github.com/Popkultur/Kamopticon
I can't really help you with specifics, as I don't use pyinstaller, but the invalid async mode error means that the file async_eventlet.py is missing, so whatever you are doing in your configuration to include this file does not seem to be working.
The second import error on eventlet.greenio
suggests eventlet is also not being imported as a complete package. It seems eventlet.convenience
is there, but then with this file tries to import eventlet.greenio
it founds that this module is missing. You need to find a way to get the whole thing included in the package.
Thank you very much. I managed to get past the import errors although some pyinstaller warnings persist and now I've run into some python34.dll
error. It would seem my Windows registry got caught up installing multiple versions of python and win23com (=pypiwin32 I think) - should have taken a snapshot beforehand. I'll get back to you as soon as make any progress on this. Might be a valuable addition to the example for bootstrapping web-based desktop apps.
@espretto If you have a spec file that works well with regards to the imports, it would be awesome if you can post it here, so that people looking for the same answers will find it. Thanks!
Got it! eventlet==0.20.1
doesn't play well with pyinstaller==3.2.1
. Here goes my setup - detail-peppered because obviously things break easily:
find a pyinstaller-spec for the wsgi example to bundle in a dep-less executable file.
pyinstaller --onefile latency.spec
.\dist\latency.exe
systeminfo | findstr "OS"
OS Name: Microsoft Windows 7 Professional
OS Version: 6.1.7601 Service Pack 1 Build 7601
[...]
python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 20:20:57) [MSC v.1600 64 bit (AMD64)] on win32
pip freeze
(note eventlet version is not the latest 0.20.1
)appdirs==1.4.2
click==6.7
enum-compat==0.0.2
eventlet==0.19.0
Flask==0.12
future==0.16.0
greenlet==0.4.12
itsdangerous==0.24
Jinja2==2.9.5
MarkupSafe==0.23
packaging==16.8
PyInstaller==3.2.1
pyparsing==2.1.10
pypiwin32==219
python-engineio==1.2.3
python-socketio==1.7.1
six==1.10.0
Werkzeug==0.11.15
although pyinstaller
depends on pypiwin32
(confirm by pip freeze
) it may still not work because that does not register any .dll
-files as is (maybe?) required by pyinstaller
. to this end, download the version corresponding to the one of your venv's (mind 32/64bit modes) from their sourceforge mirror and install as admin (they won't ask but report at the end instead).
the pyinstaller latency.spec
file (put next to latency.py
):
# -*- mode: python -*-
result = Analysis(
['latency.py'],
hiddenimports=['engineio.async_eventlet'],
hookspath=None,
runtime_hooks=None
)
# from pyinstaller wiki
def Datafiles(*filenames, **kw):
import os
def datafile(path, strip_path=True):
parts = path.split('/')
path = name = os.path.join(*parts)
if strip_path:
name = os.path.basename(path)
return name, path, 'DATA'
strip_path = kw.get('strip_path', True)
return TOC(
datafile(filename, strip_path=strip_path)
for filename in filenames
if os.path.isfile(filename)
)
templates = Datafiles('templates/latency.html', strip_path=False)
static = Datafiles('static/style.css', strip_path=False)
pyz = PYZ(result.pure)
exe = EXE(
pyz,
result.scripts,
result.binaries,
result.zipfiles,
result.datas,
templates,
static,
name='latency.exe',
debug=False,
strip=None,
upx=True,
console=True
)
The resulting executable does work, even on a clean machine with none of the prerequisites installed. the only one thing missing is static files. Somehow they still slip through but this would be a question for their board I guess.
Another issue is that disconnecting throws an error. That however may very well be a shortcoming of the minimal example implementation that doesn't explicitely handle the on_close
event of engineio
. Here is the output on closing the browser tab:
(2500) accepted ('127.0.0.1', 49697)
(2500) accepted ('127.0.0.1', 49698)
(2500) accepted ('127.0.0.1', 49699)
(2500) accepted ('127.0.0.1', 49700)
127.0.0.1 - - [24/Feb/2017 13:15:41] "GET /socket.io/?EIO=3&transport=websocket&
sid=e6fa88f2dfa94fa29cb31085c883fff9 HTTP/1.1" 200 0 12.054688
Traceback (most recent call last):
File "site-packages\eventlet\greenpool.py", line 82, in _spawn_n_impl
File "site-packages\eventlet\wsgi.py", line 719, in process_request
File "socketserver.py", line 675, in __init__
File "site-packages\eventlet\wsgi.py", line 636, in finish
File "site-packages\eventlet\greenio\base.py", line 474, in shutdown_safe
OSError: [WinError 10038] An operation was attempted on something that is not a
socket
Update:
The missing static files were very well included as indicated by pyinstaller's .toc
report. It was jinja2 that wasn't aware of being run from within a package. To make it so one can swap jinja2's loader:
# [...]
app = Flask(__name__)
# [...]
import sys, jinja2, os.path as ospath
class MeipassLoader(jinja2.BaseLoader):
def __init__(self, path):
self.path = ospath.join(sys._MEIPASS, path)
def get_source(self, environment, template):
path = ospath.join(self.path, template)
if not ospath.exists(path):
raise jinja2.TemplateNotFound(template)
mtime = ospath.getmtime(path)
with open(path) as f:
source = f.read()
return source, path, lambda: mtime == ospath.getmtime(path)
if hasattr(sys, '_MEIPASS'):
app.jinja_loader = jinja2.ChoiceLoader([
app.jinja_loader,
MeipassLoader('templates')
])
todo: greenify the above loader.
credits:
@espretto awesome, thanks for the super fine details!
Another issue is that disconnecting throws an error
Yeah, that is fairly common. Eventlet on Linux/Mac is a bit better, on Windows several people already reported this error. It's benign though, it does not affect the server.
I couldn't get around all the errors with eventlet & PyInstaller, so I uninstalled eventlet, and am trying to use the async_mode="threading"
. I also put import threading
in my main script file. When I try to run after pyinstaller--onefile socket_app.py
, I get a Invalid async_mode
specified error. This comes from \site-packages\engineio\server.py
__init__
method. I'm baffled. Any ideas?
@crobertsbmw make sure the engineio/async_threading.py
module is also imported, so that pyinstaller pulls it, along with all of its dependencies.
@miguelgrinberg Perfect. All I had to do was add from engineio import async_threading
to the top of my script file to trick PyInstaller into pulling it, and that did the trick. Thanks so much!
eventlet==0.20.1 doesn't play well with pyinstaller==3.2.1.
Thanks @crobertsbmw, this helped me too. Or is it that pyinstaller
doesn't play well with eventlet
?Either way, falling back to threading is a good workaround.
I've looked into why pyinstaller doesn't work with eventlet under flask-socketio. First off there are two hidden (run-time dynamic) imports required (first is part of flask-socketio; the second is from within eventlet):
engineio.async_eventlet eventlet.support.dns
Unfortunately eventlet\support\greendns.py
loads eventlet.support.dns
by adding the full path to eventlet\support
to sys.path
(see code below) and tries to import the eventlet.support.dns
module using just the identifier dns
. This method (modifying sys.path at run-time) is incompatible with pyinstaller. eventlet\support\greendns.py
needs to be modified to work with pyinstaller
The added ##
comments are mine
def import_patched(module_name):
# Import cycle note: it's crucial to use _socket_nodns here because
# regular evenlet.green.socket imports *this* module and if we imported
# it back we'd end with an import cycle (socket -> greendns -> socket).
# We break this import cycle by providing a restricted socket module.
# if (module_name + '.').startswith('dns.'):
# module_name = 'eventlet.support.' + module_name
modules = {
'select': select,
'time': time,
'os': os,
'socket': _socket_nodns,
}
return patcher.import_patched(module_name, **modules)
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__))) ## This does not do the expected thing in a pyinstaller package
dns = import_patched('dns') ## Hence this is not found even after using --hidden-import=eventlet.support.dns
for pkg in dns.__all__:
setattr(dns, pkg, import_patched('dns.' + pkg))
for pkg in dns.rdtypes.__all__:
setattr(dns.rdtypes, pkg, import_patched('dns.rdtypes.' + pkg))
for pkg in dns.rdtypes.IN.__all__:
setattr(dns.rdtypes.IN, pkg, import_patched('dns.rdtypes.IN.' + pkg))
for pkg in dns.rdtypes.ANY.__all__:
setattr(dns.rdtypes.ANY, pkg, import_patched('dns.rdtypes.ANY.' + pkg))
del import_patched
sys.path.pop()
@tonk777, I see that you quoted and commented the relevant passages from greendns.py
, indicating what does not work.
Do you know what would work? How should greendns.py
be modified?
I did not investigate this particular issue any further because I switched to using an alternate library supported by flask-socketio which worked for me with pyinstaller (gevent). However I can point you towards a solution
The key for pyinstaller support is to avoid this -> os.path.dirname(__file__
) as __file__
is not defined in the expected way in a "bundled" environment. You need to do a run-time check for if you are in a bundled environment and go from there. The relevant info that should allow a good fix is in this section of the pyinstaller docs.
http://pyinstaller.readthedocs.io/en/stable/runtime-information.html
It may just be that when in a "bundled" environment (sys.frozen exists) just a different path evaluation to the eventlet.support.dns
module is needed (based off sys._MEIPASS and not __file__
) and the sys.path temporary modification is still usable
Hope this helps
fwiw - for closure - https://github.com/eventlet/eventlet/pull/486 seems to permit PyInstaller to pack a script that uses eventlet, given this hook-eventlet.support.greendns.py
file in a PyInstaller --additional-hooks-dir
directory:
# PyInstaller hook for dynamic import of dnspython
from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('dns')
It works!!! I add 'engineio.async_gevent' to hiddenimports in spec file. very, very grateful for your help!! lol hahahahaha
thanks a lot ,resolve my problem
It appears you now need
'engineio.async_drivers.eventlet' not 'engineio.async_eventlet'
in hidden imports using the below version:
python-engineio 3.3.0 python-socketio 3.1.2 eventlet 0.24.1
@CpHarding Yes, that is correct, the async drivers have been through a restructure in the v3 engineio package.
I'm still struggling with this. I'm using the following command:
python -m PyInstaller -F signage.py --hidden-import=engineio.async_drivers.eventlet
My versions:
eventlet==0.24.1
PyInstaller==3.4
python-engineio==3.4.3
python-socketio==3.1.2
Error result:
Traceback (most recent call last):
File "W:\Source\Sign\signage.py", line 21, in <module>
socketio = SocketIO(app)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\flask_socketio\__init__.py", line 163, in __init__
self.init_app(app, **kwargs)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\flask_socketio\__init__.py", line 221, in init_app
self.server = socketio.Server(**self.server_options)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\socketio\server.py", line 89, in __init__
self.eio = self._engineio_server_class()(**engineio_options)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\engineio\server.py", line 129, in __init__
raise ValueError('Invalid async_mode specified')
ValueError: Invalid async_mode specified
[4936] Failed to execute script signage
just importing gevent from engineio.async_drivers worked for me.
from engineio.async_drivers import gevent
I wrote a server in eventlet, came across all the nasty incompatibility issues listed above, switched to gevent, still no luck trying all proposed solutions in this thread.
PyInstaller develop branch (06f7da78) via pip install -e
Went ahead and rewrote it using aiohttp. used this as a template: https://github.com/miguelgrinberg/python-socketio/blob/master/examples/server/aiohttp/app.py Finally worked!
no hidden imports needed, no bajillion dns module required lol
I solved this issue by copy whole "engineio" directory to my dist/app/"engineio" For example, copy your {{python installl dir}}/Lib/site-packages/engineio to /dist/{{your app}}/engineio. then run your app
just importing gevent from engineio.async_drivers worked for me.
from engineio.async_drivers import gevent
I also have to --hidden-import=gevent
from engineio.async_drivers import eventlet worked for me socketio = SocketIO(app, async_mode='eventlet') --hidden-import=engineio.async_eventlet
Limiting to requirements inside an virtualenv fixed all issues for me. Somehow pyinstaller doesn't like that with the regular:
sudo python3 -m pip install --user virtualenv
python3 -m venv env
source env/bin/activate
pip3 install --requirement requirements.txt
This reduced the file size for me too.
I use eventlet
. None of the above solved my problem. Finally I got it run by these 3 steps:
app.spec
file:hiddenimports=['engineio.async_drivers.eventlet'],
app.py
fileAdd these lines at file head, it's useless to the python file, but it's necessary for pyinstaller packing.
from eventlet.hubs import epolls, kqueue, selects
from dns import dnssec, e164, hash, namedict, tsigkeyring, update, version, zone
pyinstaller
pyinstaller app.spec
Do NOT run pyinstaller app.py
, it will overwrite app.spec
.
Maybe the solution is just as simple as importing that module like this: from engineio.async_drivers import gevent
Right! If you have to use multi-threading mode like me, you can add line as below and pyinstall it:
from engineio.async_drivers import threading
and also force it to use threading mode when create it:
socketio = SocketIO(
app,
async_mode="threading"
)
just importing gevent from engineio.async_drivers worked for me.
from engineio.async_drivers import gevent
Thank you very much !!!!!!!!
pip install eventlet
did the trick for me
Flask-SocketIO is Running under Werkzeug
hiddenimports=['engineio.async_drivers.threading']
I can package and run .exe file as normal,but the socketio is disabled.The front end cannot be automatically updated.
Can anyone give me some advice? Thank you
@fengerzh You saved me, thanks alot!
Right! If you have to use multi-threading mode like me, you can add line as below and pyinstall it:
from engineio.async_drivers import threading
and also force it to use threading mode when create it:
socketio = SocketIO( app, async_mode="threading" )
This might be useful for those using eventlet. I have verified this works for the package and OS combinations below
Using Pyinstaller 5.4.1 and Eventlet 0.33.1 in Windows11 required the following hidden imports when invoking Eventlet from Flask-SocketIO 5.2.0
hiddenimports=[
'engineio.async_drivers.eventlet',
'eventlet.hubs.epolls',
'eventlet.hubs.kqueue',
'eventlet.hubs.selects',
'dns',
'dns.asyncbackend',
'dns.asyncquery',
'dns.asyncresolver',
'dns.e164',
'dns.namedict',
'dns.tsigkeyring',
'dns.versioned'
]
Note that using the solution
from engineio.async_drivers import threading
socketio = SocketIO (
app,
async_mode="threading"
)
generates the following warning at runtime on a console output window
WARNING: This is a development server. Do not use it in a production deployment. Use a production
WSGI server instead.
* Serving Flask app <...>'
* Debug mode: off
The WebSocket transport is not available, you must install a WebSocket server that is compatible with
your async mode to enable it. See the documentation for details. (further occurrences of this error will
be logged with level INFO)
I.e. using async_mode="eventlet" is prefered over async_mode="threading" for a production release of your application. Sure "threading" may remove the headache of trying to work out what hidden modules you need via eventlet but in the long run its not a robust solution
The biggest problem in all this is that Flask-SocketIO hides the stack traces generated when a module is missing and thus hiding from the developer vital clues as to what modules need to be added to pyinstaller's hiddenimports list. Instead you get the bland
...
File "site-packages\engineio\server.py", line 100, in __init__
ValueError: Invalid async_mode specified
It would be more than useful if the underlying stack trace and exception reason was also output so that discovering the contents of the hiddenimports list could be built up by the developer
> This might be useful for those using eventlet. I have verified this works for the package and OS combinations below
>
> Using Pyinstaller 5.4.1 and Eventlet 0.33.1 in Windows11 required the following hidden imports when invoking Eventlet from Flask-SocketIO 5.2.0
>
> ```
> hiddenimports=[
> 'engineio.async_drivers.eventlet',
> 'eventlet.hubs.epolls',
> 'eventlet.hubs.kqueue',
> 'eventlet.hubs.selects',
> 'dns',
> 'dns.asyncbackend',
> 'dns.asyncquery',
> 'dns.asyncresolver',
> 'dns.e164',
> 'dns.namedict',
> 'dns.tsigkeyring',
> 'dns.versioned'
> ]
Thanks for this list, all above solutions were not working in my case. 🔥🫶
I also struggled with this error, quite a bit. Different stuff worked differently for different people. I fixed the error by adding eventlet to the --collect-submodels param in pyinstaller. Here is more scources to check if anybody still encounter this error:
ValueError: Invalid async_mode specified" when bundling a Flask app using cx_Freeze Recommended way to [create executables] of a Python app for [deployment] on a virtual machine
Right! If you have to use multi-threading mode like me, you can add line as below and pyinstall it:
from engineio.async_drivers import threading
and also force it to use threading mode when create it:
socketio = SocketIO( app, async_mode="threading" )
This solved my issue, thanks!!!
add to the hidden imports in spec file
hiddenimports=[ 'flask_socketio','engineio'],
add his line into your server
from engineio.async_drivers import threading
and i update the socket to :
socket = SocketIO(app, cors_allowed_origins="*",async_mode="threading")
add to the hidden imports in spec file
hiddenimports=[ 'flask_socketio','engineio'],
add his line into your server
from engineio.async_drivers import threading
and i update the socket to :
socket = SocketIO(app, cors_allowed_origins="*",async_mode="threading")
Modify the content of the engineio.async_drivers
import threading
Adding 'engineio.async_drivers.eventlet',
to specfile's hiddenimports fixed the problem!
hello! Everything was OK when I run my python file, but it raised error run after be packed with pyinstaller:
the code :
self.socketio = socketio.Server(async_mode='gevent')
I tried self.socketio = socketio.Server(), it's also useless, and i have installed gevent.