search

IBM / ibm-cos-sdk-python

ibm-cos-sdk-python
Apache License 2.0
45 stars 26 forks source link

ibm_boto3 cannot work along with pyinstaller? #19

Closed thetime1102 closed 5 years ago

thetime1102 commented 5 years ago

Does there anyone use pyinstaller to generate execution file for ibm_boto3 ? I always got error during "The 's3' resource does not exist." please help me.

paul-carron commented 5 years ago

@thetime1102 could you provide more information? If you could provide a full error trace, the command you're running and anything else that might be relevant.

Also, does it work before you package it with pyinstaller?

thetime1102 commented 5 years ago

@paul-carron

To reproduce:

  1. Install library

     pip install ibm-cos-sdk
 pip install pyinstaller

  2. Create test_cos.py:

    import ibm_boto3
from ibm_botocore.client import Config
api_key = 'API_KEY'
service_instance_id = 'SERVICE_INSTANCE_ID'
auth_endpoint = 'https://iam.bluemix.net/oidc/token'
service_endpoint = 'SERVICE_ENDPOINT'

new_bucket = 'testbucket99999'

cos = ibm_boto3.resource('s3',
                     ibm_api_key_id=api_key,
                     ibm_service_instance_id=service_instance_id,
                     ibm_auth_endpoint=auth_endpoint,
                     config=Config(signature_version='oauth'),
                     endpoint_url=service_endpoint)

cos.create_bucket(Bucket=new_bucket)

for bucket in cos.buckets.all():
    print(bucket.name)

  3. Run normally, it works:

    python test_cos.py

  4. Run pyinstaller to generate execution test_cos.exe file:

    pyinstaller -F test_cos.py

  5. Run test_cos.exe, it fails

    test_cos.exe

  6. Bellow is trace back

    Traceback (most recent call last):
  File "test_cos.py", line 1, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-    packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_boto3\__init__.py", line 16, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-    packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_boto3\session.py", line 27, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-    packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\session.py", line 38, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\client.py", line 16, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\waiter.py", line 18, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\docs\__init__.py", line 15, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\docs\service.py", line 18, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\docs\bcdoc\restdoc.py", line 16, in <module>
  File "c:\users\user_name\appdata\local\programs\python\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\ibm_botocore\docs\bcdoc\docstringparser.py", line 16, in <module>
  File "site-packages\ibm_botocore\vendored\six.py", line 89, in __get__
  File "site-packages\ibm_botocore\vendored\six.py", line 108, in _resolve
  File "site-packages\ibm_botocore\vendored\six.py", line 79, in _import_module
ModuleNotFoundError: No module named 'html.parser'
[1564] Failed to execute script test_cos

  7. Platform info:

    Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.machine()
'AMD64'
>>> platform.platform()
'Windows-10-10.0.17134-SP0'
>>> platform.processor()
'Intel64 Family 6 Model 58 Stepping 9, GenuineIntel'
>>> platform.system()
'Windows'
>>> platform.uname()
uname_result(system='Windows', node='DESKTOP', release='10', version='10.0.17134',         machine='AMD64', processor='Intel64 Family 6 Model 58 Stepping 9, GenuineIntel')

>>>pip show ibm-cos-sdk
Name: ibm-cos-sdk
Version: 2.3.1
>>>pip show pyinstaller
Name: PyInstaller
Version: 3.5.dev0+0ffa574bf
paul-carron commented 5 years ago

The issue seems to be with pyinstaller rather than the cos-sdk.

Please try removing the docs and the tests folders from ibm-cos-sdk-python-core and ibm-cos-sdk-python before packaging. I believe these files should not be packaged.

Failing that, you may need to consult the pyinstalled documentation.

thetime1102 commented 5 years ago

@paul-carron i asked pyinstaller team. can you link to that issue? https://github.com/pyinstaller/pyinstaller/issues/3822

paul-carron commented 5 years ago

Did you try try removing the docs and the tests folders from ibm-cos-sdk-python-core and ibm-cos-sdk-python before packaging?

thetime1102 commented 5 years ago

@paul-carron

if removed the docs and the tests folders before packaging, when run script bellow => it's failed

```
python test_cos.py
```

```
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import ibm_boto3
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-        packages\ibm_boto3\__init__.py", line 16, in <module>
    from ibm_boto3.session import Session
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-    packages\ibm_boto3\session.py", line 27, in <module>
    import ibm_botocore.session
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-    packages\ibm_botocore\session.py", line 38, in <module>
    import ibm_botocore.client
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-packages\ibm_botocore\client.py", line 16, in <module>
    from ibm_botocore import waiter, xform_name
  File "C:\Users\user_name\AppData\Local\Programs\Python\Python36\lib\site-    packages\ibm_botocore\waiter.py", line 18, in <module>
    from ibm_botocore.docs.docstring import WaiterDocstring
ModuleNotFoundError: No module named 'ibm_botocore.docs'
```
paul-carron commented 5 years ago

This seems to be common with PyInstaller.

You could try passing the below as a hook:

from PyInstaller.utils.hooks import collect_data_files
from PyInstaller.compat import is_py2
from PyInstaller.utils.hooks import is_module_satisfies

if is_module_satisfies('ibm_botocore >= 2.0.0'):
    if is_py2:
        hiddenimports = ['HTMLParser']
    else:
        hiddenimports = ['html.parser']

datas = collect_data_files('ibm_botocore')
  1. Create the hook file(hook-ibm_botocore.py)
  2. Run this command to package the application pyinstaller -F --additional-hooks-dir=. --hidden-import=ibm_botocore test_cos.py
  3. Run the application

Here's some info that may help: https://pythonhosted.org/PyInstaller/hooks.html https://github.com/pyinstaller/pyinstaller/issues/2094 https://github.com/pyinstaller/pyinstaller/commit/6f455216013b60974c11daf807e604edcc9991b0

paul-carron commented 5 years ago

@thetime1102 Did the above information help you resolve your issue?

thetime1102 commented 5 years ago

@thetime1102 Did the above information help you resolve your issue?

@paul-carron I tried but can not packaging.

To reproduce:

  1. create hook-ibm_botocore.py file

  2. add hook-ibm_botocore.py file to C:\Users\user_name\AppData\Local\Programs\Python\Python36\Lib\site-packages\PyInstaller\hooks\

  3. run command to package the application 

    pyinstaller -F --additional-hooks-dir=. --hidden-import=ibm_botocore test_cos.py

  4. when packaging i got a error like bellow

    15556 INFO: Caching module hooks...
15556 INFO: Analyzing C:\Tutorial\test_ibm_cos\test_cos.py
15783 INFO: Processing pre-safe import module hook   six.moves
25248 INFO: Processing pre-safe import module hook   urllib3.packages.six.moves
30060 INFO: Loading module hooks...
30060 INFO: Loading module hook "hook-certifi.py"...
30074 INFO: Loading module hook "hook-cryptography.py"...
31709 INFO: Loading module hook "hook-encodings.py"...
32079 INFO: Loading module hook "hook-ibm_botocore.py"...
  File "<string>", line 2
    import ibm-botocore as m
              ^
SyntaxError: invalid syntax
32377 INFO: Loading module hook "hook-ibm_botocore.py"...
  File "<string>", line 2
    import ibm-botocore as m
SyntaxError: invalid syntax
32645 INFO: Loading module hook "hook-pydoc.py"...
32645 INFO: Loading module hook "hook-xml.etree.cElementTree.py"...
32660 INFO: Loading module hook "hook-xml.py"...
32932 INFO: Looking for ctypes DLLs
32949 INFO: Analyzing run-time hooks ...
32964 INFO: Looking for dynamic libraries

  5. but .... when i try remove hook-ibm_botocore.py file and add bellow code to test_cos.py file

    import html.parser

  6. and run this command to repackage the application pyinstaller -F test_cos.py

  7. then when run test_cos.exe, i got error like the top comment in this topic ↑

    Traceback (most recent call last):
  File "site-packages\ibm_boto3\session.py", line 422, in resource
  File "site-packages\ibm_botocore\loaders.py", line 132, in _wrapper
  File "site-packages\ibm_botocore\loaders.py", line 378, in load_service_model
ibm_botocore.exceptions.UnknownServiceError: Unknown service: 's3'. Valid service names are:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_cos.py", line 17, in <module>
  File "site-packages\ibm_boto3\__init__.py", line 92, in resource
  File "site-packages\ibm_boto3\session.py", line 428, in resource
ibm_boto3.exceptions.ResourceNotExistsError: The 's3' resource does not exist.
The available resources are:
  -

[7424] Failed to execute script test_cos
cpcunningham commented 5 years ago

Create hook file --- hook-ibm_boto3.py (paste contents between lines)

from PyInstaller.utils.hooks import collect_data_files, collect_submodules

hiddenimports = ( collect_submodules('ibm_boto3.s3'))

datas = collect_data_files('ibm_boto3')

Create hook file --- hook-ibm_botocore.py (paste contents between lines)

from PyInstaller.utils.hooks import collect_data_files from PyInstaller.compat import is_py2 from PyInstaller.utils.hooks import is_module_satisfies

hiddenimports = ['html.parser']

datas = collect_data_files('ibm_botocore')

Run pyinstaller with app pyinstaller --additional-hooks-dir=. pytest.py

cpcunningham commented 5 years ago

I have tested the above and it works, We will not be able to provide any more help on this as pyinstaller is not supported by ibmcos...