search

ansys / pyansys-protos-generator

Automatically generate a python package from protofiles stored according to gRPC protofile conventions
3 stars 1 forks source link

Bug located in generator.py #12

Open mmoales opened 2 years ago

mmoales commented 2 years ago

šŸ” Before submitting the issue

šŸž Description of the bug

Running the tool on Windows 10 results in ".proto: No such file or directory" error. This is because .proto is not automatically expanded when the command is executed in either PowerShell or a command prompt on Windows 10. For example:

PS D:\ANSYSDev\cpbu\DataManagement> python -m ansys.tools.protos_generator .\protos\ansys\api\data_management\v0
.\protos\ansys\api\data_management\v0\*.proto: No such file or directory
Traceback (most recent call last):
  File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\mmoales\AppData\Local\pypoetry\Cache\virtualenvs\data-management-_2qRC6Wq-py3.9\lib\site-packages\ansys\tools\protos_generator\__main__.py", line 46, in <module>
    main()
  File "C:\Users\mmoales\AppData\Local\pypoetry\Cache\virtualenvs\data-management-_2qRC6Wq-py3.9\lib\site-packages\ansys\tools\protos_generator\__main__.py", line 42, in main
    out = package_protos(args.protosfiles_path, args.outdir)
  File "C:\Users\mmoales\AppData\Local\pypoetry\Cache\virtualenvs\data-management-_2qRC6Wq-py3.9\lib\site-packages\ansys\tools\protos_generator\generator.py", line 222, in package_protos
    grpc_source_path = build_python_grpc(protos_path)
  File "C:\Users\mmoales\AppData\Local\pypoetry\Cache\virtualenvs\data-management-_2qRC6Wq-py3.9\lib\site-packages\ansys\tools\protos_generator\generator.py", line 179, in build_python_grpc
    raise RuntimeError(f'Failed to run:\n\n{cmd}')
RuntimeError: Failed to run:

python -m grpc_tools.protoc -I.\protos\ansys\api\data_management\v0 --python_out=C:\Users\mmoales\AppData\Local\Temp\ivtgnivsed --grpc_python_out=C:\Users\mmoales\AppData\Local\Temp\ivtgnivsed --python_out=C:\Users\mmoales\AppData\Local\Temp\ivtgnivsed --grpc_python_out=C:\Users\mmoales\AppData\Local\Temp\ivtgnivsed .\protos\ansys\api\data_management\v0\*.proto

To fix this locally, I modified generator.py in my local site-packages as follows:

    2   +import platform
175     -    cmd += f'--python_out={rndpath} --grpc_python_out={rndpath} {proto_glob}'
    176 +
    177 +    # Windows does not automatically expand *.proto when executing the command
    178 +    if platform.system() == "Linux":
    179 +        cmd += f'--python_out={rndpath} --grpc_python_out={rndpath} {proto_glob}'
    180 +    else:
    181 +        for proto_file in files:
    182 +            cmd += f'{proto_file} '

After making this change, I then get an encoding error:

  File "C:\Users\mmoales\AppData\Local\pypoetry\Cache\virtualenvs\data-management-_2qRC6Wq-py3.9\lib\site-packages\ansys\tools\protos_generator\generator.py", line 243, in package_protos
    py_source[module_name] = open(filename).read()
  File "D:\Python39\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3918: character maps to <undefined>

To fix this, I made an additional change:

235      -        py_source[module_name] = open(filename).read()
236      -
237      -    # Replace all imports for each module with an absolute import with
    242  +        # On Windows, the generated files are encoded as utf-8
    243  +        if platform.system() == "Linux":
    244  +            py_source[module_name] = open(filename).read()
    245  +        else:
    256  +            with open(filename, "r", encoding="utf-8") as py_file:
    247  +                py_source[module_name] = py_file.read()
    248  +
    249  +                # Replace all imports for each module with an absolute import with

šŸ“ Steps to reproduce

Using a command prompt or PowerShell on Windows 10, run the command to generate python code in a project that has multiple .proto files.

šŸ’» Which operating system are you using?

Windows

šŸ Which Python version are you using?

3.9

šŸ“¦ Installed packages

ansys-tools-protoc-helper @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/db/45/2f/0c3e16291c2823252d30148589448945ead6acc3ca68a25e66e740431a/ansys_tools_protoc_helper-0.1.1-py3-none-any.whl
ansys-tools-protos-generator @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/c5/19/b9/22bc82c42d56d5b07cbcae42cdf12c42bbb87ea4b6936b33afccf2da2d/ansys-tools-protos-generator-0.1.0.tar.gz
atomicwrites @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/1b/09/43/f8f36d57c7cc6f6a038e649e082fa3ea231c686664e203db89b869ada7/atomicwrites-1.4.1.tar.gz
attrs @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/5b/25/94/5d165cd3190cb7dfdc98de86c28e8131fc6e19d83a504412664abffec0/attrs-21.4.0-py2.py3-none-any.whl
autopep8 @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/15/55/a2/7075f9dfff25e2826bbab4bb098990d8ba18525cc73af0bb5ee2a2f837/autopep8-1.6.0-py2.py3-none-any.whl
beautifulsoup4 @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/5a/51/f6/ced036f2b2ef6b792e8138e7bd9584e5eac63a3d4481b6e62402d050c7/beautifulsoup4-4.11.1-py3-none-any.whl
bleach @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/4b/c1/10/5c27ed639f813cb93dc7844a6c9ff9f36477cd1787b62fc34080816fbe/bleach-5.0.1-py3-none-any.whl
certifi @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/b0/0d/df/4aa04d9b421fc5f1438b55d4b37516d21ef0ba40158bf7f9dfb28122ca/certifi-2022.6.15-py3-none-any.whl
charset-normalizer @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/bf/59/3f/6838dadc818c6595ae4a00c31af0da89039c1eaa370871d38072845e8d/charset_normalizer-2.1.0-py3-none-any.whl
colorama @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/28/7e/1b/4d5c7485d20628c199b003e9107afc53d57aea621842a1bc23c31c0da1/colorama-0.4.5-py2.py3-none-any.whl
commonmark @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/11/56/f6/d054064b623fab5c7e4420f60d931f49fea2dacdebe1dc991201010c84/commonmark-0.9.1-py2.py3-none-any.whl
coverage @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/9f/ec/88/d11ba3dc5ba7b4685187d8ca5464f9e5e92621ef01bd5fffc904efce7b/coverage-6.4.1-cp39-cp39-win_amd64.whl
docutils @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/63/d6/60/f5c4d603a43ca45f1af4c8b3fe289e841d3612db3d464d24bdb8582159/docutils-0.19-py3-none-any.whl
google @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/43/d6/5b/456ed25006e331f5aa28f0cf503c6d16416eb46e98f6d8912a57f499ad/google-3.0.0-py2.py3-none-any.whl
grpcio @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/9b/93/49/d1444567eb8dda716f863592982c64b94337b3e18d3795a8027e1c20d1/grpcio-1.47.0-cp39-cp39-win_amd64.whl
grpcio-testing @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/f8/da/d2/15ee49a7cc25579a22370c44529a6b9b5a6568cff9124d7f8ea8c19681/grpcio_testing-1.47.0-py3-none-any.whl
grpcio-tools @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/98/f4/bd/801428bcd47ac5218e46c6efc4e92fedde5b4a8b15c07e161325152a54/grpcio-tools-1.17.0.tar.gz
idna @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/d5/04/80/9c17fd3240a37d12ff2ef042b0306aeb1abd2d8b95f150fd60be938352/idna-3.3-py3-none-any.whl
importlib-metadata @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/f8/0d/7d/ca2de6219731687bee983bc0d41e131dfe5284437d92c69e3a76eea562/importlib_metadata-4.12.0-py3-none-any.whl
importlib-resources @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/17/80/4e/fa2632c5e9b11a3dba00055bc8aa405771e04df64b49415157913343cb/importlib_resources-5.8.0-py3-none-any.whl
iniconfig @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/fa/b0/c6/10cfac68c9e6de9d2a1678366ca89fd9292b362c1760dbe758e41691cb/iniconfig-1.1.1-py2.py3-none-any.whl
jsonschema @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/b9/ee/4e/fc5771115226f7827b32ced727aeea54a3b761132fd1ea505037e4e9e3/jsonschema-4.6.2-py3-none-any.whl
keyring @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/26/e6/0a/08255801aee39a9f296ad967c812e867cfef7441a6fdc147af2a1b5f08/keyring-23.6.0-py3-none-any.whl
mccabe @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/55/c8/90/13b9930158ea6f8556dc549fd07ca894cbf0316f5f2f1da02d29d19f85/mccabe-0.7.0-py2.py3-none-any.whl
mypy-protobuf @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/2e/61/18/c2985f204fa3a33f4cf938fcc960f51f37b2d0d7d113a18c30421fe5e5/mypy_protobuf-3.1.0-py3-none-any.whl
numpy @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/9d/d6/73/cf1767b5e8acb38f454ea768ecb9878d2b2594db520b1a6528e28f71fa/numpy-1.23.1-cp39-cp39-win_amd64.whl
packaging @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/47/3f/ce/b240169f7d8bef1ff24a0269b709721ce86543c2ec25e0b6adb2c2d7ac/packaging-21.3-py3-none-any.whl
pkginfo @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/06/41/87/1244b3324c42f41a2fb1e4ee0cd4dbc9e843e5654a0d8c7a5a68249a54/pkginfo-1.8.3-py2.py3-none-any.whl
pluggy @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/81/78/ca/13f743a3628faf5a0b7f021efb45f2193acba3a13663d498f6b34bf02e/pluggy-1.0.0-py2.py3-none-any.whl
protobuf @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/39/94/1a/9f470c620e18198ec4eead704c0f316dd6cc6f64cb066eac96d3d81c8a/protobuf-3.19.3-cp39-cp39-win_amd64.whl
py @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/b3/5c/47/ba5a596e01a2b61fa2daa6a438252483ad8c04e6c99e5dc22eaf8a489a/py-1.11.0-py2.py3-none-any.whl
pycodestyle @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/80/25/e5/ced4effd2487693e96521aebb353ce58ad0d81417f71c480419ea8f44a/pycodestyle-2.8.0-py2.py3-none-any.whl
pydocstyle @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/75/e7/e5/1acad15a51efd39cf39259c7888c205fd787a92efea28f7afc5a9e315c/pydocstyle-6.1.1-py3-none-any.whl
pyflakes @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/ad/8e/6d/e46117c96aa3e955c40ed2f0d35b29910b0e557e327fbbba17fd8c390c/pyflakes-2.4.0-py2.py3-none-any.whl
Pygments @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/7b/08/56/6328ccebb5deef75b77c4ac4fae65b8e5302a1c0b847754cd4c5063f9e/Pygments-2.12.0-py3-none-any.whl
pylama @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/e3/17/6a/6e3aa35286a89c0dd178003cd84437cceeff5b95200efac7b2bc694059/pylama-8.3.8-py3-none-any.whl
pyparsing @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/c9/49/7b/434b2668226040d1097efcdddb802b28d05b02854b659ac17913a9a90c/pyparsing-3.0.9-py3-none-any.whl
pyrsistent @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/a8/a0/90/feff5541a3ee6cc5dd96709e418499d1f6fe2f51327ae5580e54cbc1c3/pyrsistent-0.18.1-cp39-cp39-win_amd64.whl
pytest @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/a6/1c/db/e6f5a63ca07ec1971c4169e071b56ba077ceaf0abc2f69b803f2adcb2c/pytest-7.1.2-py3-none-any.whl
pytest-cov @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/50/df/c5/4bd35027c6247daac4ba547aff90d97ccdc7daf4aa62e110d69cfd39de/pytest_cov-3.0.0-py3-none-any.whl
pywin32-ctypes @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/23/c0/7f/323e49b70e624cbaf28fc19cb426d77dd8dfd370725f3ad7daa4b65d51/pywin32_ctypes-0.2.0-py2.py3-none-any.whl
readme-renderer @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/bd/a6/98/7a97d238fe0b13e99326d5dfa6088d3272ebf9645c3481cba665785b68/readme_renderer-35.0-py3-none-any.whl
requests @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/96/68/40/1b2019f1ca6a6d07b2a3107daab4dca1e11111be3af28a435f7cd0463a/requests-2.28.1-py3-none-any.whl
requests-toolbelt @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/8e/17/53/6cd3006356454687a5ea705caf4f164292347c1fcf4939b1a43716ddb6/requests_toolbelt-0.9.1-py2.py3-none-any.whl
rfc3986 @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/29/b1/1a/3deca527f1d75a95e45234e848cdec1cd8cb232a56a9b86451cad82e7d/rfc3986-2.0.0-py2.py3-none-any.whl
rich @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/b3/da/06/f753ae26f3cc205a3def96a552e410661396615bece9067caaeadca921/rich-12.5.1-py3-none-any.whl
six @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/08/9f/47/c16ae03035fc69eaf100ea39657a49baaeef714e25a52575710c34cd48/six-1.16.0-py2.py3-none-any.whl
snowballstemmer @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/a4/95/b0/c0f70d4b9bb0bac123e716da53ba9b012071cedf7c99bcf030757530f4/snowballstemmer-2.2.0-py2.py3-none-any.whl
soupsieve @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/b7/15/27/3ba8eed3fabe07703423999fa8c50ba16ab30344fb2670163cb6510481/soupsieve-2.3.2.post1-py3-none-any.whl
tiledb @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/ba/96/b3/32fec494dd36dbbe42aaa28b1b85ee23da25d2cd96af7a04d362bef2b5/tiledb-0.16.1-cp39-cp39-win_amd64.whl
toml @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/6b/6a/c9/53b19f7870a77d855e8b05ecdc98193944e5d246dafe11bbcad850ecba/toml-0.10.2-py2.py3-none-any.whl
tomli @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/73/7c/d9/9f2752fc5b05f9176c6f3adc6484be1cec75a68925b8c5fe39d6493a07/tomli-2.0.1-py3-none-any.whl
twine @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/d9/ed/99/b71d600d6ee79a947b95a83878dd458e173012f66de1b420f3c53adeb5/twine-4.0.1-py3-none-any.whl
types-protobuf @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/7e/7a/69/8b2eb939c1c65ad88b8f4abcd8e9b6342fd401ddc53f9b3472e6e424e7/types_protobuf-3.19.22-py3-none-any.whl
urllib3 @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/fb/d7/ea/afeac748fa6a875802b02d2f5f9375bd0154a9bb765a0181f43e283b0e/urllib3-1.26.10-py2.py3-none-any.whl
webencodings @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/ed/d4/da/61384706cfac042ba3bd148746d66e50695463993be117c7c8dadeef7a/webencodings-0.5.1-py2.py3-none-any.whl
zipp @ file:///C:/Users/mmoales/AppData/Local/pypoetry/Cache/artifacts/8a/08/12/1ce534dd356211524f826d50d2a2bfe2e99a8f3cac7355cac56639b06b/zipp-3.8.0-py3-none-any.whl
mmoales commented 2 years ago

The encoding issue was actually due to some weird characters in our python doc strings. Once those were removed, the encoding option is no longer needed when reading the python files. Here are my current changes in generator.py

-    cmd += f'--python_out={rndpath} --grpc_python_out={rndpath} {proto_glob}'
+    cmd += f'--python_out={rndpath} --grpc_python_out={rndpath} '
+
+   # Windows does not automatically expand *.proto when executing the command
+    if platform.system() == "Linux":
+        cmd += f'{proto_glob}'
+    else:
+        for proto_file in files:
+            cmd += f'{proto_file} '