search

Azure / autorest.python

Extension for AutoRest (https://github.com/Azure/autorest) that generates Python code
MIT License
78 stars 57 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 122: invalid start byte #2705

Closed GrahamMThomas closed 1 month ago

GrahamMThomas commented 1 month ago

Looks related: https://github.com/Azure/autorest.python/issues/2497

PS C:\Users\gthomas\sdk-repos\azure-rest-api-specs\specification\healthdataaiservices\HealthDataAIServices.DeidServices> ..\..\..\eng\scripts\TypeSpec-Generate-Sdk.ps1 C:\azure-sdk-for-python\ . 2771da5baeee73dfd70b2a5f2813a55549c2aa73 https://github.com/Azure/azure-rest-api-specs
Using module powershell-yaml with version 0.4.7.
updated tsp-location.yaml commit to 2771da5baeee73dfd70b2a5f2813a55549c2aa73
updated tsp-location.yaml repo to Azure/azure-rest-api-specs
updated tsp-location.yaml directory to specification/healthdataaiservices/HealthDataAIServices.DeidServices
updated tsp-location.yaml additionalDirectories to
finished updating tsp-location.yaml in C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification
Calling TypeSpec-Project-Sync.ps1
Using module powershell-yaml with version 0.4.7.
Reading configuration from C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\tsp-location.yaml
Copying spec from C:\Users\gthomas\sdk-repos\azure-rest-api-specs\specification\healthdataaiservices\HealthDataAIServices.DeidServices to C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification/TempTypeSpecFiles
Calling TypeSpec-Project-Generate.ps1
Using module powershell-yaml with version 0.4.7.
Reading configuration from C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\tsp-location.yaml
Generating from C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices
Removing existing package.json
Removing existing node_modules
Removing existing package-lock.json
Copying package.json from C:\azure-sdk-for-python\eng\common\scripts\..\..\emitter-package.json
Copying package-lock.json from C:\azure-sdk-for-python\eng\common\scripts\..\..\emitter-package-lock.json
> npm ci
Command succeeded (00:00:33.4586609)

npx tsp compile C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\client.tsp --emit @azure-tools/typespec-python --option @azure-tools/typespec-python.emitter-output-dir=C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification/
Unrecognized definition type "eTag" is found, falling back it as "string"!
Error: failed to format C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\venv\Lib\site-packages\win32com\demos\iebutton.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\black.py", line 71, in <module>
    BlackScriptPlugin(output_folder=args.output_folder, **unknown_args).process()
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\black.py", line 34, in process
    list(
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\black.py", line 57, in format_file
    file_content = self.read_file(file)
                   ^^^^^^^^^^^^^^^^^^^^
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\__init__.py", line 44, in read_file
    return fd.read()
           ^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
  File "C:\Users\gthomas\.pyenv\pyenv-win\versions\3.12.4\Lib\encodings\utf_8_sig.py", line 69, in _buffer_decode
    return codecs.utf_8_decode(input, errors, final)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 122: invalid start byte
Error: Command failed: python3 C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python/scripts/run_tsp.py --output-folder=C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification/ --cadl-file=C:/Users/gthomas/AppData/Local/Temp/cadl-codegen/typespec-python-yaml-map3ad8f047-23f6-412a-ad48-5f797d101b9a.yaml --package-version=1.0.0b1 --generate-packaging-files=true --flavor=azure --package-dir=azure-health-deidentification --package-name=azure-health-deidentification --package-mode=azure-dataplane --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true
Emitter "@azure-tools/typespec-python" crashed! This is a bug.
Please file an issue at https://github.com/Azure/autorest.python/issues

Error: Command failed: C:\Program Files\nodejs\node.exe C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python/scripts/run-python3.cjs C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python/scripts/run_tsp.py --output-folder=C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification/ --cadl-file=C:/Users/gthomas/AppData/Local/Temp/cadl-codegen/typespec-python-yaml-map3ad8f047-23f6-412a-ad48-5f797d101b9a.yaml --package-version=1.0.0b1 --generate-packaging-files=true --flavor=azure --package-dir=azure-health-deidentification --package-name=azure-health-deidentification --package-mode=azure-dataplane --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true
Unrecognized definition type "eTag" is found, falling back it as "string"!
Error: failed to format C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\venv\Lib\site-packages\win32com\demos\iebutton.py
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\black.py", line 71, in <module>
    BlackScriptPlugin(output_folder=args.output_folder, **unknown_args).process()
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\black.py", line 34, in process
    list(
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\black.py", line 57, in format_file
    file_content = self.read_file(file)
                   ^^^^^^^^^^^^^^^^^^^^
  File "C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python\generator\pygen\__init__.py", line 44, in read_file
    return fd.read()
           ^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
  File "C:\Users\gthomas\.pyenv\pyenv-win\versions\3.12.4\Lib\encodings\utf_8_sig.py", line 69, in _buffer_decode
    return codecs.utf_8_decode(input, errors, final)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 122: invalid start byte
Error: Command failed: python3 C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\TempTypeSpecFiles\HealthDataAIServices.DeidServices\node_modules\@azure-tools\typespec-python/scripts/run_tsp.py --output-folder=C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification/ --cadl-file=C:/Users/gthomas/AppData/Local/Temp/cadl-codegen/typespec-python-yaml-map3ad8f047-23f6-412a-ad48-5f797d101b9a.yaml --package-version=1.0.0b1 --generate-packaging-files=true --flavor=azure --package-dir=azure-health-deidentification --package-name=azure-health-deidentification --package-mode=azure-dataplane --models-mode=dpg --emit-cross-language-definition-file=true --from-typespec=true

    at genericNodeError (node:internal/errors:984:15)
    at wrappedFn (node:internal/errors:538:14)
    at checkExecSyncError (node:child_process:890:11)
    at execFileSync (node:child_process:926:15)
    at Object.$onEmit [as emitFunction] (file:///C:/azure-sdk-for-python/sdk/healthdataaiservices/azure-health-deidentification/TempTypeSpecFiles/HealthDataAIServices.DeidServices/node_modules/@azure-tools/typespec-python/dist/src/emitter.js:89:9)
    at async runEmitter (file:///C:/azure-sdk-for-python/sdk/healthdataaiservices/azure-health-deidentification/TempTypeSpecFiles/HealthDataAIServices.DeidServices/node_modules/@typespec/compiler/dist/src/core/program.js:452:13)
    at async compile (file:///C:/azure-sdk-for-python/sdk/healthdataaiservices/azure-health-deidentification/TempTypeSpecFiles/HealthDataAIServices.DeidServices/node_modules/@typespec/compiler/dist/src/core/program.js:132:9)
    at async compileOnce (file:///C:/azure-sdk-for-python/sdk/healthdataaiservices/azure-health-deidentification/TempTypeSpecFiles/HealthDataAIServices.DeidServices/node_modules/@typespec/compiler/dist/src/core/cli/actions/compile/compile.js:37:25)
    at async compileAction (file:///C:/azure-sdk-for-python/sdk/healthdataaiservices/azure-health-deidentification/TempTypeSpecFiles/HealthDataAIServices.DeidServices/node_modules/@typespec/compiler/dist/src/core/cli/actions/compile/compile.js:21:9)

--------------------------------------------------
Library Version                0.25.0
TypeSpec Compiler Version      0.57.0
--------------------------------------------------
Write-Error: C:\Users\gthomas\sdk-repos\azure-rest-api-specs\eng\scripts\TypeSpec-Generate-Sdk.ps1:92
Line |
  92 |      . $commonScript -TypeSpecProjectDirectory $TypeSpecProjectDirecto …
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Failed to generate sdk project at C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification
iscai-msft commented 1 month ago

Thank you for opening this issue @GrahamMThomas , @msyyc can you take a look? thank you!

msyyc commented 1 month ago

@GrahamMThomas According to log failed to format C:\azure-sdk-for-python\sdk\healthdataaiservices\azure-health-deidentification\venv\Lib\site-packages\win32com\demos\iebutton.py, it seems your target folder contains unexpected folder venv. Please clean your target folder then try again.

GrahamMThomas commented 1 month ago

venv is convention for a virtualenv folder. It should probably be ignored by the linter.

GrahamMThomas commented 1 month ago

I recreated using env instead of venv but the same error still exists. That is the name of the folder the Getting Started docs recommend. If I delete both it succeeds.

This means that once you start developing and sdk with a virtual environment, recommended by the Getting Started docs, you have to delete it every time you want to regenerate the SDK with typespec.

I recommend we exclude env and venv (by convention) from black linter if possible.

msyyc commented 1 month ago

We will consider to exclude files under env and venv. As work around, please try to create virtual environment under root folder of sdk repo.