Prefect agent crashes when subprocess exits on Windows

nicholasjng commented 2 years ago

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Prefect documentation for this issue.
[X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

When running a flow to completion in Prefect on Windows via a Prefect agent tied to a deployment, I encounter an exception related to the decoding of some data chunks in the shutdown process.

Reproduction

# Code is private right now unfortunately - I will try to come up with an MWE in the following days and amend the issue.

Error

Traceback (most recent call last):
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\cli\_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\utilities\asyncutils.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\utilities\asyncutils.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\_core\_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "C:\Users\nju\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\nju\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 646, in run_until_complete
    return future.result()
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\cli\agent.py", line 93, in start
    async with OrionAgent(
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\agent.py", line 271, in __aexit__
    await self.shutdown(*exc_info)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\agent.py", line 260, in shutdown
    await self.task_group.__aexit__(*exc_info)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\infrastructure\submission.py", line 48, in submit_flow_run
    return await infrastructure.run(task_status=task_status)
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\infrastructure\process.py", line 66, in run
    process = await run_process(
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\utilities\processutils.py", line 76, in run_process
    await consume_process_output(
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\utilities\processutils.py", line 90, in consume_process_output
    async with anyio.create_task_group() as tg:
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\prefect\utilities\processutils.py", line 104, in stream_text
    async for item in source:
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\abc\_streams.py", line 31, in __anext__
    return await self.receive()
  File "C:\Users\nju\Workspaces\python\venv\lib\site-packages\anyio\streams\text.py", line 44, in receive
    decoded = self._decoder.decode(chunk)
  File "C:\Users\nju\AppData\Local\Programs\Python\Python310\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 95: invalid continuation byte
An exception occurred.

Versions

Version:             2.0.4
API version:         0.8.0
Python version:      3.10.6
Git commit:          39db6fb1
Built:               Wed, Aug 10, 2022 1:19 PM
OS/Arch:             win32/AMD64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.37.2

Additional context

No response

nicholasjng commented 2 years ago

Quick follow-up: I can confirm that this issue does not occur on my local development machine (macOS M1). Specs are the following:

➜ prefect version
Version:             2.0.4
API version:         0.8.0
Python version:      3.10.6
Git commit:          39db6fb1
Built:               Wed, Aug 10, 2022 1:19 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.39.2

Quick log showing the expected baseline results:

20:33:26.892 | INFO    | Flow run 'pearl-swift' - Finished in state Completed()
20:33:27.651 | INFO    | prefect.infrastructure.process - Process 'pearl-swift' exited cleanly.

zanieb commented 2 years ago

If encountered, ensure that PYTHONUTF8=1 is set when starting the agent.

nicholasjng commented 2 years ago

If encountered, ensure that PYTHONUTF8=1 is set when starting the agent.

Yes, that fixes it. I guess this can be closed.

specialkapa commented 1 year ago

@zanieb Thanks for this! Can you please explain how to set PYTHONUTF8=1? Is this an environment variable? Is it sufficient to add in to my user/system environment variables?

PrefectHQ / prefect