I work with Meltano/DBT in a Windows environment, and containerize the solution into Debian. With the v0.2.0 release I started getting the issue which can be seen by the callstack below.
elt-pipeline: [main ≡ +1 ~1 -0 !]> meltano invoke dbt-snowflake:compile
2024-05-23T11:16:33.760512Z [info ] Environment 'dev' is active
Extension executing `dbt clean`...
Traceback (most recent call last):
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\meltano\edk\extension.py", line 126, in pass_through_invoker
self.pre_invoke(None, *command_args)
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\dbt_ext\extension.py", line 72, in pre_invoke
self.dbt_invoker.run_and_log("clean")
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\meltano\edk\process.py", line 179, in run_and_log
result = asyncio.run(self._exec(sub_command, *args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\meltano\edk\process.py", line 137, in _exec
loop.add_signal_handler(
File "C:\Program Files\Python311\Lib\asyncio\events.py", line 578, in add_signal_handler
raise NotImplementedError
NotImplementedError
pre_invoke failed with uncaught exception, please report to maintainer
Traceback (most recent call last):
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\meltano\edk\extension.py", line 126, in pass_through_invoker
self.pre_invoke(None, *command_args)
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\dbt_ext\extension.py", line 72, in pre_invoke
self.dbt_invoker.run_and_log("clean")
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\meltano\edk\process.py", line 179, in run_and_log
result = asyncio.run(self._exec(sub_command, *args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\Git\elt-pipeline\.meltano\utilities\dbt-snowflake\venv\Lib\site-packages\meltano\edk\process.py", line 137, in _exec
loop.add_signal_handler(
File "C:\Program Files\Python311\Lib\asyncio\events.py", line 578, in add_signal_handler
raise NotImplementedError
NotImplementedError
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x00000250E1B10540>
Traceback (most recent call last):
File "C:\Program Files\Python311\Lib\asyncio\base_subprocess.py", line 126, in __del__
self.close()
File "C:\Program Files\Python311\Lib\asyncio\base_subprocess.py", line 104, in close
proto.pipe.close()
File "C:\Program Files\Python311\Lib\asyncio\proactor_events.py", line 109, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Program Files\Python311\Lib\asyncio\base_events.py", line 762, in call_soon
self._check_closed()
File "C:\Program Files\Python311\Lib\asyncio\base_events.py", line 520, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000250E1B11C60>
Traceback (most recent call last):
File "C:\Program Files\Python311\Lib\asyncio\proactor_events.py", line 116, in __del__
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\proactor_events.py", line 80, in __repr__
info.append(f'fd={self._sock.fileno()}')
^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\windows_utils.py", line 102, in fileno
raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000250E1B11C60>
Traceback (most recent call last):
File "C:\Program Files\Python311\Lib\asyncio\proactor_events.py", line 116, in __del__
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\proactor_events.py", line 80, in __repr__
info.append(f'fd={self._sock.fileno()}')
^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\windows_utils.py", line 102, in fileno
raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
I work with Meltano/DBT in a Windows environment, and containerize the solution into Debian. With the v0.2.0 release I started getting the issue which can be seen by the callstack below.
Thanks to Derek Visch's insight, I had to pin this EDK back to v0.1.0. It seems that the EDK is using a code here https://github.com/meltano/edk/blob/v0.4.1/meltano/edk/process.py#L139C39-L139C50 that breaks in Windows, but not in Linux environments. Supposedly the fix to this is found here https://stackoverflow.com/questions/35772001/how-to-handle-a-signal-sigint-on-a-windows-os-machine/72637975#72637975.
https://meltano.slack.com/archives/C069CQNHDNF/p1716463234362109