Open blarghmatey opened 3 years ago
Fizzadar
commented
3 years ago 🤦 had not tested the operations with Path objects. I've added tests for non-string StringCommand bits in https://github.com/Fizzadar/pyinfra/commit/2e4053f28f278e873499f6b716f20e4900dbcd1c and handled them in https://github.com/Fizzadar/pyinfra/commit/b91281e9e00106e4c1860b0209aa3bd8abf0ee19. I have just released this in v1.4.7 which should fix this issue!
blarghmatey
commented
3 years ago Thanks for the quick turnaround!
blarghmatey
commented
3 years ago After testing it is unfortunately not fixed yet. For now I can coerce the Path objects to str in the function call. Longer term it's a design question of whether you want to support Path objects as inputs to the function. If that is the case then it might be beneficial to add a helper that does that coercion early in the function calls for operations that take paths as input to make the internal representation uniform rather than having to do the coalescing at multiple locations throughout the code.
I had tried out moving the change that you made in line 99 of https://github.com/Fizzadar/pyinfra/commit/b91281e9e00106e4c1860b0209aa3bd8abf0ee19 above line 94 where the current error is happening. That resolved the traceback noted here, but resulted in a different traceback at a separate code location because of the parameter still being a Path object. That is why I think this merits a more thorough design consideration as opposed to attempting to patch the immediate issue.
Fizzadar
commented
3 years ago This should now be handled properly in https://github.com/Fizzadar/pyinfra/commit/48d6ff7e445856fdc20e971536872dbfcdcf5f4c - Path objects should absolutely be supported IMO but the string conversion wasn't in the right place. Have confirmed this is now working properly for both facts & operations.
I've also added https://github.com/Fizzadar/pyinfra/issues/622 to track adding some proper tests for this!
blarghmatey
commented
3 years ago So, unfortunately I'm still running into bugs with Path arguments. I think the problem is that the type conversion is happening too far down in the stack. What would probably be more robust and easier to maintain is to do the conversion at the perimeter of the system as opposed to doing a whack-a-mole bug hunt.
What I am envisioning is to have a decorator for all operations that accept a Path object that does the conversion before it even makes it into the body of the operation function. Alternatively, if there is a common layer within the operations mechanism that can serve as this choke point then that would allow for file (and other operations that would benefit from Path objects) to be agnostic to the argument type and do the conversion automatically.
I'm happy to have a real-time conversation to discuss the design/architecture aspects.
Fizzadar
commented
3 years ago @blarghmatey agreed - the commit fixes unrelated issues but I definitely jumped the gun a bit here! Would be great to chat real-time about this and understand the issue better before identifying a proper fix (I've emailed you!).
Describe the bug
When attempting to execute the
files.fileorfiles.directoryoperations defined here an exception is raised while attempting to execute the file fact.This error is new as of v1.4.5 and v1.4.6. Previous versions work without error.
To Reproduce
Clone the repository at github.com/mitodl/ol-infrastructure and create a deploy file that contains:
Expected behavior
The expected behavior is to download and unpack the Vault binary and then set the permissions and create a configuration directory. Instead it generates a traceback.
Meta
Include output of
pyinfra --support.-vvand--debug.During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 624, in print_exception context = self.format_context(context) File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 67, in saferepr return _safe_repr(object, {}, None, 0, True)[0] File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 569, in _safe_repr rep = repr(object) File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.repr File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in repr return 'StringCommand({0})'.format(self.get_masked_value()) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value for bit in self._get_all_bits(lambda bit: bit.get_masked_value()) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits bit = shlex_quote(bit) File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command return self.executor.run_shell_command(self.state, self, *args, **kwargs) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command actual_command = command.get_raw_value() File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value return self.separator.join(self._get_all_bits( File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits bit = bit_accessor(bit) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 624, in print_exception context = self.format_context(context) File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 67, in saferepr return _safe_repr(object, {}, None, 0, True)[0] File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 569, in _safe_repr rep = repr(object) File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.repr File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in repr return 'StringCommand({0})'.format(self.get_masked_value()) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value for bit in self._get_all_bits(lambda bit: bit.get_masked_value()) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits bit = shlex_quote(bit) File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "src/gevent/greenlet.py", line 896, in gevent._gevent_cgreenlet.Greenlet._Greenletreport_error File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 541, in handle_error self.print_exception(context, type, value, tb) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 627, in print_exception context = repr(context) File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.repr__ File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in repr return 'StringCommand({0})'.format(self.get_masked_value()) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value for bit in self._get_all_bits(lambda bit: bit.get_masked_value()) File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits bit = shlex_quote(bit) File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object 2021-06-28T21:04:59Z <callback at 0x7fcfd9ddd5c0 stopped> failed with TypeError
--> An unexpected exception occurred in: src/bilder/images/edxapp/deploy.py:
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/util.py", line 79, in exec_file exec(PYTHON_CODES[filename], data) File "src/bilder/images/edxapp/deploy.py", line 124, in
install_hashicorp_products(hashicorp_products)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/deploy.py", line 142, in decorated_func
func(*args, kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/src/bilder/components/hashicorp/steps.py", line 71, in install_hashicorp_products
files.file(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operation.py", line 365, in decorated_func
commands = unroll_generators(func(*actual_args, *actual_kwargs))
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/util.py", line 192, in unroll_generators
for item in generator:
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/operations/files.py", line 1075, in file
info = host.get_fact(File, path=path)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 138, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/facts.py", line 336, in get_host_fact
fact_data = get_facts(state, name, args=args, kwargs=kwargs, ensure_hosts=(host,))
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/facts.py", line 276, in get_facts
status, combined_output_lines = greenlet.get()
File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
raise value.with_traceback(tb)
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
return self.executor.run_shell_command(self.state, self, args, kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
actual_command = command.get_raw_value()
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
bit = bit_accessor(bit)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object