search

argoproj-labs / hera

Hera makes Python code easy to orchestrate on Argo Workflows through native Python integrations. It lets you construct and submit your Workflows entirely in Python. ⭐️ Remember to star!
https://hera.rtfd.io
Apache License 2.0
559 stars 105 forks source link

f-strings break the workflow generator #584

Open sergiynesterenko90 opened 1 year ago

sergiynesterenko90 commented 1 year ago

Hi, I'm trying to use an f-string in my script and am running into issues. I think the presence of curly brackets confuses the workflow compiler somehow. I'm using hera 5.1.3.

For example, I can run the basic example script just fine. But if I make a small change to include an f-string:

from hera.workflows import Steps, Workflow, script

@script()
def echo(message: str):
    print(f"message: {message}")

with Workflow(
    generate_name="single-script-",
    entrypoint="steps",
) as w:
    with Steps(name="steps"):
        echo(arguments={"message": "A"})

w.create()

I get an error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 18
     15     with Steps(name="steps"):
     16         echo(arguments={"message": "A"})
---> 18 wf = {"Workflow": w.to_dict()}
     20 # submit the workflow to Argo
     21 argo.create_workflow(wf)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:319](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:319), in Workflow.to_dict(self)
    317 def to_dict(self) -> Any:
    318     """Builds the Workflow as an Argo schema Workflow object and returns it as a dictionary."""
--> 319     return self.build().dict(exclude_none=True, by_alias=True)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:212](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/workflow.py:212), in Workflow.build(self)
    209     template = template._dispatch_hooks()
    211 if isinstance(template, Templatable):
--> 212     templates.append(template._build_template())
    213 elif isinstance(template, get_args(TTemplate)):
    214     templates.append(template)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:172](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:172), in Script._build_template(self)
    144 def _build_template(self) -> _ModelTemplate:
    145     assert isinstance(self.constructor, ScriptConstructor)
    146     return self.constructor.transform_template_post_build(
    147         self,
    148         _ModelTemplate(
    149             active_deadline_seconds=self.active_deadline_seconds,
    150             affinity=self.affinity,
    151             archive_location=self.archive_location,
    152             automount_service_account_token=self.automount_service_account_token,
    153             daemon=self.daemon,
    154             executor=self.executor,
    155             fail_fast=self.fail_fast,
    156             host_aliases=self.host_aliases,
    157             init_containers=self.init_containers,
    158             inputs=self._build_inputs(),
    159             memoize=self.memoize,
    160             metadata=self._build_metadata(),
    161             metrics=self.metrics,
    162             name=self.name,
    163             node_selector=self.node_selector,
    164             outputs=self._build_outputs(),
    165             parallelism=self.parallelism,
    166             plugin=self.plugin,
    167             pod_spec_patch=self.pod_spec_patch,
    168             priority=self.priority,
    169             priority_class_name=self.priority_class_name,
    170             retry_strategy=self.retry_strategy,
    171             scheduler_name=self.scheduler_name,
--> 172             script=self._build_script(),
    173             security_context=self.pod_security_context,
    174             service_account_name=self.service_account_name,
    175             sidecars=self.sidecars,
    176             synchronization=self.synchronization,
    177             timeout=self.timeout,
    178             tolerations=self.tolerations,
    179             volumes=self._build_volumes(),
    180         ),
    181     )

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:201](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:201), in Script._build_script(self)
    183 def _build_script(self) -> _ModelScriptTemplate:
    184     assert isinstance(self.constructor, ScriptConstructor)
    185     return self.constructor.transform_script_template_post_build(
    186         self,
    187         _ModelScriptTemplate(
    188             args=self.args,
    189             command=self.command,
    190             env=self._build_env(),
    191             env_from=self._build_env_from(),
    192             image=self.image,
    193             image_pull_policy=self._build_image_pull_policy(),
    194             lifecycle=self.lifecycle,
    195             liveness_probe=self.liveness_probe,
    196             name=self.container_name,
    197             ports=self.ports,
    198             readiness_probe=self.readiness_probe,
    199             resources=self._build_resources(),
    200             security_context=self.security_context,
--> 201             source=self.constructor.generate_source(self),
    202             startup_probe=self.startup_probe,
    203             stdin=self.stdin,
    204             stdin_once=self.stdin_once,
    205             termination_message_path=self.termination_message_path,
    206             termination_message_policy=self.termination_message_policy,
    207             tty=self.tty,
    208             volume_devices=self.volume_devices,
    209             volume_mounts=self._build_volume_mounts(),
    210             working_dir=self.working_dir,
    211         ),
    212     )

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:336](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/script.py:336), in InlineScriptConstructor.generate_source(self, instance)
    330     script += "\n"
    332 # We use ast parse/unparse to get the source code of the function
    333 # in order to have consistent looking functions and getting rid of any comments
    334 # parsing issues.
    335 # See https://github.com/argoproj-labs/hera/issues/572
--> 336 content = roundtrip(inspect.getsource(instance.source)).splitlines()
    337 for i, line in enumerate(content):
    338     if line.startswith("def") or line.startswith("async def"):

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:949](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:949), in roundtrip(source)
    947 if hasattr(ast, "unparse"):
    948     return ast.unparse(tree)
--> 949 return unparse(tree)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:942](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:942), in unparse(ast_obj)
    940 def unparse(ast_obj):
    941     unparser = _Unparser()
--> 942     return unparser(ast_obj)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:70](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:70), in _Unparser.__call__(self, node)
     67 """Outputs a source code string that, if converted back to an ast
     68 (using ast.parse) will generate an AST equivalent to *node*"""
     69 self._source = []
---> 70 self.traverse(node)
     71 return "".join(self._source)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
    181         self.traverse(item)
    182 else:
--> 183     super().visit(node)

File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
    369 method = 'visit_' + node.__class__.__name__
    370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:194](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:194), in _Unparser.visit_Module(self, node)
    192 def visit_Module(self, node):
    193     self._type_ignores = {ignore.lineno: f"ignore{ignore.tag}" for ignore in node.type_ignores}
--> 194     self._write_docstring_and_traverse_body(node)
    195     self._type_ignores.clear()

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190), in _Unparser._write_docstring_and_traverse_body(self, node)
    188     self.traverse(node.body[1:])
    189 else:
--> 190     self.traverse(node.body)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181), in _Unparser.traverse(self, node)
    179 if isinstance(node, list):
    180     for item in node:
--> 181         self.traverse(item)
    182 else:
    183     super().visit(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
    181         self.traverse(item)
    182 else:
--> 183     super().visit(node)

File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
    369 method = 'visit_' + node.__class__.__name__
    370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:374](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:374), in _Unparser.visit_FunctionDef(self, node)
    373 def visit_FunctionDef(self, node):
--> 374     self._function_helper(node, "def")

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:392](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:392), in _Unparser._function_helper(self, node, fill_suffix)
    390     self.traverse(node.returns)
    391 with self.block(extra=self.get_type_comment(node)):
--> 392     self._write_docstring_and_traverse_body(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:190), in _Unparser._write_docstring_and_traverse_body(self, node)
    188     self.traverse(node.body[1:])
    189 else:
--> 190     self.traverse(node.body)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:181), in _Unparser.traverse(self, node)
    179 if isinstance(node, list):
    180     for item in node:
--> 181         self.traverse(item)
    182 else:
    183     super().visit(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
    181         self.traverse(item)
    182 else:
--> 183     super().visit(node)

File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
    369 method = 'visit_' + node.__class__.__name__
    370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:233](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:233), in _Unparser.visit_Assign(self, node)
    231     self.traverse(target)
    232     self.write(" = ")
--> 233 self.traverse(node.value)
    234 if type_comment := self.get_type_comment(node):
    235     self.write(type_comment)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:183), in _Unparser.traverse(self, node)
    181         self.traverse(item)
    182 else:
--> 183     super().visit(node)

File [~/anaconda3/envs/quilt/lib/python3.8/ast.py:371](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/ast.py:371), in NodeVisitor.visit(self, node)
    369 method = 'visit_' + node.__class__.__name__
    370 visitor = getattr(self, method, self.generic_visit)
--> 371 return visitor(node)

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:511](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:511), in _Unparser.visit_JoinedStr(self, node)
    509 for value in node.values:
    510     meth = getattr(self, "_fstring_" + type(value).__name__)
--> 511     meth(value, self.buffer_writer)
    512     buffer.append((self.buffer, isinstance(value, Constant)))
    513 new_buffer = []

File [~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:546](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/sergiy/git/quilt/quilt/notebooks/sergiy/~/anaconda3/envs/quilt/lib/python3.8/site-packages/hera/workflows/_unparse.py:546), in _Unparser._fstring_FormattedValue(self, node, write)
    544 unparser.set_precedence(_Precedence.TEST.next(), node.value)
    545 expr = unparser.visit(node.value)
--> 546 if expr.startswith("{"):
    547     write(" ")  # Separate pair of opening brackets as "{ {"
    548 if "\\" in expr:

AttributeError: 'NoneType' object has no attribute 'startswith'
flaviuvadan commented 1 year ago

Hey @sergiynesterenko90! Thanks for reporting this! We are aware of this, and it's actually clearly manifesting in the 2 PRs we currently have up #613 and #606. This is caused by the AST unparse functionality we added for backwards compatibility with Py3.8. I think #613 has a potential fix but we also have to fix Hera's CI for this because different Py versions AST modules result in different scripts being tested, which breaks CI. Going to try to fix this issue as part of those PRs and will post back here

yiftachn commented 3 months ago

I think this should be highlighted in the docs as the error is ambiguous and took me quite some time to figure out. Perhaps adding a better warning message?

samj1912 commented 3 months ago

This only happens with Python 3.8 which is deprecated and will be end of life soon. Our preference would be to drop support for 3.8 soon and ask users to upgrade to the next minor version of python instead.