U038JAU853L: Hey, we are managing our Spark clusters “manually”, so are ending up using create_shell_command_op to submit a Spark job. Since we want to create the command dynamically during execution we end up using shell_op, however the https://docs.dagster.io/_apidocs/libraries/dagster-shell|documentation is missing there.
Is this the way to go? Any chance there is documentation of the shell_op?
Dagster Documentation Gap
This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1652939569223799?thread_ts=1652939569.223799&cid=C01U954MEER
Conversation excerpt
U038JAU853L: Hey, we are managing our Spark clusters “manually”, so are ending up using
create_shell_command_op
to submit a Spark job. Since we want to create the command dynamically during execution we end up usingshell_op
, however the https://docs.dagster.io/_apidocs/libraries/dagster-shell|documentation is missing there.Is this the way to go? Any chance there is documentation of the
shell_op
?Thanks in advance to the always helpful Dagster team :pray: U015C9U9RLK: Hi Pieter, that’s an oversight that it’s missing a doc string, thanks for catching that. The code for
shell_op
is here: https://github.com/dagster-io/dagster/blob/7207a6e2dc3fd3a6e9705ca361b9f5a18204c1e3/python_modules/libraries/dagster-shell/dagster_shell/solids.py#L51-L82|https://github.com/dagster-io/dagster/blob/7207a6e2dc3fd3a6e9705ca361b9f5a18204c1e[…]/python_modules/libraries/dagster-shell/dagster_shell/solids.py U015C9U9RLK: So it takesshell_command
as an input, andenv
,cwd
, etc. as config U038JAU853L: Thanks a lot <@U015C9U9RLK>! U015C9U9RLK: <@U018K0G2Y85> docs shell_op has no API docsMessage from the maintainers:
Are you looking for the same documentation content? Give it a :thumbsup:. We factor engagement into prioritization.