dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.7k stars 1.48k forks source link

[docs] - shell_op has no API docs #7989

Open dagsir[bot] opened 2 years ago

dagsir[bot] commented 2 years ago

Dagster Documentation Gap

This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1652939569223799?thread_ts=1652939569.223799&cid=C01U954MEER

Conversation excerpt

U038JAU853L: Hey, we are managing our Spark clusters “manually”, so are ending up using create_shell_command_op to submit a Spark job. Since we want to create the command dynamically during execution we end up using shell_op, however the https://docs.dagster.io/_apidocs/libraries/dagster-shell|documentation is missing there.

Is this the way to go? Any chance there is documentation of the shell_op?

Thanks in advance to the always helpful Dagster team :pray: U015C9U9RLK: Hi Pieter, that’s an oversight that it’s missing a doc string, thanks for catching that. The code for shell_op is here: https://github.com/dagster-io/dagster/blob/7207a6e2dc3fd3a6e9705ca361b9f5a18204c1e3/python_modules/libraries/dagster-shell/dagster_shell/solids.py#L51-L82|https://github.com/dagster-io/dagster/blob/7207a6e2dc3fd3a6e9705ca361b9f5a18204c1e[…]/python_modules/libraries/dagster-shell/dagster_shell/solids.py U015C9U9RLK: So it takes shell_command as an input, and env, cwd, etc. as config U038JAU853L: Thanks a lot <@U015C9U9RLK>! U015C9U9RLK: <@U018K0G2Y85> docs shell_op has no API docs


Message from the maintainers:

Are you looking for the same documentation content? Give it a :thumbsup:. We factor engagement into prioritization.

yuhan commented 2 years ago

apidoc is missing here: https://docs.dagster.io/_apidocs/libraries/dagster-shell#dagster_shell.shell_op