search

treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.3k stars 221 forks source link

Trying to call a bash script works in local mode but not client mode #325

Closed westwick closed 7 years ago

westwick commented 7 years ago

I have a bash script in my workflow that makes a curl call. It works locally, but when I push the workflow up to TD and run it with digdag start (instead of digdag run locally of course), I get the following error:

Parameter 'repositories' is required but not set (config)

When I remove the bash script step it works as expected, so I'd guess it's something with the bash script (permissions?) or else the curl call?

westwick commented 7 years ago

After supplying dummy values for repositories and dependencies I got the real error message: Unknown task type: sh (config). So I think there is a problem with sh>: command in client mode. I tried running in a docker container as suggested in the docs but got the same error message, using following definition:

_export:
  docker:
    image: ubuntu:14.04
westwick commented 7 years ago

I see rob mentioned and the docs now say these (sh, py, rb) are not supported at all except in local mode. I'm just curious as to why even have them in local mode then? Isn't local mode just for developing? Is there some other use I'm not seeing, or how are other people using these commands?

frsyuki commented 7 years ago

I think the precise answer there is that hosted environment doesn't support scripting operators. Server mode of digdag itself can run scripting operators. A reason of this limitation is that the hosted environment is multi-tenant which is shared by multiple users who do not trust each other. To avoid security issues, scripts must run in a sandbox. Digdag uses Docker for this purpose. However, digdag doesn't have complete support for using Docker as a sandbox environment for multi-tenancy: docker itself doesn't have storage quota per container. Thus digdag needs to implement extra code to limit storage consumption per container, or the system around digdag needs to survive on disk-full errors.

frsyuki commented 7 years ago

Good news. Docker supports storage size limit per container since v0.12.0 apparently. https://github.com/docker/docker/commit/b16decfccfdb0749c490be9272cb7b4789be87b4 We should try this.