treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.3k stars 221 forks source link

If error in `_export` then `_error` task is not execute #1203

Open katsuyan opened 5 years ago

katsuyan commented 5 years ago

Overview

if error in _export then _error task is not execute.

Reproduction

Reference to huga in _export, but huga is not difined. And, I expect run echo error!!!!!!!!! in error, but ReferenceError: "huga" is not defined error occur again.

+task1:
  _export:
    hoge: ${huga}
  sh>: echo "hoge"

_error:
  sh>: echo error!!!!!!!!!
$ java -jar ~/src/github.com/katsuyan/digdag/pkg/digdag-0.9.40-SNAPSHOT.jar run workflow.dig --rerun
2019-08-18 20:57:53 +0900: Digdag v0.9.40-SNAPSHOT
2019-08-18 20:57:55 +0900 [INFO] (main): Database migration started
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20151204221156
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160602123456
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160602184025
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160610154832
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160623123456
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160719172538
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160817123456
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160818043815
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160818220026
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160908175551
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160926123456
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20160928203753
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20161005225356
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20161028112233
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20161110112233
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20161209001857
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20170116082921
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20170116090744
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20170223220127
2019-08-18 20:57:55 +0900 [INFO] (main): Applying database migration:20190318175338
2019-08-18 20:57:55 +0900 [INFO] (main): Database migration successfully finished.
2019-08-18 20:57:56 +0900 [WARN] (main): Reusing the last session time 2019-08-17T00:00:00+00:00.
2019-08-18 20:57:56 +0900 [INFO] (main): Using session /Users/katsuya.tajima/src/github.com/katsuyan/digdag-example/notificaiton/.digdag/status/20190817T000000+0000.
2019-08-18 20:57:56 +0900 [INFO] (main): Starting a new session project id=1 workflow name=workflow session_time=2019-08-17T00:00:00+00:00
2019-08-18 20:57:57 +0900 [ERROR] (0018@[0:default]+workflow+task1): Configuration error at task +workflow+task1: Failed to evaluate a variable ${huga} (ReferenceError: "huga" is not defined) (config)
2019-08-18 20:57:57 +0900 [ERROR] (0019@[0:default]+workflow^error): Configuration error at task +workflow^error: Failed to evaluate a variable Failed to evaluate a variable ${huga} (ReferenceError: "huga" is not defined) (config) (ReferenceError: "huga" is not defined) (config)
2019-08-18 20:57:57 +0900 [INFO] (0018@[0:default]+workflow^failure-alert): type: notify
error:
  * +workflow+task1:
    Failed to evaluate a variable ${huga} (ReferenceError: "huga" is not defined) (config)
  * +workflow^error:
    Failed to evaluate a variable Failed to evaluate a variable ${huga} (ReferenceError: "huga" is not defined) (config) (ReferenceError: "huga" is not defined) (config)

Task state is saved at /Users/katsuya.tajima/src/github.com/katsuyan/digdag-example/notificaiton/.digdag/status/20190817T000000+0000 directory.
  * Use --session <daily | hourly | "yyyy-MM-dd[ HH:mm:ss]"> to not reuse the last session time.
  * Use --rerun, --start +NAME, or --goal +NAME argument to rerun skipped tasks.

Reference

I wrote details at https://qiita.com/katsuyan/items/9846749fa8ccb98cd022 (Japanese)

yoyama commented 4 years ago

Sorry for my late response. I agree with that invalid _export definitions cause fail of _error: task is not good behavior. We need to think about how to solve this issue.