Using pre-defined variables inside _export block

xerial commented 8 years ago

It would be useful if we can use pre-defined variables within _export:

_export:
  query_start: 2016-03-01
  query_end: ${session_date}

xerial commented 8 years ago

Even if the variable is defined inside a task, ${xxx} will not be replaced.

frsyuki commented 8 years ago

Doesn't it work? What's the current behavior and what's the expected behavior?

xerial commented 8 years ago

Here is an example to reproduce the problem:

run: +main

_export:
  td:
    database: sampledb
  account: test

+main:
  _export:
    prefix: q_${account}

  +hello:
    td>: sql/select1.sql
    create_table: ${prefix}_hello

$ digdag run -f test.yml -p apikey=$TD_API_KEY -a -E                                                     [11:03:56 Mar 10 2016]
2016-03-11 11:03:58 +0900: Digdag v0.3.5
2016-03-11 11:03:58 +0900 [WARN] (main): --session-time argument, --hour argument, or _schedule in yaml file is not set. Using today's 00:00:00 as ${session_time}.
2016-03-11 11:03:58 +0900 [INFO] (main): Using state files at digdag.status/20160311T000000+0900.
2016-03-11 11:03:58 +0900 [INFO] (main): Starting a new session repository id=1 workflow name=+main session_time=2016-03-11T00:00:00+09:00
2016-03-11 11:03:59 +0900 [INFO] (0021@+main+hello): td>: sql/select1.sql
2016-03-11 11:03:59 +0900 [WARN] (0021@+main+hello):
  td:
    database: "sampledb"
  account: "test"
  timezone: "Asia/Tokyo"
  apikey: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  prefix: "q_${account}"
  session_id: 1
  session_uuid: "63700d1d-91b4-4fc5-84cd-0065ef32f7a0"
  session_time: "2016-03-11T00:00:00+09:00"
  session_date: "2016-03-11"
  session_date_compact: "20160311"
  session_local_time: "2016-03-11 00:00:00"
  session_tz_offset: "+0900"
  session_unixtime: 1457622000
  repository_id: 1
  td>: "sql/select1.sql"
  create_table: "q_${account}_hello"
  _type: "td"
  _command: "sql/select1.sql"

2016-03-11 11:03:59 +0900 [INFO] (0021@+main+hello): td-client version: unknown
2016-03-11 11:03:59 +0900 [INFO] (0021@+main+hello): Logging initialized @1311ms
2016-03-11 11:04:01 +0900 [WARN] (0021@+main+hello): [404:Not Found] API request to /v3/table/delete/sampledb/q_%24%7Baccount%7D_hello has failed: Resource not found
2016-03-11 11:04:01 +0900 [INFO] (0021@+main+hello): Started 55275625 job id=presto:
CREATE TABLE "q_${account}_hello" AS
select 1
2016-03-11 11:04:03 +0900 [WARN] (0021@+main+hello): Job 55275625:
===
started at 2016-03-11T02:04:00Z
executing query: CREATE TABLE "q_${account}_hello" AS select 1
finished at 2016-03-11T02:04:00Z

Query 20160311_020400_21445_9ah8x failed: line 1:1: Column name not specified at position 1

===

frsyuki commented 8 years ago

Fixed at https://github.com/treasure-data/digdag/commit/9c1296343c25877e25f1e026601bd3463c75836f

treasure-data / digdag

Using pre-defined variables inside _export block #8