civitaspo / digdag-operator-athena

digdag plugin for operating a query on athena.
Apache License 2.0
23 stars 6 forks source link

can not extract nested var on `athena.add_partition>:` at partition keys #84

Open h-imaoka opened 3 years ago

h-imaoka commented 3 years ago

Hi, I use athena.add_partition>: to make partition in hourly. So calc current date & hour from session_time by digdag's feature.I guess other ops such as ctas , query ,,, supports nested variables, but add_partition (and maybe drop_partiton) doesn't work well.

In the my example.dig, thrown exception about s3 path, but in our production environment, the partition was created with the value that ${moment(session_time...} was not expanded.

AWS_Glue_Console

example.dig If you set ${ny_dt} insted of ${curr_dt} at partition_kv: work normaly.

_export:
  plugin:
    repositories:
      - file://${repos}
      # - https://jitpack.io
    dependencies:
      - pro.civitaspo:digdag-operator-athena:0.4.0
  athena:
    auth_method: profile
  curr_dt: ${moment(session_time).format("YYYY-MM-DD")}
  ny_dt: 2020-01-01

+step7:
  athena.ctas>: select 1 as a, 2 as b, '${ny_dt}' as dt union all select 4 as a, 5 as b, '${curr_dt}' as dt
  database: ${database}
  table: hoge
  location: ${output}
  partitioned_by: [dt]

# +step8:
#   athena.drop_partition>:
#   database: ${database}
#   table: hoge
#   partition_kv:
#     dt: "${curr_dt}"

+step9:
  athena.add_partition>:
  database: ${database}
  table: hoge
  partition_kv:
    dt: "${curr_dt}"

error

  * +example+step9:
    Illegal character in path at index 37: s3://imaoka-oregon-athena/digdag/dt=${moment(session_time).format("YYYY-MM-DD")}/ (illegal argument)
civitaspo commented 3 years ago

This issue will be resolved when https://github.com/treasure-data/digdag/issues/862 is resolved.

ref. https://github.com/treasure-data/digdag/pull/1077