Hi, I use athena.add_partition>: to make partition in hourly. So calc current date & hour from session_time by digdag's feature.I guess other ops such as ctas , query ,,, supports nested variables, but add_partition (and maybe drop_partiton) doesn't work well.
In the my example.dig, thrown exception about s3 path, but in our production environment, the partition was created with the value that ${moment(session_time...} was not expanded.
example.dig
If you set ${ny_dt} insted of ${curr_dt} at partition_kv: work normaly.
_export:
plugin:
repositories:
- file://${repos}
# - https://jitpack.io
dependencies:
- pro.civitaspo:digdag-operator-athena:0.4.0
athena:
auth_method: profile
curr_dt: ${moment(session_time).format("YYYY-MM-DD")}
ny_dt: 2020-01-01
+step7:
athena.ctas>: select 1 as a, 2 as b, '${ny_dt}' as dt union all select 4 as a, 5 as b, '${curr_dt}' as dt
database: ${database}
table: hoge
location: ${output}
partitioned_by: [dt]
# +step8:
# athena.drop_partition>:
# database: ${database}
# table: hoge
# partition_kv:
# dt: "${curr_dt}"
+step9:
athena.add_partition>:
database: ${database}
table: hoge
partition_kv:
dt: "${curr_dt}"
error
* +example+step9:
Illegal character in path at index 37: s3://imaoka-oregon-athena/digdag/dt=${moment(session_time).format("YYYY-MM-DD")}/ (illegal argument)
Hi, I use
athena.add_partition>:
to make partition in hourly. So calc current date & hour fromsession_time
by digdag's feature.I guess other ops such asctas , query ,,,
supports nested variables, but add_partition (and maybe drop_partiton) doesn't work well.In the my example.dig, thrown exception about s3 path, but in our production environment, the partition was created with the value that ${moment(session_time...} was not expanded.
example.dig If you set ${ny_dt} insted of ${curr_dt} at
partition_kv:
work normaly.error