This PR provides two options for passing S3 destination key into the S3-to-Snowflake operator:
s3_destination_key: original, direct approach
s3_destination_dir and s3_destination_filename: new, split approach
The new approach is necessary in Texas when using dynamic task mapping: the directory and filename are both dynamic kwargs that cannot be combined into a single key before being passed as an argument.
feature/s3_to_snow_split_path
Description & motivation
This PR provides two options for passing S3 destination key into the S3-to-Snowflake operator:
s3_destination_key: original, direct approachs3_destination_dirands3_destination_filename: new, split approachThe new approach is necessary in Texas when using dynamic task mapping: the directory and filename are both dynamic kwargs that cannot be combined into a single key before being passed as an argument.
PR Merge Priority:
This is necessary for Texas prod.
Changes to existing files:
See above.
New files created:
Tests and QC done:
Robbie has tested this locally on Texas Stadium.