Please describe the feature you'd like to see
Check if there is a way to restrict the columns while loading from file to table using Astro SDK load_file operator?
Example Scenario - I got a file in S3 having 6 columns, now the requirement is that I load only the required 5 columns from the file on S3 to the GBQ table. Relevant thread https://astronomer.slack.com/archives/C02B8SPT93K/p1663775775875379
Describe the solution you'd like
A clear and concise description of what you want to happen, if possible with code examples.
Are there any alternatives to this feature?
Is there another way we could solve this problem or enable this use-case?
Additional context
Add any other context about the feature request here.
Acceptance Criteria
[ ] All checks and tests in the CI should pass
[ ] Unit tests (90% code coverage or more, once available)
[ ] Integration tests (if the feature relates to a new database or external service)
[ ] Example DAG
[ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
[ ] Exception handling in case of errors
[ ] Logging (are we exposing useful information to the user? e.g. source and destination)
[ ] Improve the documentation (README, Sphinx, and any other relevant)
sunank200
commented
1 year ago
Please describe the feature you'd like to see Check if there is a way to restrict the columns while loading from file to table using Astro SDK load_file operator? Example Scenario - I got a file in S3 having 6 columns, now the requirement is that I load only the required 5 columns from the file on S3 to the GBQ table. Relevant thread https://astronomer.slack.com/archives/C02B8SPT93K/p1663775775875379
Describe the solution you'd like A clear and concise description of what you want to happen, if possible with code examples.
Are there any alternatives to this feature? Is there another way we could solve this problem or enable this use-case?
Additional context Add any other context about the feature request here.
Acceptance Criteria