search

astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
347 stars 43 forks source link

Add ability to turn off reading and writing of headers with load_file and export_file #910

Open pankajastro opened 2 years ago

pankajastro commented 2 years ago

Please describe the feature you'd like to see https://www.notion.so/astronomerio/ML-Batch-Inference-with-Astro-SDK-Issues-Summary-0703c2fe6abf485da514b45e5db9651f https://github.com/astronomer/sagemaker-batch-inference/blob/lineage-test/dags/sagemaker-batch-inference.py

Relevant thread https://astronomer.slack.com/archives/C02B8SPT93K/p1663942705854959

Tasks:

kaxil commented 2 years ago

Another one for you @feluelle if you like to take a stab at this some point next week :)

phanikumv commented 2 years ago

Moving to 1.3.0 as we are not sure that this will be completed by the deadline.

phanikumv commented 1 year ago

Native path implementation is pending (For example: BigQuery)