Github tasks can run in parallel

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

https://ask.astronomer.io/

Apache License 2.0

192 stars 47 forks source link

Github tasks can run in parallel #115

Closed mpgreg closed 10 months ago

mpgreg commented 10 months ago

https://github.com/astronomer/ask-astro/blob/c45487c7f12a9424dbe885580c687e35e30b7de4/airflow/dags/ingestion/ask-astro-load.py#L253-L262

After switching Airflow docs ingest to HTML it is no longer necessary to ingest github tasks in parallel for API rate limits.

Lee-W commented 10 months ago

I am not sure whether I understand correctly, but according to the title, it looks like we want to run the GitHub tasks in parallel which seems to be there already. Closing this one. Please reopen it if I misunderstand it.