search

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
https://ask.astronomer.io/
Apache License 2.0
196 stars 47 forks source link

Setup nightly slack ingest #23

Closed mpgreg closed 1 year ago

mpgreg commented 1 year ago

Current bulk and incremental dags only ingest from slack archives. Need to get a slack token and setup nightly incremental ingest.

mpgreg commented 1 year ago

Also need to fix slack message links. Currently they point to an older URL format.

alex-astronomer commented 1 year ago

Has this DAG already been set up? Is it different than the nightly Slack DAG?

EDIT: Looks like I linked the incremental load DAG which was mentioned in OP

mpgreg commented 1 year ago

Yes this is closed by #50 https://github.com/astronomer/ask-astro/blob/main/airflow/dags/ingestion/ask-astro-load-slack.py