Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer

https://ask.astronomer.io/

Apache License 2.0

196 stars 47 forks source link

Create TaskFlow API to use WeaviateHook and use taskflow API for ask-astro-load.py #132

Closed sunank200 closed 12 months ago

sunank200 commented 1 year ago

I have taken airflow Docs ingestion for my implementation. Hence commits from Michael.
Create AskAstroWeaviateHook which inherits from OSS WeaviateHook.
Add get_schema, check_schema, create_schema, handle_upsert_rollback and ingest_data
Use Taskflow API for ask-astro-load.py.

closes: #134

cloudflare-workers-and-pages[bot] commented 1 year ago

Deploying with Cloudflare Pages

Latest commit:	`93ea3d8`
Status:	✅ Deploy successful!
Preview URL:	https://3db4b985.ask-astro.pages.dev
Branch Preview URL:	https://oss-weaviate-integration.ask-astro.pages.dev

View logs

sunank200 commented 12 months ago

Some logic errors but mostly I'm concerned about readablity and generalizability. For the sake of the reference implementation we want to be very clear about DAG/task code and what should (eventually) go into the hook.

@mpgreg I have fixed most of your comments. Please check. Also just to be clear. This hook is not a drop-in replacement for LLM providers. It requires work contextual change to be added for LLM provider