search

astronomer / ask-astro

An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
https://ask.astronomer.io/
Apache License 2.0
192 stars 47 forks source link

Upsert does not remove successfully upserted documents #174

Closed mpgreg closed 10 months ago

mpgreg commented 10 months ago

https://github.com/astronomer/ask-astro/blob/515f3386c4eac8aa4ddcdc3ad12c46b52e4aad8a/airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py#L328C9-L340C1

If no errors occur in upsert this will never be called. Need to move this to its own function and call it after rollback and also at line 405 if there are no errors.

mpgreg commented 10 months ago

For suggested fix https://github.com/mpgreg/ask-astro/blob/ed65a354013b5ce2170f98448bd510ed3c4201be/airflow/include/utils/weaviate/hooks/weaviate.py#L350

https://github.com/mpgreg/ask-astro/blob/ed65a354013b5ce2170f98448bd510ed3c4201be/airflow/include/utils/weaviate/hooks/weaviate.py#L301

https://github.com/mpgreg/ask-astro/blob/ed65a354013b5ce2170f98448bd510ed3c4201be/airflow/include/utils/weaviate/hooks/weaviate.py#L464-L489

sunank200 commented 10 months ago

@mpgreg Isn't this section doing the removing the upserted documents: https://github.com/astronomer/ask-astro/blob/main/airflow/include/tasks/extract/utils/weaviate/ask_astro_weaviate_hook.py#L328?