Airflow incremental ingest failed for 2 DAGs due to an error from Weaviate

[davidgxue]

Describe the bug Airflow periodic incremental ingest failed for 2 DAGs, one for astro docs and one for astro providers Error is from weaviate

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
    result = execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/weaviate/operators/weaviate.py", line 179, in execute
    insertion_errors = self.hook.create_or_replace_document_objects(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/weaviate/hooks/weaviate.py", line 1054, in create_or_replace_document_objects
    batch_delete_error = self._delete_all_documents_objects(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/weaviate/hooks/weaviate.py", line 897, in _delete_all_documents_objects
    document_objects = batch.delete_objects(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/weaviate/batch/crud_batch.py", line 1368, in delete_objects
    res = _decode_json_response_dict(response, "Delete in batch")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/weaviate/util.py", line 798, in _decode_json_response_dict
    raise UnexpectedStatusCodeException(location, response)
weaviate.exceptions.UnexpectedStatusCodeException: Delete in batch! Unexpected status code: 500, with response body: {'error': [{'message': 'batch delete objects: cannot find objects: find matching doc ids in shard "isecbk6jb7qt": could not get uuid from doc_id=14990'}]}.

[davidgxue]

issue no longer observed in 0.3.0 release. Closing issue for now