Connectors Remain Stuck in 'Deleting' State; Unable to Remove from UI

[quareth]

After scheduling several connectors for deletion, they became stuck in the "Deleting" state. Subsequently, all indexing processes began to fail. Despite waiting for an extended period with no changes, I rebooted the server and rebuilt the Docker image, which resolved the indexing issues. I can now delete newly indexed connectors without problems. However, the older connectors remain stuck in the "Deleting" state. I'm uncertain whether this is an issue with the UI or if these connectors are still present in the database. Is there a way to remove these stuck connectors directly from the UI?

All of them web connectors.

[shametim]

maybe the 'background' container is no longer running?

[quareth]

Background container looks fine. When I start Run Complete Re-indexing, Connectors start to do indexing again and looks like able to do indexing. However, their state never changes from "Deleting", which UI not allowing new Schedule for Deletion because it is currently Deleting. But this Delete state never changes and Connector never deleted.

[zarlor]

Just wanted to add (having posted this on the Discord without checking here, sorry) that I've been running into the same issue and the background container seems to otherwise work fine. In fact adding a new file connector, while slow, does seem to get indexed eventually but I think the deleting connectors cause some major delays if one of them seems to get stuck or the like

[zarlor]

Potential duplicate, I might add, of https://github.com/danswer-ai/danswer/issues/1202 and https://github.com/danswer-ai/danswer/issues/1275, so it seems to be a consistent problem that others are running into as well.

[acidgeneration]

Same issue with google drive connector, cant delete it

[quareth]

If for some reason the Document set sync cannot be completed, it cannot be removed manually later. Mine remained in sync due to disk fullness and this state does not change even if I delete the Connector after solving the disk problem. I am writing here thinking that the reason for this problem may be the same.

[Weves]

hey @acidgeneration / @quareth we put in a couple improvements to the syncing reliability. If you pull in the latest do you still run into this problem?

[mmanolakeris]

Running with the latest and still having this issue with both of the Confluence connectors

Edit: Background container logs in docker also seem to be fine, apart from the consistent spam of:

Document set '2' is already syncing. Skipping.

2024-05-24 21:19:14 [2024-05-24 19:19:13,982: INFO/MainProcess] Task check_for_document_sets_sync_task[de75cce7-bb1f-49e7-a2c7-9322685fb531] succeeded in 0.013694385008420795s: None 2024-05-24 21:19:14 05/24/2024 07:19:13 PM celery.py 195 : Document set '2' is already syncing. Skipping. 2024-05-24 21:19:18 [2024-05-24 19:19:18,056: INFO/MainProcess] Scheduler: Sending due task check-for-document-set-sync (check_for_document_sets_sync_task) 2024-05-24 21:19:19 [2024-05-24 19:19:18,985: INFO/MainProcess] Task check_for_document_sets_sync_task[77149f64-d292-4676-ae97-e9efaba4ab6c] received 2024-05-24 21:19:19 [2024-05-24 19:19:18,988: INFO/MainProcess] Document set '2' is already syncing. Skipping. 2024-05-24 21:19:19 [2024-05-24 19:19:18,998: INFO/MainProcess] Task check_for_document_sets_sync_task[77149f64-d292-4676-ae97-e9efaba4ab6c] succeeded in 0.012558657996123657s: None 2024-05-24 21:19:19 05/24/2024 07:19:18 PM celery.py 195 : Document set '2' is already syncing. Skipping.

The document set was unrelated to these connectors that are being deleted.

Edit 2: Both connectors were, however, connected to another document set which got deleted when I triggered the connectors to delete. After rebuilding the image I was able to remove one of them, but now the remaining connector throws this:

background-1 | 05/24/2024 07:41:51 PM connector_deletion.py 119 : Cleaning up Document Sets for CC Pair with ID: '13' background-1 | 05/24/2024 07:41:51 PM celery.py 89 : Failed to run connector_deletion due to Cannot delete CC pair while it is attached to a document set that is syncing. Please wait for the document set to finish syncing, and then try again. background-1 | Traceback (most recent call last): background-1 | File "/app/danswer/background/celery/celery.py", line 83, in cleanup_connector_credential_pair_task background-1 | return delete_connector_credential_pair( background-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ background-1 | File "/app/danswer/background/connector_deletion.py", line 183, in delete_connector_credential_pair background-1 | cleanup_synced_entities(cc_pair, db_session) background-1 | File "/app/danswer/background/connector_deletion.py", line 121, in cleanup_synced_entities background-1 | mark_cc_pairdocument_set_relationships_to_be_deletedno_commit( background-1 | File "/app/danswer/db/document_set.py", line 294, in mark_cc_pairdocument_set_relationships_to_be_deletedno_commit background-1 | raise ValueError( background-1 | ValueError: Cannot delete CC pair while it is attached to a document set that is syncing. Please wait for the document set to finish syncing, and then try again.

I also tried navigating to: http://localhost:3005/admin/documents/sets/13 but I get:

[quareth]

hey @acidgeneration / @quareth we put in a couple improvements to the syncing reliability. If you pull in the latest do you still run into this problem?

I've pulled latest today and tried for few documents and managed to Sync all of them. However it needs to be tested for a while longer to say anything definitive since problem occurs randomly. Other than that the previously stuck processes are still stuck and there is no way to delete them in UI. Some are stuck in Connector set deleting state, some are stuck in Sync state.

Thank you guys for the effort by the way. I appreciate.

Edit 12.06.2024: Since the update I've uploaded plenty of documents and created many connector, document sets and bots. Never faced with stuck issue since last update. So it seems the latest improvements solved the problem.

[acidgeneration]

Hi there, I'll try update and check it later, right now I have got banned my openai API key((

[XxDzonaxX]

I'm still having the same problem as you guys. I cannot delete a conector; it is stuck on deleting. Any solutions to get rid of it?

[shlomidvir]

I've managed to delete them by going to the connectors interface and smashing delete a few times.