dust-tt / dust

Amplify your team's potential with customizable and secure AI assistants.
https://dust.tt
MIT License
926 stars 103 forks source link

[MS connector] Garbage collection #5751

Closed tdraier closed 1 month ago

tdraier commented 2 months ago

Implement garbage collection : remove indexed resources that do not exist anymore

philipperolet commented 1 month ago

codependency with #6290

philipperolet commented 1 month ago

Theoretically, garbage collection is not needed since deletions are picked up in incremental syncs In practice, a sanity check that each node is still live would give us a stronger guarantee:

Add the garbage collect workflow to monitoring

philipperolet commented 1 month ago

Note: while garbage collection flow is not theoretically required, deletion code is, and should be written when folders are deselected from add/remove resources