BUG: race condition in add groundtruth and deleting a dataset

[ekorman]

valor version checks

• [X] I have confirmed this bug exists on the latest version of valor.

Reproducible Example

import time

dset = Dataset.create("VizDrone")
def create():
    print("started create\n")
    # for i, gt in enumerate(tqdm(gts)):
    dset.add_groundtruths(gts)
    #     dset.add_groundtruth(gt)
    print("finished create\n")

def delete():
    print("sleeping inside delete\n")
    time.sleep(1)
    print("done sleeping")
    client.get_dataset("VizDrone").delete()
    print("dataset delete\n")

thread1 = threading.Thread(target=create)
thread2 = threading.Thread(target=delete)

thread1.start()
thread2.start()

thread1.join()
thread2.join()

this has given two different errors at times

1. looks like from the delete call

ClientException: {"name": "IntegrityError", "detail": "(psycopg2.errors.ForeignKeyViolation) update or delete on table \"annotation\" violates foreign key constraint \"groundtruth_annotation_id_fkey\" on table \"groundtruth\"\nDETAIL:  Key (id)=(73597) is still referenced from table \"groundtruth\".\n\n[SQL: DELETE FROM annotation USING (SELECT annotation.id AS id \nFROM annotation JOIN datum ON datum.id = annotation.datum_id \nWHERE datum.dataset_id = %(dataset_id_1)s) AS anon_1 WHERE annotation.id = anon_1.id RETURNING annotation.id]\n[parameters: {'dataset_id_1': 2}]\n(Background on this error at: https://sqlalche.me/e/20/gkpj)", "timestamp": 1713911923.499323}

2. looks like from the add groundtruth call

valor.exceptions.ClientException: {"name": "ObjectDeletedError", "detail": "Instance '<Annotation at 0x16841d8d0>' has been deleted, or its row is otherwise not present.", "timestamp": 1713932291.931047}

### Issue Description

There can be a race condition between requests to add groundtruth to a dataset and requests to delete it.

### Expected Behavior

At minimum, more useful error messages to the user but ideally we need better state control and blocking in the backend.

[ntlind]

Not super high prio since you can always delete twice.