Manage picture set in directories endpoints

sylvanie85 commented 2 weeks ago

Context :

There are already endpoints to manage blob storage folders: one to create a directory, one to delete it, and another to get all directory names of a container. Since we now have a database, these folders are related to the picture set table.

Objective :

Therefore, we need to update these routes to create a picture set when a folder is created, delete it when the delete route is called, and retrieve the directories from the database instead of the blob storage because it's more efficient.

Delete use case diagram:

sequenceDiagram
    participant User
    participant FE
    participant BE
    participant DS

    User->>FE: Delete Folder
    rect rgb(191, 223, 255)
      FE->>BE: /delete-request
    end
    rect rgb(191, 223, 255)
          BE->>DS: Check if there are validated inferences or pictures from a batch import
note left of DS: Check for picture_seed entities linked to picture id<br> since a verified inference and a batch upload has those
    end
    alt picture_seed exist
        DS-->>BE: Validated inference status or pictures from batch import
        BE->>FE: True : Request user confirmation
        rect rgb(191, 223, 255)
            FE->>User: Ask to keep data for training
note left of FE : "Some of those pictures were validated or upload via the batch import.<br>Do you want us to keep them for training ? <br>If yes, your folder will be deleted but we'll keep the validated pictures.<br>If no, everything will be deleted and unrecoverable.
        end
        alt No
            User ->>FE: delete all
            rect rgb(191, 223, 255)
                FE-->BE:  /delete-permanently
            end
            BE->>DS: delete picture_set
        else YES
            User ->>FE: Keep them
            rect rgb(191, 223, 255)
                FE-->BE: /delete-with-archive
            end
            rect rgb(191, 223, 255)
                BE->>DS: archive data for validated inferences and batch import in picture_set
                    note left of DS: "Pictures are moved in different container <br> DB entities updated" 
                BE->>DS: delete picture_set
                   note left of DS: "Folder and all the files left are deleted, <br>related pictures, inference are deleted." 
            end
       else CANCEL
            User ->>FE: cancel : nothing happens
       end
    else no picture_seed exist
        DS-->>BE: No pictures with validated inference status or from batch import
        BE-->>FE: False: confirmation to delete the folder
        rect rgb(191, 223, 255)
            FE->>User: Ask confirmation
note left of FE : "Are you sure ? Everything in this folder will be deleted and unrecoverable"
        end
        alt Yes
            User ->>FE: delete all
            rect rgb(191, 223, 255)
                FE-->BE:  /delete-permanently
            end
            BE->>DS: delete picture_set
        else CANCEL
            User ->>FE: cancel : nothing happens

        end
    end

Acceptance criteria :

[x] picture_set table has a name column
[x] https://github.com/ai-cfia/nachet-datastore/issues/46
[x] https://github.com/ai-cfia/nachet-datastore/issues/50
[x] /create-dir route also add a picture set in the database
[x] /dir route retrieves directories names from the database instead of from the blob storage
[ ] https://github.com/ai-cfia/nachet-datastore/issues/63
[ ] Retrieve pictures and inferences with folders
[ ] /delete-request route checks for validated pictures in the folder
[ ] /delete-permanently route delete all pictures, picture set, blobs using datastore function
[ ] /delete-with-archive route move verified pictures and delete others using datastore function
[ ] update frontend to ask confirmation and to use the new routes

Francois-Werbrouck commented 3 days ago

I love the new look of this issue. tagging @ChromaticPanic for him to take a look at what will be needed for the FE 👍🏻

ChromaticPanic commented 3 days ago

Looks good, we should probably deal with Retrieve first. Since that's more useful to the user for processing images and providing feedback. Create -> Retrieve -> Update -> Delete . Not sure if we already have all the back end for retrieving pictures from a folder, retrieving inferences for a picture, retrieving feedback for a picture.

sylvanie85 commented 2 days ago

Not sure if we already have all the back end for retrieving pictures from a folder, retrieving inferences for a picture, retrieving feedback for a picture.

Currently, the backend can only retrieve the directory ID and name, as well as the number of images uploaded to it. This could be a new issue on datastore to create functions to retrieve images and their inference with the directory.

Francois-Werbrouck commented 2 days ago

This could be a new issue on datastore to create functions to retrieve images and their inference with the directory. It would also be a low effort issue for the datastore in my opinion. Most of the function to get the objects exists, we need to create a high level function to exactly output the right json.

Would we want to load all the inferences of a directory and store them in the cache of the BE which can then send them over depending on the user's requests without needing to operate a transaction with the DB each time a user access a picture within it's folder?

ai-cfia / nachet-backend