Recover Orphans from ObjectStore

[ArtificialOwl]

The idea is to improve the management of objects from ObjectStorage and helps administrator to manage orphans that might appears on a de-sync between the table filecache and the objects stored on S3.

The development of this feature can be split in 3 steps:

first step:

• [ ] an occ command to list Orphans (by comparing the Objects available on the Storage and the one available in the table filecache),
• [ ] an occ command to delete Orphan by its ex-fileId.

second step:

• [ ] add metadata (filename, owner, ...) about the current local file for each entries of the ObjectStore,
• [ ] manage metadata on file creation, on file rename, on ownership transfer,
• [ ] an occ command to get metadata from the ObjectStore about a file, from its [ex-]fileId

third step:

• [ ] an occ command to recover an Orphan by creating a fresh entry in the table filecache
• [ ] an occ command to fill metadata of objects from the ObjectStore without metadata, based on local information from filecache

side note: the occ command to list Orphans should be able to also compare filesize and fix the data from the table filecache.

[mrAceT]

This I've done in a stand alone program.. and I'm willing to share the days of testing, debugging, more testing etc I did to rescue/migrate my NextCloud instance with the community.. but I do have a few "conditions".

I see you have contributed to this project in the past and that you also seem to have knowledge of "the AWS S3" part, could you please give me an answer to:

March 24, 2022: AWS s3 as primary storage & redundancy https://help.nextcloud.com/t/aws-s3-as-primary-storage-redudancy/136202 In the current structure the table 'oc_filecache' is the Achilles heal of the NexCloud S3 structure. I have a master-slave setup to mitigate this, but In my opinion this problem would be easily fixed when a "S3 file" would NOT have the name "urn:oid:123456" but its actual file and folder name?

Then one could also manually remove stuff and perform a "files:scan --all" to clean the data etc etc. Then the S3 storage structure could perform much more the same as the "regular one". Now S3 needs to be used as a database with a relational database.. one can not exist without the other making them 100% dependent on each other AND much more vulnerable to trouble..

Oct 4, 2022: quota S3 storage wrong WITH Q&D FIX! https://github.com/nextcloud/server/issues/34422 This one already shared completely. There is a quota problem somewhere where sometimes "a bit flips" more or less bricking the S3 quota system. I've had this problem on and off for about a year and in "utter frustration" I've spent quit a bit of time mitigating this problem and I managed to "hack" a Q&D fix (I'll admit quite "D" ;) ). I'd very much appreciate it when someone like you would take a look at it (and hopefully create a real fix why that darn un-editable 2GB quota pops up)..

That stand alone program.. what it does for me:

• does basic checks and warns for risky oc_storages setups (there are so many things that can go wrong on migrating from local > S3..)
• it can remove old previews
• it checks for orphans in S3 compared to the oc_filecache
• it checks for orphans in oc_filecache compared to S3 (for migrating local > S3)
• it checks for newer files in oc_filecache compared to S3 (for migrating local > S3)
• it can alter local settings for migration to S3

As you might have guessed, I built it for migrating from local to S3. But when already migrated it can function as a "S3 sanity check"

[joshtrichards]

Related: #30824