Disk space is not reclaimed from docker volume after files are deleted

[prohtex]

Describe the bug

A clear and concise description of what the bug is.

Steps to reproduce

1. Upload large files
2. Delete files

Expected behavior

Expected behavior is that docker image disk space usage is reduced after files are removed from "Deleted files" using "Empty trash bin"

Actual behavior

Docker image disk usage remains the same even after files are removed

Setup

Deploy fresh Ubuntu Server VM, deploy OCIS with ocis_wopi deployment

Below is the result of uploading 28GB of files, then deleting 28GB of files.

sudo docker system df -v
Local Volumes space usage:
ocis_wopi_ocis-data                                                2         28.87GB

[prohtex]

Hi all, in my testing, I've noticed this only occurs in the WOPI deployment docker container version. The Darwin binary deployment I'm running reclaims space fine. I don't know why the docker volumes are not having the free space reclaimed after deleting files, but I noticed that when files are added after deletion, they do not utilize any of the space from the previously deleted files.

For example, upload a 10GB file, delete it with OCIS web, empty trash. Then upload another 5GB file. The total space utilized on the Docker volume will be 15GB, not 5GB as expected, or even 10GB if the files were "writing over" previously deleted ones.

[dragonchaser]

Can reproduce locally without docker.

[prohtex]

Can reproduce locally without docker.

Great. Hopefully when the solution is found, there will be a convenient way to reclaim that space on existing deployments.

[dragonchaser]

@prohtex There is a bug in the routine that purges the trashcan, atm only the first level is removed. Meaning: File deletion works fine, deleting folders would require to recursively walk over the structure, which is not happening atm. Working on it.

[dragonchaser]

Can reproduce locally without docker.

Great. Hopefully when the solution is found, there will be a convenient way to reclaim that space on existing deployments.

As soon as https://github.com/cs3org/reva/pull/4533 is merged and bumped into ocis, you should be able to clean it up using the binary of storage users by running:

# single binary installations:
$> ocis storage-users trash-bin purge-expired <space-id>
# microservice installations:
$> storage-users trash-bin purge-expired <space-id>

[profiteroles1]

Can reproduce locally without docker.

Great. Hopefully when the solution is found, there will be a convenient way to reclaim that space on existing deployments.

As soon as cs3org/reva#4533 is merged and bumped into ocis, you should be able to clean it up using the binary of storage users by running:

# single binary installations:
$> ocis storage-users trash-bin purge-expired <space-id>
# microservice installations:
$> storage-users trash-bin purge-expired <space-id>

Thank you! Please forgive me-I am new to Docker and oCIS. I tried this:

sudo docker exec -it <container> ocis storage-users trash-bin purge-expired

Does this command with no space specified purge the trash on all spaces? If I need to specify a space-id, how do I get a list of space id's? I do not see this in the command reference.

[prohtex]

Can reproduce locally without docker.

Great. Hopefully when the solution is found, there will be a convenient way to reclaim that space on existing deployments.

As soon as cs3org/reva#4533 is merged and bumped into ocis, you should be able to clean it up using the binary of storage users by running:

# single binary installations:
$> ocis storage-users trash-bin purge-expired <space-id>
# microservice installations:
$> storage-users trash-bin purge-expired <space-id>

Thank you @dragonchaser! How does one determine the id of a given space?

[prohtex]

Is it something like this?

curl http://localhost/graph/v1.0/me/drives

For me this returns "Permanently Moved." I'd really appreciate the help-we are trying to reclaim several hundred gigs of junk that was not deleted properly. @micbar ? Thank you!!

[micbar]

Please try curl https://localhost/graph/v1.0/me/drives. There is no http:// protocol available.

[prohtex]

Please try curl https://localhost/graph/v1.0/me/drives. There is no http:// protocol available.

Thanks for the answer! Unfortunately I am getting 404. I am running WOPI deployment in Docker. Do I need to do something else to enable the API? Or is there some other means of determining the Space ID, for example with the command line? I can't find anything in the command reference.

vm1:~$ wget --no-check-certificate https://localhost/graph/v1.0/me/drives?$filter=driveType+eq+project
--2024-02-26 22:47:22--  https://localhost/graph/v1.0/me/drives?=driveType+eq+project
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:443... connected.
WARNING: cannot verify localhost's certificate, issued by ‘CN=TRAEFIK DEFAULT CERT’:
  Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
    requested host name ‘localhost’.
HTTP request sent, awaiting response... 404 Not Found
2024-02-26 22:47:22 ERROR 404: Not Found.

Update: I was able to pull what I believe was the Space ID off the URL for the space from oCIS Web, but I am still not seeing space reclaimed...

sudo docker exec -it f526771e3270 ocis storage-users trash-bin purge-expired 50063552-5aa2-4820-af85-ed0402d3e5af%24b6b60561-e99d-42d1-a30b-49e8eb10535e%21b6b60561-e99d-42d1-a30b-49e8eb10535e

Some of the space I am trying to reclaim is also from spaces that were removed. I assume these files persisted somehow. I do not know how to get those Space ID's.

[prohtex]

Hi @dragonchaser @kulmann I'm very sorry for beating a dead horse here but I have a deployment with 200gb of dead files that I cannot for the life of me figure out how to reclaim. I'd also love to verify trash removal is now working properly following upgrading to RC5.

Can you help with some additional steps to purge these files? I've been trying to determine the Space ID's with the API but it seems to return a 404. Does running purge-expired without a Space ID specified do anything? According to documentation, STORAGE_USERS_PURGE_TRASH_BIN_USER_ID env var is used to specify user, but like Space ID, I am not sure how to determine this.

There's also some information here https://owncloud.dev/services/storage-users/ that seems to imply I should be authenticating the CLI command. I'm quite confused and would be so grateful for some pointers.

[micbar]

You should fix your curl statement. Make sure you have the right port.

on localhost, ocis uses https protocol on port 9200

[prohtex]

You should fix your curl statement. Make sure you have the right port.

on localhost, ocis uses https protocol on port 9200

Hi @micbar thank you so much for taking the time to respond. Unfortunately that is not working for me. Does the WOPI deployment include the API service or do I need to enable or install that another way?

user@vm1:~$ curl https://localhost:9200/graph/v1.0/me/drives
curl: (7) Failed to connect to localhost port 9200 after 0 ms: Connection refused

[leepfrog-ger]

As I needed to solve this myself, these instructions may be helpful for others. I've used that with a docker compose setup, so adjustments may be required for your use case.

1. obtain a valid bearer token from interacting with the OC web interface and taking the Bearer Token from the Browsers Dev Tools. These are only valid a view minutes, so you will need to immediately use it
2. You can get a list of spaces by running the following command (this can be from any system that can access the owncloud endpoints): curl https://replace.me/graph/v1.0/drives -H 'Authorization: Bearer <insert your bearer token> > output.json
3. Parse the part of the space IDs that you need: cat output.json | jq '.value[].id' | awk --field-separator '$' '{ print $2 }' | sed 's/.$//'
4. This should print the parts of the ID that you need to run docker compose exec ocis ocis storage-users trash-bin list <space id> or docker compose exec ocis ocis storage-users trash-bin purge-expired <space id>

[prohtex]

As I needed to solve this myself, these instructions may be helpful for others. I've used that with a docker compose setup, so adjustments may be required for your use case.

1. obtain a valid bearer token from interacting with the OC web interface and taking the Bearer Token from the Browsers Dev Tools. These are only valid a view minutes, so you will need to immediately use it
2. You can get a list of spaces by running the following command (this can be from any system that can access the owncloud endpoints): curl https://replace.me/graph/v1.0/drives -H 'Authorization: Bearer <insert your bearer token> > output.json
3. Parse the part of the space IDs that you need: cat output.json | jq '.value[].id' | awk --field-separator '$' '{ print $2 }' | sed 's/.$//'
4. This should print the parts of the ID that you need to run docker compose exec ocis ocis storage-users trash-bin list <space id> or docker compose exec ocis ocis storage-users trash-bin purge-expired <space id>

Thanks @leepfrog-ger for outlining these steps. You can also just pull the space ID off the URL for the space in oCIS Web. In my case, I still have not been able to reclaim anything. I believe the files I'm trying to purge were in a space that I removed, yet somehow, they persist somewhere. I guess it is time to start from scratch.

[prohtex]

@micbar quick question. Given that I have this docker volume with files I can't remove (in this case ocis_wopi_ocis-data), is there some way for me to reinitialize storage without re-deploying?

[micbar]

What do yo expect when you „reinitialize the storage“?

[prohtex]

What do yo expect when you „reinitialize the storage“?

I'm hoping to keep all users and settings but to wipe out the docker volume and recreate spaces and files.

Unless there is another command to run, I have a docker volume with hundreds of gigabytes of dead files. I already successfully ran the purge-expired command on all existing spaces and no space was reclaimed. I have 5gb of files in a volume taking up 200gb and can't for the life of me figure out how to reclaim the space.