search

huggingface / dataset-viewer

Backend that powers the dataset viewer on Hugging Face dataset pages through a public API.
https://huggingface.co/docs/dataset-viewer
Apache License 2.0
680 stars 76 forks source link

Create a cron job to clean the dangling assets and cached assets #1284

Open severo opened 1 year ago

severo commented 1 year ago

Related to https://github.com/huggingface/datasets-server/issues/1122

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

severo commented 1 year ago

keep open

severo commented 7 months ago

As we now use signed URLs, only the assets referenced by a cache entry will be accessible. The other ones, if any, will "just" waste space on S3

severo commented 7 months ago

Also: the files on S3 should now be deleted when we delete a dataset (see https://github.com/huggingface/datasets-server/blob/c22f563f350a36148a1deaaf12531d4da991a677/libs/libcommon/src/libcommon/orchestrator.py#L678)

severo commented 7 months ago

Note: as of today, the production S3 bucket contains 3TB of assets. It would be interesting to know which part is not needed anymore.

Also: /cached-assets directory could have a TTL of 1 day, or less, since the assets are generated on demand.