search

OdyseeTeam / odysee-api

API server for Odysee
https://odysee.com
MIT License
228 stars 49 forks source link

Consider to add file upload expiration time to clean up long standing unfinished upload #389

Closed rnd closed 2 years ago

rnd commented 2 years ago

With the addition of resumable upload on publishing endpoint, currently it's possible for the server to run out of storage space because we have too many unfinished file upload as TUS does not handle file upload expiration. see: https://github.com/tus/tusd/issues/435

This ticket is created to discuss the possibility to solve this issue. There are 2 ways that I think can be implemented:

  1. Spawn a Go routine that will check last updated time for each uploaded files and remove it if there is no activity within a duration that we agreed upon.
  2. Same as above, but instead of having this clean up process on odysee-api server -- we can technically implement this as k8s cron job, that way we can deploy this job on each server that host odysee-api and we can configure the script on the fly without needing to restart odysee-api.

As for the file expiration time, I might need input from @tzarebczan or @andybeletsky as this might affect user experience and require changes / docs on the frontend too

anbsky commented 2 years ago

I'm in favor of #2, we are running several service instances, running cleanup routine on each of them independently would probably create issues. 48 hours expiration time should be enough.

rnd commented 2 years ago

I agree, it's more reasonable to take the 2nd approach then. Not sure how we proceed with this as i don't have access to the cluster. Should i wrote the bash script to clean up the files and you put it on k8s cron manifest after?

tzarebczan commented 2 years ago

Do we think this is blocking shipping, or can we go live on UI side and see how things go? I assume we can always cleanup manually in the interim if needed.

rnd commented 2 years ago

I don't think its a blocker. And you're right -- in case we run out of storage space because of these files, we can remove it manually until we have the cron deployed.

tzarebczan commented 2 years ago

Found a bug or two on the UI side, so we'll get it shipped tomorrow!

rnd commented 2 years ago

closing this as Andrey already deployed it.