Handle concurrent uploads using a counter and locking mechnaism.
The "upload_availablespace" key is set in Redis when Hue boots up. It calculates the actual available /tmp space and stores it. Whenever a new file upload is started, a new key is created in the format "upload" and stores the upload file size in Redis. Also, a second key in the format "upload__timestamp" (stores the timestamp) is created, and we reduce the "upload_availablespace" value by the new upload file size. When the upload completes, the "upload" and "upload__timestamp" keys are deleted, and the reserved space is given back to "upload_available_space".
Assume we have 1GB free space:
upload_available_space = 1GB
When file1 (100MB) upload is triggered:
upload_<uuid1> = 100MB
upload_<uuid1>_timestamp = <timestamp>
upload_available_space = 1GB - 100MB = 900MB
When file2 (100MB) upload is triggered:
upload_<uuid2> = 100MB
upload_<uuid2>_timestamp = <timestamp>
upload_available_space = 900MB - 100MB = 800MB
When the uploads are complete, the keys (upload_<uuid1>, upload_<uuid1>_timestamp) and (upload_<uuid2>, upload_<uuid2>_timestamp) are deleted, and upload_available_space is updated back:
In case of failed uploads, the same process is repeated for each retry made by the user. However, we would have leftover upload_ keys for each retry since the keys will only be deleted on a successful upload. So we run a periodic job cleanup_stale_uploads to clean up these keys. This job runs every CLEANUP_STALE_UPLOADS_IN_REDIS_PERIODIC_INTERVAL minutes and deletes upload_* keys if the timestamp difference is greater than 60 minutes.
Moved task server configs to TASK_SERVER_V2.
Moved reserve, release upload space methods to filebrowser utils. New configs for periodic scheduling under task server
Display max_file_upload_size_limit on upload modal. Parse redis broker url from configs.
Changed task server configuration parsing from awk to grep. Set autorestart to false in redis and celery template.
Show task_server tab in Admin Server, based on task_server_v2 configs
Moving Uploaded chunk log message to Debug from Info
Setting celery default log level to info
Pull timezone from hue.ini. /tmp_cleaner job checks the timestamp of each file and deletes it based on timedelta=60mins
What changes were proposed in this pull request?
Handle concurrent uploads using a counter and locking mechnaism. The "upload_availablespace" key is set in Redis when Hue boots up. It calculates the actual available /tmp space and stores it. Whenever a new file upload is started, a new key is created in the format "upload" and stores the upload file size in Redis. Also, a second key in the format "upload__timestamp" (stores the timestamp) is created, and we reduce the "upload_availablespace" value by the new upload file size. When the upload completes, the "upload" and "upload__timestamp" keys are deleted, and the reserved space is given back to "upload_available_space".
Assume we have 1GB free space:
upload_available_space = 1GBWhen file1 (100MB) upload is triggered:
upload_<uuid1> = 100MBupload_<uuid1>_timestamp = <timestamp>upload_available_space = 1GB - 100MB = 900MBWhen file2 (100MB) upload is triggered:
upload_<uuid2> = 100MBupload_<uuid2>_timestamp = <timestamp>upload_available_space = 900MB - 100MB = 800MBWhen the uploads are complete, the keys (
upload_<uuid1>,upload_<uuid1>_timestamp) and (upload_<uuid2>,upload_<uuid2>_timestamp) are deleted, andupload_available_spaceis updated back:upload_available_space = 800MB + 100MB + 100MB = 1GBIn case of failed uploads, the same process is repeated for each retry made by the user. However, we would have leftover
upload_keys for each retry since the keys will only be deleted on a successful upload. So we run a periodic jobcleanup_stale_uploadsto clean up these keys. This job runs everyCLEANUP_STALE_UPLOADS_IN_REDIS_PERIODIC_INTERVALminutes and deletesupload_*keys if the timestamp difference is greater than 60 minutes.Moved task server configs to TASK_SERVER_V2.
Moved reserve, release upload space methods to filebrowser utils. New configs for periodic scheduling under task server
Display max_file_upload_size_limit on upload modal. Parse redis broker url from configs.
Changed task server configuration parsing from awk to grep. Set autorestart to false in redis and celery template.
Show task_server tab in Admin Server, based on task_server_v2 configs
Moving Uploaded chunk log message to Debug from Info
Setting celery default log level to info
Pull timezone from hue.ini. /tmp_cleaner job checks the timestamp of each file and deletes it based on timedelta=60mins
How was this patch tested?
Tested on local machine and using docker builds.
Please review Hue Contributing Guide before opening a pull request.