Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

colossus: fully qualified path for temp folder #4977

Closed mnaamani closed 8 months ago

mnaamani commented 9 months ago

Background

The temp folder path where active uploads to the node and objects being fetched by sync worker from other nodes is not configurable, but is always created as a sub folder under the uploads directory (long term storage of data objects).

Some ideas for scaling storage capacity of the node will require separation of temp files from the long term storage path. If we decide to use object storage (s3 compatible) then its not possible to store incomplete download files there.

Solution

Add new parameter to server command to specify a fully qualified path for the temp folder to use.

Challenges ?

Typically the path to the uploads folder will be on a volume with considerable capacity, and this volume might be mounted from a network storage cluster. It might be more suitable for the temp folder to be a path on the local host file system with not as much disk capacity. Should the node actively try to manage the number of downloads and uploads at a rate that takes the temp folder size into consideration?