search

iiasa / accelerator_service

Docker compose for services and jobs related to scenario explorer and data processing
0 stars 0 forks source link

Ability to create folder and move files in data repository #24

Closed Wrufesh closed 6 months ago

Wrufesh commented 6 months ago

Give warning while moving files. Warning should be about the file might be being used by a program.

Wrufesh commented 6 months ago

Moving and creation cannot be done in job outputs and cover image folder.

Wrufesh commented 6 months ago

Moving files across app buckets with different domain or physical location will not be supported for now.

Wrufesh commented 6 months ago

Copying files greater than 5GB is not supported by minio. Thus, file is copied into several chunks and later merged. Minio compose object api: https://min.io/docs/minio/linux/developers/python/API.html#compose_object

Which make use of S3's UploadPartCopy API can be used.

There can be several ComposeSource object used in compose object from same file with limit and offset to facilitate very large file copy.

Wrufesh commented 6 months ago

Revisit delete policy.