observingClouds / slkspec

fsspec filesystem for stronglink tape archive
5 stars 2 forks source link

Temporary Files #20

Open wachsylon opened 1 year ago

wachsylon commented 1 year ago

Some feature requests:

observingClouds commented 1 year ago

Is there a way to resume an old retrieval? Could we add that?

slkspec only downloads the files that are not yet or not any longer in the temporary directory. So in this way it resumes already. What were you thinking about?

observingClouds commented 1 year ago

what happens if I delete the temporary file before retrieval started? It seems to me like the process hangs...

Can you give a minimal example?

neumannd commented 1 year ago

what happens if I delete the temporary file before retrieval started? It seems to me like the process hangs...

Probably that's an issue of slk retrieve? What should happen when the file is delete? Probably an error should be thrown as soon as slk retrieve realizes that the file is gone?

neumannd commented 1 year ago

Is there a way to resume an old retrieval? Could we add that?

@wachsylon Do you mean that what you asked in our meeting 2 hours ago? => [...] that slk retrieve resumte to retrieve a partly retrieved files? Not possible and probably won't be possible in near future.

But this resume-thing is maybe a good reason why people should not archive too large files (e.g. 2 TB). If the retrieval dies, you need to start from the beginning. Instead, if a user has 200 files for 10 GB, the resume needs only to retrieve the missing files and not the whole 2 TB.

wachsylon commented 1 year ago

But this resume-thing is maybe a good reason why people should not archive too large files

I agree.

Probably an error should be thrown as soon as slk retrieve realizes that the file is gone?

At least sth should happen... :)