Challenge: demonstrate the "streaming" of a download into a local file - Githubissues

MellonScholarlyCommunication / css-suite

css-functionality

2 stars 0 forks source link

Challenge: demonstrate the "streaming" of a download into a local file #18

Open phochste opened 2 years ago

phochste commented 2 years ago

Context: research data projects would like to put Solid on top of a collection of big files (range 100 GB - 1 TB). The need to know if a command line tool exists to download these files (without crashing the client or solid pod).

Is this supported by the current tools (css-suite + CSS >=4.0):

CSS provides access to a mount which contains a huge file (e.g. the 500 GB "file of evil")
An ACL protects the file with read-only access to a WebID
Css-suite tools are used to authenticate with the WebID and download the "file of evil" to a local directory

Requirements:

The memory footprint of CSS and css-suite should stay stable, the only concern should be the disk/network speed
[ ] is this possible with the current tools?

phochste commented 2 years ago

Not streaming at the moment aparently