SOLID POD: Large file management

[cdawei]

The current approach to manage a large file in POD consists of

• A dedicated container / directory with data chunks (and their ACL files)
• A Turtle file describing the large file size in bytes and the URLs of the corresponding data chunks
• An ACL file describing the permissions to access the large file

Other options to manage large files including encoding the content into strings and store them in a Turtle file, or using nextcloud-solid plugin to potentially store them on a nextcloud instance.

[cdawei]

Explored the approach that encodes the chunks of content into strings (base64 encoding) and store them in a single Turtle file, the observations are:

• Uploading worked with either SPARQL INSERT queries or N3 patches
• It is computational expensive to run SPARQL INSERT queries or apply N3 patches to append data chunks, it used 70%-100% CPU cycles during the uploading.
• The more chunks to send, the slower the uploading progress becomes.
• The resulting Turtle file size is about 1.3X of the original binary file (tested using a 46MB binary file).
• Retrieving the resulting Turtle file worked.
• When uploading, solid server reported error when data chunk is equal or larger than 512 KB:

  {"name":"BadRequestHttpError","message":"Maximum call stack size exceeded","statusCode":400,"errorCode":"H400","details":{}}