datalad / datalad-ria

Adds functionality for RIA stores to DataLad
http://datalad.org
Other
0 stars 1 forks source link

Abandon IO abstractions, switch to a multi-flavor ORA remote #99

Open mih opened 1 year ago

mih commented 1 year ago

Combining the information in #80 and #98 it seems advisable to abandon the IOBase and friends framework.

The current ORA implementation assumes that there is a finite set of "IO operations" that can be represented by all "protocols". This has worked well for SSH and local access (both actually work with a file system). It has not scaled to HTTP requests. it is unlikely to work for general object stores and other possibilities to store files.

I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.

christian-monch commented 1 year ago

I believe it would be more meaningful to have separate implementations for special remote operations (e.g. STORE, RETRIEVE, ...), rather than have one implementation that call special implementations for a fixed procedure of "filesystem operations". I think the resulting code would be leaner, more readable, and yield a more efficient operation.

I agree with that. Especially if we take into account the support of other storage backends, e.g. s3-buckets. Different storage systems might best be supported by a combination of uncurl-remotes and specific implementations.

What I am currently wondering about are the backward compatibility requirements:

  1. Existing stores must remain accessible with the new implementation and the schemes: ria+file, ria+ssh, and ria+http.
  2. New schemes, e.g. ria+s3, ria+webdav, ria+googledrive, etc. can be supported by any implementation with no necessity for backward compatibility.