search

datalad / datalad-ria

Adds functionality for RIA stores to DataLad
http://datalad.org
Other
0 stars 1 forks source link

RIA-specific URL handlers #105

Open mih opened 9 months ago

mih commented 9 months ago

A number of issues have been determined that prevent the direct use of the UrlOperations provided by datalad-next:

One way to side-step this issue is to have a RIA-specific collection of handlers. We could use the standard URL handler selection mechanism (via config) to specifically choose these (bases on the standard ria+ prefix).

Each of these to-be-implemented handlers could subclass the analog from datalad-next, and amend the implementation with a suitable API and functionality.

The downside of such a move would be that any protocol would need a dedicated ria-variant to become compatible.

christian-monch commented 9 months ago

With regard to #103, I would opt for dropping it. An explanation for this verdict is provided in this comment: https://github.com/datalad/datalad-ria/issues/103#issuecomment-1746383180

christian-monch commented 9 months ago

W.r.t to #102, I would currently opt for an upload_atomic-operation or an additional parameter to the upload-operation in order to keep the number of UrlOperations-classes low and the usage patterns orthogonal. Please find a more detailed comment here: https://github.com/datalad/datalad-ria/issues/102#issuecomment-1746508183

christian-monch commented 9 months ago

Commented on https://github.com/datalad/datalad-next/issues/454 here: https://github.com/datalad/datalad-next/issues/454#issuecomment-1746579621. My current point of view is that we should not support a force-mode, but let the user take action, if UrlOperations report an error, if a resource would be overridden. (I generally like to reduce the amount of "intelligent" and silent error handling in lower-level components.)

christian-monch commented 3 months ago

With #596 close to the finish line and an improved understanding of the usage scenarios, I would like to describe the current state.

The datalad-ria code I am working on () has dedicated handlers for file, for ssh, and for http, that implement the basic annex remote operations, i.e. INITREMOTE, PREPARE, TRANSFER RETRIEVE, TRANSFER STORE, CHECKPRESENT and REMOVE.

Implementing an ensure_writable-method in the file-handler and the ssh-handler is easy. The ssh-handler implementation is done, it implements ensure_writable for OSX and Linux. The file-handler will just use os.chmod().

Up until now, HTTP was only used to read from an annex remote, i.e. the HTTP-handler would only implement TRANSFER RETRIEVE. If we also want to implement TRANSFER STORE via HTTP, the server must be configured accordingly. This could include the ability to ensure that a given directory is writeable (TRANSFER STORE over HTTP is already a challenge because of the atomicity-requirement w.r.t. CHECKPRESENT, see #102).

christian-monch commented 2 months ago

I opened PR Atomic UrlOperations.upload for ssh- and file-URLs in datalad-next which addresses the atomicity-requirement for ssh: and file:-URLs