Change storage provider

pietercolpaert commented 2 years ago

Actors/actresses

Two solid storage providers and an end-user

Preconditions

One storage provider stores nothing yet, another provider stores already quite some resources for an end-user.

Steps

The end-user indicates they wants to change storage providers
All data is copied towards the new storage provider
All references are updated to the new URLs
Data with the old storage provider is removed

Postconditions

All apps keep working on top of the new storage
Old storage provider is empty now

pietercolpaert commented 2 years ago

@dexagod is doing relevant work: can you elaborate on that one?

Dexagod commented 2 years ago

What

We created a toolset for Solid on the CLI and Node.js, mainly focused and tested for the Community Solid Server.

How

It provides a set of functions that mimick the linux commands for managing files/resources. Relevant here is the copy function. It is available through the CLI interface as

  node bin/solid.js [auth options] copy [copy options] <source> <destination>

enabling you to copy files

from local to remote data pod
from remote data pod to local
from remote data pod to same or other data pod (make sure you have adequate permissions in both locations)

In Node.js, you can use the same functionality as follows:

import { copyData } from "install/location''

let options = { ... }
await copyData(source, destination, options);

to copy files in the same manner.

Authenticated resources

The tool provides authentication options both for the CLI and for Node.js to work with authenticated resources

Node.js

All functions in Node.js take a fetch parameter in their options. This allows you to authenticate however you wish, and pass the authenticated fetch function. You can use the createAuthenticatedSession() function exposed by the lib that enables authentication using username, identity provider and password parameters, but this is currently only supported for the Community Solid Server v2.x.x For more general options, see the inrupt docs on Solid authentication for Node.js

CLI

The CLI interface allows you to authenticate using either a config file with user credentials (only currently for Community Solid Server v2.x.x), via a config file (again only CSS v2.x.x), or using an interactive browser window (should support all versions), that caches session information between calls where possible.

Dexagod commented 2 years ago

I still have an additional comment here:

Point 3: All references are updated to the new URLs -> does this mean that every resource should be forwarded? Because this seems like something that is a hell to maintain: e.g. you move / delete resources in their new location.

Would it make sense to only forward the root, so that discovery of resources is still maintained through the pod root?

In this case direct links e.g. shared photos could break if they use absolute URIs, but to catch these cases maybe this should be caught using a URL rewrite?

Straight up forwarding all resources seems a tad excessive

rubensworks commented 2 years ago

-> does this mean that every resource should be forwarded? Because this seems like something that is a hell to maintain: e.g. you move / delete resources in their new location.

I would say yes, all resources should be redirected. Could be achieved via some kind of moved.meta file, which could act similar to a UNIX softlink.

But this should not be guaranteed anymore IMO once another resources is placed over that location.

pheyvaer commented 2 years ago

If I understand this scenario correctly, does it mean that you are moving data from one pod to the other? With storage provider I first thought about the storage system that is used by CSS in the back-end.

sjoerdvangroning commented 2 years ago

I think it should be important to leave breadcrumbs in order to be able to find data after it has been moved. If we don't do that, links will break just as fast in the current web.

What do you think of the https://github.com/pdsinterop/solid-migrator?

pheyvaer commented 2 years ago

@sjoerdvangroning Why is that? From my understanding the URLs of all resources remain the same.

sjoerdvangroning commented 2 years ago

@sjoerdvangroning Why is that? From my understanding the URLs of all resources remain the same.

Maybe we are thinking of a different use case? Can you describe in more detail your use case and how URL can remain the same?

I am thinking of external links to POD/Data and when you change storage provider, the URL will change. (for example when you move your POD from solid.community to solidcommunity.net).

pheyvaer commented 2 years ago

I assumed that the user has control over the whole domain, so they can redirect the domain to the new storage provider. If the domain is tied to the storage provider then you would indeed have issues. I'm also wondering what @pietercolpaert means with step 3 actually.

pietercolpaert commented 2 years ago

In step 3 I was thinking about organizational hypermedia on top of LDP. E.g., if you use AS2 pagination, or shape trees, or TREE/LDES, that you update the link URLs to resources on the new server instead of on the old server, and then having to follow bread crumbs or instead of following a redirect.

I agree with @sjoerdvangroning that breadcrumbs should be provided. A redirect is I believe a specific implementation of such a breadcrumb.

sjoerdvangroning commented 2 years ago

We have given this some thought in our migrator project. We created a specification and a working implementation for moving data between storage providers. Being able to move your data is important for the SOLID eco-system. If servers would follow the metadata and give redirects, data can still be found and it could give clients time to update their links (honour a permanent redirect for example and update the links).

@pietercolpaert : Would really be honoured with feedback on our ideas: https://pdsinterop.org//solid-link-metadata/

pietercolpaert commented 2 years ago

@pietercolpaert : Would really be honoured with feedback on our ideas: https://pdsinterop.org//solid-link-metadata/

I think what you do makes sense!

I also think this is a nice way of having both a server-side solution (e.g., a solid server should be able to redirect based on breadcrumbs like this), and a client-side solution (e.g., a client just interprets the bread crumbs and follows the link when the server doesn’t understand them).

SolidLabResearch / Challenges