Open pietercolpaert opened 2 years ago
@dexagod is doing relevant work: can you elaborate on that one?
We created a toolset for Solid on the CLI and Node.js, mainly focused and tested for the Community Solid Server.
It provides a set of functions that mimick the linux commands for managing files/resources. Relevant here is the copy function. It is available through the CLI interface as
node bin/solid.js [auth options] copy [copy options] <source> <destination>
enabling you to copy files
In Node.js, you can use the same functionality as follows:
import { copyData } from "install/location''
let options = { ... }
await copyData(source, destination, options);
to copy files in the same manner.
The tool provides authentication options both for the CLI and for Node.js to work with authenticated resources
All functions in Node.js take a fetch
parameter in their options.
This allows you to authenticate however you wish, and pass the authenticated fetch function.
You can use the createAuthenticatedSession()
function exposed by the lib that enables authentication using username, identity provider and password parameters, but this is currently only supported for the Community Solid Server v2.x.x
For more general options, see the inrupt docs on Solid authentication for Node.js
The CLI interface allows you to authenticate using either a config file with user credentials (only currently for Community Solid Server v2.x.x), via a config file (again only CSS v2.x.x), or using an interactive browser window (should support all versions), that caches session information between calls where possible.
I still have an additional comment here:
Point 3: All references are updated to the new URLs -> does this mean that every resource should be forwarded? Because this seems like something that is a hell to maintain: e.g. you move / delete resources in their new location.
Would it make sense to only forward the root, so that discovery of resources is still maintained through the pod root?
In this case direct links e.g. shared photos could break if they use absolute URIs, but to catch these cases maybe this should be caught using a URL rewrite?
Straight up forwarding all resources seems a tad excessive
-> does this mean that every resource should be forwarded? Because this seems like something that is a hell to maintain: e.g. you move / delete resources in their new location.
I would say yes, all resources should be redirected.
Could be achieved via some kind of moved.meta
file, which could act similar to a UNIX softlink.
But this should not be guaranteed anymore IMO once another resources is placed over that location.
If I understand this scenario correctly, does it mean that you are moving data from one pod to the other? With storage provider I first thought about the storage system that is used by CSS in the back-end.
I think it should be important to leave breadcrumbs in order to be able to find data after it has been moved. If we don't do that, links will break just as fast in the current web.
What do you think of the https://github.com/pdsinterop/solid-migrator?
@sjoerdvangroning Why is that? From my understanding the URLs of all resources remain the same.
@sjoerdvangroning Why is that? From my understanding the URLs of all resources remain the same.
Maybe we are thinking of a different use case? Can you describe in more detail your use case and how URL can remain the same?
I am thinking of external links to POD/Data and when you change storage provider, the URL will change. (for example when you move your POD from solid.community to solidcommunity.net).
I assumed that the user has control over the whole domain, so they can redirect the domain to the new storage provider. If the domain is tied to the storage provider then you would indeed have issues. I'm also wondering what @pietercolpaert means with step 3 actually.
In step 3 I was thinking about organizational hypermedia on top of LDP. E.g., if you use AS2 pagination, or shape trees, or TREE/LDES, that you update the link URLs to resources on the new server instead of on the old server, and then having to follow bread crumbs or instead of following a redirect.
I agree with @sjoerdvangroning that breadcrumbs should be provided. A redirect is I believe a specific implementation of such a breadcrumb.
We have given this some thought in our migrator project. We created a specification and a working implementation for moving data between storage providers. Being able to move your data is important for the SOLID eco-system. If servers would follow the metadata and give redirects, data can still be found and it could give clients time to update their links (honour a permanent redirect for example and update the links).
@pietercolpaert : Would really be honoured with feedback on our ideas: https://pdsinterop.org//solid-link-metadata/
@pietercolpaert : Would really be honoured with feedback on our ideas: https://pdsinterop.org//solid-link-metadata/
I think what you do makes sense!
I also think this is a nice way of having both a server-side solution (e.g., a solid server should be able to redirect based on breadcrumbs like this), and a client-side solution (e.g., a client just interprets the bread crumbs and follows the link when the server doesn’t understand them).
Actors/actresses
Two solid storage providers and an end-user
Preconditions
One storage provider stores nothing yet, another provider stores already quite some resources for an end-user.
Steps
Postconditions