Rebranded dserver, refactored REST API

[jotelha]

This PR picks up https://github.com/jic-dtool/dtool-lookup-server/pull/56 again.

The sole fork now lives at https://github.com/livMatS/dserver, I have archived any dserver-related repos on my personal jotelha account. Once we merge, it will be relatively easy to replace livMatS in the whole repo with jic-dtool or whatever the organization will be called then.

Everything has been rebranded from dtool-lookup-server to dserver.

Major modifications and the rationale behind them are outlined briefly in the changelog.

You see the openapi docs at https://demo.dtool.dev/lookup/doc/swagger, as usual with many mock datasets and testable with a token generated with

curl --insecure -H "Content-Type: application/json"  \
  -X POST -d '{"username": "testuser", "password": "test_password" }'  \
  https://demo.dtool.dev/token

After discussion with Tjelvar on 28 Feb 2024, decided to first submit full manuscript unaltered to PLOS ONE. Then, if PLOS ONE permits, we also submit the dserver repository to JOSS for code review shortly after. We mention this intention in the cover letter to PLOS ONE. I suspect there will be no objection, as PLOS review focuses on the manuscript only without rigorous code review (see https://journals.plos.org/plosone/s/submission-guidelines#loc-methods-software-databases-and-tools and https://journals.plos.org/plosone/s/materials-software-and-code-sharing) , while JOSS does the opposite, the scopes don't overlap.

For the PLOS submission, not much work left:

• [x] Set manuscript into PLOS-required format
• [x] Submit

For JOSS submission, we need to

• [x] Prepare 250-1000 words brief summary without overlap with PLOS manusript
• [x] Embed mini manuscript within the repo
• [x] Think about whether to embed our design document (what we used to call "suppl. inf.") with the user stories as well
• [ ] Discuss about debranding the organization from jic-dtool to sth. jic-less ( dtool is taken, dserver as well, arbitrary suggestions: decosystem, decentral-dtool, dtool-dserver ...)
• [x] Discuss about the benefits / downsides of having PUT and PATCH methods. Get rid of them again if deemed superfluous.
• [x] Reevaluate the need for all the schemas and reduce their number and fields where possible.
• [ ] Reevaluate the need for any further routes
• [ ] Document remodeled API (I have only very superficially replaced the requested URIs in the README with what they would look like with the remodeled API)
• [ ] Actually test documentation snippets. (What I like a lot about the matscipy is that the documentation is partially generated from Jupyter notebooks, e.g. http://libatoms.github.io/matscipy/applications/electrochemistry_1.html from https://github.com/libAtoms/matscipy/blob/master/docs/applications/electrochemistry_1.py, thereby serving as validation tests at the same time, on the intermediate term we could think about doing that for the dserver docs as well)
• [x] Autogenerate docs with sphinx and deploy (workflow and docs stub are set up, see https://livmats.github.io/dserver)
• [ ] Make a pre-v1 release of packages once everything settled
• [ ] Make the v1 release once publication accepted by JOSS

We decided NOT to

• rebrand the dtool-lookup-server-related repositories to dserver
• rename the PyPI packages

[jotelha]

We decided on a few details.

• [x] For maintainability, remove POST and PATCH routes. Only offer GET, PUT, and DELETE routes.
• [x] Replace /users/me route with /me (NOT /whoami) route.
• [ ] always escape whole base and dataset URIs embedded in URL, do not break them into components and do not add server-side logic to rewrite URLs.
• [x] Rebrand dtool-lookup-server as dserver only in docs, keep repository and package names in the dtool-lookup-server scheme. (dserver, for example, is taken on PyPI)
• [x] See whether schemas User and RegisterUser can be summarized as single schema
• [x] See whether schemas BaseURI and UserPermissionsOnBaseURI can be summarized as single schema

@tjelvar-olsson does that sum up the technical decisions of our conversation yesterday accurately?

There are a few follow-up ideas I would like to discuss.

• If we decide to always escape the whole dataset URI, i.e. /uris/s3%3A%2F%2Ftest-bucket%2F1a1f9fad-8589-413e-9602-5bbd66bfe675 instead of /uris/s3/test-bucket/a1f9fad-8589-413e-9602-5bbd66bfe675, then nothing speaks against /uris/{uri}/manifest, /uris/{uri}/readme, /uris/{uri}/annotations, /uris/{uri}/tags instead of the current /manifests/{uri}, /readmes/{uri} ... anymore.
• Would it make sense to implement PUT methods on those manifest, readme, annotations and tags routes to allow updating them selectively instead of using the RegisterDataset schema on the PUT /uris/{uri} route that requires the full dataset metadata?

[jotelha]

@tjelvar-olsson Another thought: what about rebranding dtool_lookup_server repo & PyPI package to dservercore in analogy to dtoolcore? I have had rebranded most of the livMatS-side ecosystem and forks already at https://github.com/livMatS to dserver-this and dserver-that, and I think the names are better because shorter for most repositories, e.g. dserver-retrieve-plugin-mongo or dserver-notification-plugin instead of the lengthy dtool-lookup-server-retrieve-plugin-mongo or dtool-lookup-server-notification-plugin, and I suspect the only conflict on PyPI would be the dserver package.

[jotelha]

@tjelvar-olsson, to have the documentation deployment workflow run through, I think you migh have to modify some permissions on the repository, it runs fine for the livMatS fork, but it fails with

Branch "main" is not allowed to deploy to github-pages due to environment protection rules.

here. I don't know, however, what exactly to tune. The jic-dtool/dtool-lookup-server pages settings that I can access look like this, while the livMatS/dserver pages settings look like this, Look pretty similar to me.

[jotelha]

@tjelvar-olsson, to have the documentation deployment workflow run through, I think you migh have to modify some permissions on the repository, it runs fine for the livMatS fork, but it fails with

Branch "main" is not allowed to deploy to github-pages due to environment protection rules.

here. I don't know, however, what exactly to tune. The jic-dtool/dtool-lookup-server pages settings that I can access look like this, while the livMatS/dserver pages settings look like this, Look pretty similar to me.

I think it will likely work as soon as merged. The default branch on the livMatS fork is "main", and the default branch on the jic-dtool repo is master. I think the deploy error

Branch "main" is not allowed to deploy to github-pages due to environment protection rules.

arises due to that fact . Would you want to switch default branch from "master" to "main" as well?