Notes on how to serve multiple package respositories

[stschiff]

I think I have a pretty concrete idea on how we can teach our server to host several independent package repositories.

(Note that in the following I also use a new naming idea that I had: "Community Repository" for the current published_data, and "Curated repository" for the upcoming Minotaur-packages.

First, trident serve should accept multiple base directories, each with a different name, e.g.

trident serve -d community_repository=~/repos/published_data -d curated_repository=~/repos/ppd -d poseidon_aadr=~/repos/aadr

(and just to keep things very flexible, we could even use the same name multiple times to have multiple base directory belong to the same repository, although that will hardly be necessary).

Then, we just extend all APIs to include the repo name, so instead of /packages we would have /community_repository/packages. And instead of /zip_file/2010_RasmussenNature?package_version=1.1.0 we might have /curated_repository/zip_file/2010_RasmussenNature?package_version=1.1.0 and so on.

So from a Server/API perspective this is really simple to implement.

From the client-perspective, we would have to include a mandatory flag --repository or -r for trident list --remote and trident fetch. Of course, we could implement a hard-coded default (like community_repository or so). And that's that. The User would have to take care that they download stuff into a respective folder structure.

We would then also add a new API /repositories which would list the repositories served under the current server instance.

Note that this implementation then assumes that there is full independence between the repos. All APIs will work entirely independent in each repo. Like /packages would give entirely different results when run on the different repos. I think this is the simplest setup.

[nevrome]

Sounds good! I think we could really do it like that. Just two questions and a comment:

• -d with more then a path is supposed to be limited to serve, right?
• /zip_file/ without the repo name should continue to point to the community maintained archive to not deprecate the current API, right?
• The suggested names are not good imho. Community Repository implies that the other one is not owned by the community, and Curated repository implies that there is more curation going on. Both implications are wrong to my understanding. This naming question is a nightmare.

[stschiff]

Yes, points well taken:

• Regarding -d, I think we might even leave it as is, and instead add a new option --repo_name which then is read like a list and interpreted in the same order as multiple -d options.
• Good point about deprecation. I think in fact we could just declare the first repo as the default one and continue serving the current APIs for the default repo, and then offer the APIs that are repo-specific in addition. We could also consider putting the repo_name into the query string, à la /packages?client_version=1.2.0.0&repo_name=published_data and then use the absence of that field as serving the default repo.
• Yes, I agree, the idea was bad. Let's keep thinking.

[nevrome]

I thought about it for a while and decided now that I like the idea of putting the repo names into the query string. It has some advantages:

1. It doesn't break the current API. So it would be easier to now release trident 1.2.0.0, which otherwise (e.g. with /<repo_name>/packages) would be deprecated again right away.
2. It allows to query multiple repos at once, e.g. /packages?client_version=1.2.0.0&repo_name=PBA,PAA (or whatever is the best way to encode lists in URLs :shrug: - maybe &repo_name=PBA&repo_name=PAA?)
3. This works equally for all 4 current endpoints.

I think the best default would be to query all archives, not just published_data (or how we would like to call it eventually)

[stschiff]

Definitely yes to query-string instead of path-element.

I'm interested to hear your arguments for why you would like to allow multiple repos to be queried, optional or by default. I can see no really good use case here, since they will eventually contain the same data. I think this will only lead to confusion. Also, I think we would like to encourage a workflow where users keep them in separate folder hierarchies instead of mixing them, right? I think conceptually they are really quite different worlds and we should keep them separate.

[nevrome]

Hm - from a --downloadAll perspective you're right, but from the point of view of normal list and fetch calls I think having all available is the best default. /packages, /groups and /individuals should imho show all versions of all entities Poseidon has to offer. zip_file should by default allow to download all packages, without the need to set a repo first. As we planned it now, there will be no package name overlaps between the archives.

[nevrome]

Implemented in #258