Add authenticated JSON API to define new upstream/downstream mappings

[ncoghlan]

Anitya currently keeps track of a mapping between upstream and downstream components in particular distributions.

Given an admin-only API to define new entries, much of the task of populating the mapping should be able to be automated.

Given a public API to query that mapping, other tasks dependent on the mapping (like configuring the-new-hotness) should also be able to be automated.

[pypingou]

Given a public API to query that mapping, other tasks dependent on the mapping (like configuring the-new-hotness) should also be able to be automated.

Is this covered by https://release-monitoring.org/api/#Retrieve_a_package_for_a_distro ? Or would you need anything else?

[ncoghlan]

Now that you pointed out the docs, I think https://release-monitoring.org/api/#Retrieve_a_specific_project actually addresses the second part of the request ("given the upstream project name, report the distro specific package names")

That reduces the request to just the first part: defining an authenticated endpoint that can be used to add new upstream/downstream mappings from an automated script that compares tarball hashes.

[ncoghlan]

On closer inspection, the current project retrieval endpoint works by looking up via Anitya's internal numeric project ID, rather than what we'd like to do, which is to query by ecosystem (e.g. npm, pypi, maven, rubygems, etc) plus the name within that ecosystem.

https://release-monitoring.org/api/#List_all_projects is probably the closest currently, but even that may presumably give multiple answers if projects in different ecosystems have the same name.

[ncoghlan]

I filed an RFE to allow components to be looked up based on their upstream names: https://github.com/fedora-infra/anitya/issues/326

[ncoghlan]

@pypingou I've started looking more closely at this RFE now that #326 should be largely ready to go (pending your review), and ran into the problem that Anitya doesn't currently supported authenticated token-based access to the API endpoints.

Is there a preferred approach to token management established in Fedora Infrastructure already? If yes, could you point towards the appropriate guidelines?

If not, I'll put something together based on an internal project I'm currently working on (which is also based on Flask-RESTful and Flask-Security)

[pypingou]

Is there a preferred approach to token management established in Fedora Infrastructure already? If yes, could you point towards the appropriate guidelines?

So there are no standard for this at the moment, the few apps that supports it have mostly a home-baked solution (pagure is an example). We could consider moving anitya to OpenID-Connect since Ipsilon will have support for it in its next release and we already have a test server to play with it (https://iddev.fedorainfracloud.org/). https://pagure.io/fedora-hubs has been ported to OpenID-Connect and thus contains some code and doc that might be relevant for this.

Otherwise, if we do not go for OIDC, I'm fine with adding this, keeping in mind that I like keeping the dependency list short and that I try to separate the logic linked to the framework from the core logic of the app (with the idea of trying to make porting to another framework easier).

[ncoghlan]

I realised that a necessary prerequisite to programmatically adding upstream/downstream mappings is to be able to request monitoring of additional projects, so I've filed #329 to cover that.

[ncoghlan]

@pypingou As far as authentication goes, OpenID-Connect sounds fine to me, so what I'll do from an implementation perspective is to initially build draft implementations of this API and that for #329 without any authentication set up (so you can review and comment on the proposed API structure), before I come back to look at the authentication & authorisation problem at the PyCon Australia sprints next week.

[ncoghlan]

Current status at the PyCon AU sprints: I'm taking the flask-oidc dependency and the related README notes from Fedora Hubs and adapting them for Anitya.

[jeremycline]

This looks to be the same thing as https://github.com/fedora-infra/anitya/issues/42, right? I'm inclined to close #42 even though it's older since this has more activity. Any objections?

Edit: Ah, I see #329 and it looks like the two combined cover #42 (sans rename)

[ncoghlan]

@jeremycline The main difference between #42 and this proposal is that #42 proposed defining the API in terms of the internal Anitya project IDs (so it could be used for arbitrary projects), while this is limited to the cases where the project is part of a centralised upstream ecosystem of some kind (as per https://github.com/fedora-infra/anitya/issues/326 ).

That means it currently handles the case of ecosystems like PyPI, RubyGems, or NPM, but not the cases of "C/C++ tarball uploaded to a project's website".

That said, it does occur to me that the custom webscraping backend could be brought into the ecosystem model by defining www (aka "the web") as an ecosystem and using the escaped home page URL as the ecosystem level project identifier.

[ncoghlan]

So given that qualifier, +1 for closing #42 in favour of this one - the internal Anitya IDs are much harder to work with from a client perspective than the upstream identifiers, as they're dependent on which specific Anitya instance you're talking to, while the upstream identifiers are stable across instances.

[jeremycline]

@ncoghlan Thanks for the clarification. @glensc does this work for you?

[ncoghlan]

I should note that the patch for this is currently living at https://github.com/bkabrda/anitya-docker/tree/master/patches

We haven't pushed it back upstream yet because it would be a really bad idea to merge it without adding API authentication first, and I haven't had a chance to really look at that since PyCon Australia back in August.

[ncoghlan]

https://github.com/fedora-infra/anitya/issues/329#issuecomment-240665177 on #329 has links to my working repo where the API patches applied to the docker image originated.