🔌 Plugin: Artifacts and Regitries for the Catalog, Starting with ECR

backstage / community-plugins

Community plugins for Backstage

Apache License 2.0

172 stars 238 forks source link

🔌 Plugin: Artifacts and Regitries for the Catalog, Starting with ECR #585

Open fordneild opened 5 months ago

fordneild commented 5 months ago

🔖 Summary

The plugin is able to ingest Images and Repos from AWS ECR, and emits Artifact and Registry Entities. These can be sorted into channels. Essentially, allowing the catalog to serve as a release pipeline of sorts. It is also able to use EventBridge + SQS to ingest images in real time.

🌐 Project website (if applicable)

No response

✌️ Context

I work at Anduril, and we have been using backstage for about a year. I would like to launch this plugin that I built at BackstageCon in November.

👀 Have you spent some time to check if this plugin request has been raised before?

[X] I checked and didn't find similar issue

✍️ Are you willing to maintain the plugin?

[X] I understand the responsibilities as a Plugin Maintainer Governance & will maintain the plugin

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?

Yes I am willing to submit a PR!

awanlin commented 5 months ago

It might make sense to get this in place sooner than BackstageCon that way you know it will be out.

fordneild commented 5 months ago

We can do that too! Ill need some time to decompose the module, at a high level im thinking

common plugin for Artifact and Registry entities
ECR plugin which has provider + processor
common front end plugin to display artifacts
ECR specific front end plugin

I feel this will make it easy for myself + others to contribute plugins for other registry types like Dockerhub and artifactory. Or if it makes sense to keep it all in one giant plugin, thats fine too. I suppose ill defer to more experienced plugin authors

awanlin commented 5 months ago

Adding the Discord thread: https://discord.com/channels/687207715902193673/1257397536994361375

Now that I read the Discord thread on this I do agree with @freben having all this in the Catalog as entities does not feel like good practice. The source of truth for this data is in these registries. Backstage should simply be pulling these in real time using an annotation like the vast majority of the plugin ecosystem currently does.

webark commented 4 months ago

We are wanting to release a new "artifact" kind that will encompass packages, docker repositories, and git repositories initially.

We are wanting this in the catalog to track ownership and increase discoverability, and have all of these behind the same api. We are dynamically pulling in versions and images, but not the packages and repositories for example.

Now, something that came up was items were getting duplicate names, so we did multiple kinds, but then that has also given us pause.

Is the "prescribed" way would just be to put this info in the spec of the components?Or is it more just all of this doesn't belong in the catalog.

If you don't have it in the catalog, how do you answer questions like "what components is this package managed by", or "who owns this docker repository", or "who are all the groups that own components that are leveraging this package"

awanlin commented 4 months ago

When I read this issue during the Community Plugins I didn't take the time to really read it and missed the intent. I'm not trying to block this but feel we should have given careful consideration for it.

@webark Going to follow up on Discord as that seems like the better place for this. 👍

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

webark commented 2 months ago

We implemented this a few months ago, and are using them to graph owners to security findings in running instances. We have taken the approach of Artifacts being owned by components that are owned by groups that are owned by domains and domains have leads derived from the groups that are a part of it. It seems to be helping with our automated ticketing process.

We hope to move to image signing and making the "ownership stamp" at that time, but that's a long tail development and release process change.

github-actions[bot] commented 2 days ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.