Nexus plugin is slow to load docker artifacts

[rmartine-ias]

Describe the bug When loading the "Build Artifacts" tab for the first time (in each browser), with a nexus-repository-manager/docker.image-name: annotation, the tab takes around 20 seconds to load. This is because of the requests for the docker manifests, which take 6-7 seconds each, on our Nexus. This is past the point where people will click away because it is too slow.

To Reproduce Steps to reproduce the behavior:

1. Initialize plugin
2. Add a nexus-repository-manager/docker.image-name: annotation to a catalog item
3. Click on the "Build Artifacts" tab (may need to open devtools and disable cache)
4. See time to load

Expected behavior

Loading to be near-instant. If possible, load times to be shared among users, rather than repeated per user. Thoughts on how to do this:

• Add a backend to the plugin, with either a request-level or data-level cache. Refresh this on some schedule
• Add config to only fetch the latest N artifacts of each type
• Find a way to not request the full manifests, or at least not block on it -- we should be able to display a list of tags without waiting for the requests to complete. I think only artifact size requires the manifest?
• Suggest adding Varnish or similar in-between backstage and nexus (we'd rather not do this, if possible)
• Two or more of the above

Screenshots

Additional context

My company was planning on creating our own Nexus plugin, but delayed long enough that y'all shipped an OSS one. Since we've budgeted time for this already, we'd like to spend it working to contribute back to the work you're doing. I'm reporting this issue to see if there's something you already have planned for this/start a discussion, so we don't duplicate work you're already doing, or go down a different route than you envisioned.

[schultzp2020]

Hi @rmartine-ias,

Thank you for creating an issue!

Add config to only fetch the latest N artifacts of each type

I think this would be the best option for now. We can definitely revaluate this if the TTL is still too long.

Thoughts @tumido?

[tumido]

Hey @rmartine-ias,

Thank you for filing an issue here! 🙂

If you have more ideas or if you would like to steer this plugin in a specific direction or design, please share it! We welcome it, we are open to any suggestions. 🙂

If you'd like, you can also join us on Slack and for our bi-weekly community calls.

Back to the issue: What if we paginate on the searchServiceFetcher (we can use the continuationToken for pagination)? We can use this to lazy load additional entries and populate the rows on the fly. I don't think this would be too complex to implement and it would certainly be better than limiting the list of displayed entries. WDYT?

[rmartine-ias]

Hello @tumido and @schultzp2020! We definitely have some thoughts that we'll want to share -- I'll make a post in the slack, and add GH issues for discoverability as well. Will not be able to make it to the meetings (at least regularly), unfortunately; they're three hours before I'm typically awake.

Partially good news -- I figured out why our nexus was being slow. Requesting a manifest without certain accept headers makes it think only manifest schema 1 is supported, and on-the-fly rewriting to schema 1 is slow. (That's where I stopped looking. But I do wonder why it's slow, and why nexus doesn't cache those files.) I have a code change in the works for the nexus plugin to send those headers when requesting docker manifests. That significantly speeds up requests, but for resources with hundreds of images, it still takes a long time, so I don't want to close this issue.

I like the idea of paginating, but I think it can be one step faster, maybe. I don't know exactly the right words for things, but this is what I'm picturing: each page of search dumps the manifest download URLs into a queue. That queue is fetched from in groups of \ (see image) (at least for the first request), and so we can draw the table as soon as the first \ have returned, and keep fetching in the background. This would be faster than waiting for the data from a full page of search.

This could even be snappier by displaying the data from the search results immediately (version, repo, name, sha, modified) and async loading the size field as above.

A few other thoughts:

• I think default sort should be date modified, descending (which needs a little bug fix, clicking the table's modified field seems to sort alphabetical by human-readable date -- will open an issue). This needs a sort=version parameter added to the nexus search API call. This would make the "first five results" much more likely to be meaningful. I can make this change if you all agree.
• Something we're looking to add is the ability to have multiple artifact types per Component (maybe an OpenAPI spec, two maven artifacts, and a docker image). It would be ideal to handle this so that on first paint, the latest of all types are interleaved, limited to \ in total. (Or maybe it would be better to add them to the table as they come?) This might be a "cross the bridge when we come to it" type thing, but I wanted to bring it up in case it would influence the design.
• If we want to be able to let users sort/filter the table, lazy loading the rest of the results on page navigation introduces a lot of latency at that point. If everything is fetched in the background though, that's potentially a lot of unneeded API calls if the user never navigates past the first page, sorts, or filters. I'd lean towards the second, I think, for a better user experience.

@tplass-ias, anything I'm missing?

[tplass-ias]

Thanks @rmartine-ias , very thorough

From what I've heard from our users, they are mostly interested in quickly checking that their latest version has arrived in nexus, so limiting how many entries are displayed is actually preferred imo

For browsing through the full list of versions of an artifact, I think that's what the Nexus UI itself is for, and ideally the Backstage UI would have a convenient link to click for that, e.g. ${NEXUS_URL}/#browse/search/docker=attributes.docker.imageName%3D${IMAGE_NAME} for docker images

But I try to live by the word of grug, so I am biased towards finding 80/20 solutions, and am eager to shed requirements like pagination 😛

[rmartine-ias]

For the quickly checking the latest version, I was thinking the main way they'd do that would be a home-tab card with the latest asset of each type. Saves a click over going to the dedicated tab.

Not to derail, but I'm still sort of unclear on the more general philosophical point of what Backstage plugins should do, and what they should delegate to the underlying tool via click-through. Do you have a heuristic you're using?

[rhdh-bot]

This issue has been closed due to the fact that the Janus community is being sunset.

For future plugin issues, please use https://github.com/backstage/community-plugins/issues

For future showcase issues, please use https://issues.redhat.com/browse/RHIDP

For more information on the sunset, see:

https://janus-idp.io/blog/2024/07/05/future-of-janus-community https://issues.redhat.com/browse/RHIDP-3690 https://issues.redhat.com/browse/RHIDP-1018