Get-BCArtifactUrl is very slow (mainly for Sandbox artifacts)

[Koubek]

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!

Describe the issue Command Get-BCArtifactUrl -Type Sandbox takes very long to process. It happens mainly with Sandbox artifacts. I assume it's because there is a huge list of the artifacts behind the scene. Or maybe not, I can't tell :) But if yes, the question is if all of them are needed.

Scripts used to create container and cause the issue

Get-BCArtifactUrl -Type Sandbox
Get-BCArtifactUrl -Type Sandbox -select Current
# ... and similar ...

Additional context I think the time needed to process the request grows over time but I haven't made any serious measurements. I assume some cleanup could help probably to make the query perform faster ;)

[freddydk]

You are right - it is caused by the very large list of artifacts - we do cleanup the list occasionally - will see what I can do

[freddydk]

The "real" for for this however is when we start using NuGet, Universal packages or like for artifacts instead of blob storage - that is on the horizon - but not anytime soon.

[Koubek]

I know this might be off-topic, but have you already checked the performance on such a big volume of data e.g. with NuGet? Partially, it's just a curiosity and partially not - we use NuGet and have a good experience performance-wise so far but this might change over time so someone with a significant volume of the data would be a good example for us (positive and eventually also negative).

[freddydk]

We have not yet started looking into this, so no. But knowing what we know today (with the amount of artifacts), we obviously will test this thoroughly - or structure things in a way, which makes searching easier.

[tfenster]

@Koubek if you don't mind a bit of caching in between, you could use https://bca-url-proxy.azurewebsites.net/bca-url/sandbox/de/?donotredirect=true to get the URL. It is as slow as bcch (which it uses behind the scenes) on the first request for a particular artifact, but then it is fast as it takes the result from a cache. After 1h for sandbox artifactss and 24h for onprem artifacts, the cache entry expires and fetches the URL through bcch again. So depending on your usage scenario, this might at least help :)

[Koubek]

Thanks @tfenster, this is truly a nice option and I will share it with my colleagues ;) And really very good idea to implement the cache, I haven't been monitoring the response time for a while as we use some caching as well in the system I control, but not somewhere else I suppose.

[rdebath]

I notice that current versions of the BLOB storage can assign tags to the images. These tags can be searched on and apparently changed without re-uploading the blob.

Tags like "country", "major_version" would be obvious filter candidates. Ones like "latest", "previous","recent" could be attached to provide (hopefully) super quick searches for the related bcSelect values.

[freddydk]

I have done a lot of investigations into this - and we will come up with a solution for this (both search and download speed) - we will share a design in a few weeks - hopefully

[BW-PA]

Hi Freddy, do you have any updates to share on this matter? I came here looking as a build I was monitoring today on one of our self hosted Azure DevOps agents took 2min 51secs (of an elapsed 5min 59secs total) waiting for the "Determining artifacts to use" statement to complete.

Here's the statement from the raw log in DevOps

2024-10-22T05:00:14.9283994Z Determining artifacts to use
2024-10-22T05:03:06.6964906Z ##[group]Parameters

[freddydk]

Unfortunately not. Using the public web service above can speed things up quite a bit