Errors when publishing to Digital Ocean Container Registry

[jonhilt]

Testing this for a project where I need to publish images to Digital Ocean's container registry and running into an error.

I can build and image and manually push it, like this:

dotnet publish -p:PublishProfile=DefaultContainer -p:ContainerRepository=registry.digitalocean.com/practical-asp-net/blazordemo 
docker push registry.digitalocean.com/practical-asp-net/blazordemo

With that working I figured I'd trying using dotnet to push the image as well.

My assumption was that I'd be able to do this:

dotnet publish -p:PublishProfile=DefaultContainer -p:ContainerRegistry=registry.digitalocean.com  -p:ContainerRepository=practical-asp-net/blazordemo 

(hopefully I got that syntax right).

However, I'm getting a Bad Request response when I do.

See attached binlog for details.

msbuild.zip

Thanks,

PS: I'm using .NET 8 Preview 7

[baronfel]

Hi @jonhilt - I've been able to reproduce this failure on my test harness with the latest version of the tooling - I think this may be a quirk/bug in how we do basic auth but I haven't been able to confirm that.

For now, I think the best thing to do would be to push to a local docker daemon (by removing the ContainerRegistry property) and then use docker tag and docker push to push the generated container to Digital Ocean.

[baronfel]

Yeah - the problem is an interesting chain of behaviors:

• we authenticate successfully to repository A
• we start upload layers to A
• we try to make the process faster by asking repository A to 'mount' the layer from repository B, if it exists
• DigitalOcean sees that we haven't requested access to B, and tells us we need a key that needs both write scope for repository A and read scope for repository B
• We attempt to get that key, and DO tells us we have an invalid request (because we passed a scope with a space in it: scope=repository%3achethusk-sdk-container-demo%2fsdk-container-demo%3apull%2cpushrepository%3adotnet%2faspnet%3apull)
• We blow up

I checked what the Docker CLI does, and it doesn't even attempt the 'mount' operation - it always attempts to upload the layer from the local machine to the destination. I think we should make the 'mount' operation specifically resilient to authentication failures (i.e. catch the UnableToAccessRepositoryException) and continue on - if we do so then we flow right to the same upload-from-local behavior that the Docker CLI does (though it would also be great to understand why DO doesn't seem to handle scopes like this.

[baronfel]

We are likely not going to be able to get this into .NET 8.0.100 release itself - we're about to create branches for RC2 and that's historically when we shift into a much higher bar. We can look at this enhancement for 8.0.200 in the Jan/Feb time frame, however - though it's lower priority than some of the big rocks we have like tackling multi-RID publish, etc. This one would be a great candidate for an engaged user to contribute to the tooling - some potential fixes:

More registry-level quirk knobs

We already have 'quirk' modes for a few features, like parallel upload of Layers, and chunked upload of Layers. Various registries support different modes, and we have environment variables that let users control the quirks more directly. We could add another quirk for "should we attempt to 'mount' the Layer from the registry if it exists". If yes, we would do the current pattern of "try mount, else pull from the source and upload from the local machine". If no, we would always "pull from the source and upload from the local machine"

Try to handle this kind of error directly in the "try to mount Layer" call

Alternatively, we could put more robust error handling into the Mount Layer API call to try and detect this kind of failure and recover from it (effectively returning false from that API call for more scenarios than it does today). From there the rest of the system would continue uploading the full layer.

I lean towards the first approach, and I'm sure there are more.