[Feat]: Process more than one HTTP request at a time

[the-sun-will-rise-tomorrow]

Is your feature request related to a problem? Please describe.

It looks like Zot can only process one HTTP request at a time.

If a HTTP request is made to Zot while it is already processing a HTTP request, the other request will block until the first one completes.

I am seeing this at least with HEAD /v2/$tag/blobs/$digest requests. Each request takes about 200ms for me, and trying to perform multiple requests in parallel seems to cause them to form a queue and be processed one after another.

Describe the solution you'd like

It would be nice if Zot could leverage Go's concurrency mechanisms to allow processing multiple requests simultaneously.

Describe alternatives you've considered

I suppose multiple Zot instances backed by the same storage pool might work, but this seems a much less elegant solution.

Additional context

I am experimenting with efficient uploading of images with many layers (built using Nix). In my experiments, the vast majority of the time seems to be spent on Skopeo asking Zot if a layer blob is already in the registry.

[rchincha]

@the-sun-will-rise-tomorrow

Thanks for trying out zot. Just FYI, you can go one step further in your tests using https://zotregistry.dev/v2.1.0/articles/pprofiling/

[rchincha]

@the-sun-will-rise-tomorrow can you also pls share your config (anonymized) so we can try and reproduce this.

[the-sun-will-rise-tomorrow]

OK, here it is:

{
    "distSpecVersion": "1.1.0-dev",
    "storage": {
        "dedupe": true,
        "gc": true,
        "gcDelay": "1h",
        "gcInterval": "6h",
        "rootDirectory": "/data/zot/"
    },
    "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "realm": "zot",
        "tls": {
            "cert": "/secrets/tls_crt.pem",
            "key": "/secrets/tls_key.pem"
        },
        "auth": {
            "htpasswd": {
                "path": "/etc/zot/htpasswd"
            }
        },
        "accessControl": {
            "metrics": {
                "users": ["metrics"]
            },
            "repositories": {
                "**": {
                    "defaultPolicy": ["read", "create", "update"],
                    "anonymousPolicy": ["read"]
                }
            },
            "adminPolicy": {
                "users": ["admin"],
                "actions": ["read", "create", "update", "delete"]
            }
        }
    },
    "log": {
        "level": "warn"
    },
    "extensions": {
        "search": {
            "enable": true
        },
        "sync": {
            "enable": true,
            "credentialsFile": "/secrets/sync-auth.json",
            "registries": [
                {
                    "urls": ["https://registry.example.com"],
                    "content": [
                        {
                            "prefix": "**"
                        }
                    ],
                    "onDemand": false,
                    "tlsVerify": true,
                    "pollInterval": "1h"
                }
            ]
        },
        "scrub": {
            "enable": false
        },
        "metrics": {
            "enable": true,
            "prometheus": {
                "path": "/metrics"
            }
        },
        "ui": {
            "enable": true
        }
    }
}

[rchincha]

@the-sun-will-rise-tomorrow

Thanks for trying out zot. Just FYI, you can go one step further in your tests using https://zotregistry.dev/v2.1.0/articles/pprofiling/

With your test running ...

curl -s http://localhost:5000/v2/_zot/pprof/profile?seconds=30 > cpu.prof
go tool pprof -http=:9090 cpu.prof

should quickly tell us where the latency is coming from.

[the-sun-will-rise-tomorrow]

Thank you, I have forwarded the above to the person who maintains the Zot instance, and will get back to you when I hear back from them.