search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.2k stars 4.77k forks source link

Got 414 Error code when doing an "oc adm catalog mirror" to Harbor - Nginx - Proxy #15897

Closed Zwordi closed 2 years ago

Zwordi commented 3 years ago

bHello,

I was having an issue with harbor when trying to have Openshift Operator Catalog synced for an restricted cluster setup.

I was facing this kind of error when running the command :

oc adm catalog mirror <index_image> <mirror_registry>:<port>/<namespace> [-a ${REG_CREDS}] [--insecure] [--index-filter-by-os='<platform>/<arch>'] [--manifests-only]

Source : OLM-Restricted Networks

Logs of the command :

error: unable to push registry.redhat.io/rhel8/redis-5: failed to upload blob sha256:fcbad2014f8c13c8ed53de57098dc856f6f22e8f6ba419fe6200326ae915c52f: Post "https://******l/v2/openshift/rhel8-redis-5/blobs/uploads/": error parsing HTTP 414 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>414 Request-URI Too Large</title></head>\r\n<body>\r\n<center><h1>414 Request-URI Too Large</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"

Proxy Logs:

Oct 28 09:01:34 192.168.176.1 proxy[19963]: 10.100.9.77 - "GET /service/token?account=admin&scope=repository%3Aopenshift%2F3scale-amp2-3scale-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-3scale-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-apicast-gateway-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-apicast-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-apicast-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-backend-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-backend-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-memcached-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-system-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-zync-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp2-zync-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2F3scale-amp26-3scale-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Fadvanced-cluster-security-rhacs-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Fadvanced-cluster-security-rhacs-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-init-openj9-11-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-init-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-init-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-lts-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-lts-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-lts-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-openj9-11-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-rhel8-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-broker%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-cert-manager-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-cert-manager%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-interconnect-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-interconnect-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-interconnect%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-address-space-controller%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-agent%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-auth-plugin%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-broker-plugin%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-console-init%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-console-server-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-controller-manager-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-controller-manager-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-none-auth-service%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-standard-controller%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-online-1-topic-forwarder%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-bridge-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-bridge-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-kafka-24-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-kafka-25-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-kafka-26-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-kafka-27-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-kafka-27-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-kafka-28-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amq-streams-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Famq7-amqstreams-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-20-early-access-controller-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-20-early-access-hub-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-20-early-access-platform-resource-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-20-early-access-platform-resource-runner-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-platform-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-platform-resource-operator-bundle%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-platform-resource-rhel7-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Fansible-automation-platform-platform-resource-runner-rhel7%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-backup-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-configbump-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-crw-2-rhel8-operator-metadata%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-crw-2-rhel8-operator%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-dashboard-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-devfileregistry-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-devworkspace-controller-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-devworkspace-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-jwtproxy-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-machineexec-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-plugin-java11-openj9-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-plugin-java11-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-plugin-java8-openj9-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-plugin-java8-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-plugin-kubernetes-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-plugin-openshift-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-pluginbroker-artifacts-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-pluginbroker-metadata-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-pluginregistry-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-server-operator-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-server-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-cpp-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-dotnet-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-golang-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-java-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-node-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-php-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-stacks-python-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-theia-endpoint-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-theia-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcodeready-workspaces-traefik-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcompliance-openshift-compliance-content-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcompliance-openshift-compliance-openscap-rhel8%3Apull%2Cpush&scope=repository%3Aopenshift%2Fcompliance-" 414 170 "-" "-" 0.003 - .

Looking at it, I tried to add a larger buffer at the nginx.conf settings as :

server {
    listen 8443 ssl;
#    server_name harbordomain.com;
    server_tokens off;
    # SSL
    ssl_certificate /etc/cert/server.crt;
    ssl_certificate_key /etc/cert/server.key;

    # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
    ssl_protocols TLSv1.2;
    ssl_ciphers '!aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES:';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;

    # disable any limits to avoid HTTP 413 for large image uploads
    client_max_body_size 0;
##ADDED these three lines to have a bigger buffer
    client_body_buffer_size 128k;
    client_header_buffer_size 5120k;
    large_client_header_buffers 16 5120k;
##ADDED
    # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
    chunked_transfer_encoding on;

    # Add extra headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
    add_header X-Frame-Options DENY;
    add_header Content-Security-Policy "frame-ancestors 'none'";

Then a reboot of the stack made it works as i wanted and I have a running sync using my Openshift cli workflow.

It might be interesting to have it integrated on futur release or I just miss it on the docs :)

Please let me know,

wy65701436 commented 3 years ago

thanks @Zwordi , we can open it, if a lot user encouter it, then we can consider to have it.

IwanSE commented 2 years ago

@wy65701436 Hello +1, maybe fix?)

@Zwordi thx for fix, worked!

github-actions[bot] commented 2 years ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

IwanSE commented 2 years ago

upd #16986

github-actions[bot] commented 2 years ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] commented 2 years ago

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.