search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.12k stars 4.76k forks source link

Nginx 414 Request-URI Too Large when pushing large contents #14414

Closed m4r1k closed 3 years ago

m4r1k commented 3 years ago

When mirroring the OpenShift OperatorHub to Harbor 2.2, Nginx fails with the following error

<html><head><title>414 Request-URI Too Large</title></head><body><center><h1>414 Request-URI Too Large</h1></center><hr><center>nginx</center></body></html>

Follows a sample push command logged by Nginx

Mar 10 18:57:51 172.22.0.1 proxy[20934]: 192.168.222.1 - "GET /service/token?account=ocp4&scope=repository%3Aocp4-v4.6%2F3scale-amp2-3scale-rhel7-operator-metadata%3Apull%2Cpush&scope=reposi
tory%3Aocp4-v4.6%2F3scale-amp2-3scale-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp2-apicast-gateway-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp2-a
picast-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp2-apicast-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp2-backend-rhel7%3Apull%2
Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp2-memcached-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp2-system-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp
2-zync-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2F3scale-amp26-3scale-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-broker-lts-operator-bundle%3Apull%2Cpush&scope=re
pository%3Aocp4-v4.6%2Famq7-amq-broker-lts-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-broker-lts-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-broker-
operator-bundle%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-broker-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-broker%3Apull%2Cpush&scope=repository%3Aocp4
-v4.6%2Famq7-amq-interconnect-operator-metadata%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-interconnect-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-interconnect
%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-address-space-controller%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-agent%3Apull%2Cpush&scope=repository%3
Aocp4-v4.6%2Famq7-amq-online-1-auth-plugin%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-broker-plugin%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-console
-init%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-console-server-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-controller-manager-rhel7-operator-met
adata%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-controller-manager-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-mqtt-gateway%3Apull%2Cpu
sh&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-mqtt-lwt%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-none-auth-service%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq
7-amq-online-1-standard-controller%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-online-1-topic-forwarder%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-bridge-rhel7%3
Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-cluster-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-kafka-23-rhel7%3Apull%2Cpush&scope=repository%3Aoc
p4-v4.6%2Famq7-amq-streams-kafka-24-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-kafka-25-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-kafka-26-
rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-amq-streams-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp
4-v4.6%2Famq7-amqstreams-rhel7-operator-metadata%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-adapters-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fa
mq7-tech-preview-amq-online-1-iot-auth-service%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-device-registry-datagrid%3Apull%2Cpush&scope=repository%3Aocp4-
v4.6%2Famq7-tech-preview-amq-online-1-iot-device-registry-file%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-device-registry-rhel7%3Apull%2Cpush&scope=repos
itory%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-http-adapter%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-lorawan-adapter-rhel7%3Apull%2Cpush&scope=
repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-mqtt-adapter%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-proxy-configurator%3Apull%2Cpush&scop
e=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-sigfox-adapter-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-tenant-cleaner-rhel7%3Apull
%2Cpush&scope=repository%3Aocp4-v4.6%2Famq7-tech-preview-amq-online-1-iot-tenant-service%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Famqstreams-1-amqstreams10-clusteroperator-openshift%3Apu
ll%2Cpush&scope=repository%3Aocp4-v4.6%2Fansible-automation-platform-platform-resource-operator-bundle%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fansible-automation-platform-platform-resou
rce-rhel7-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fansible-automation-platform-platform-resource-runner-rhel7%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-c
onfigbump-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-crw-2-rhel8-operator-metadata%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-crw-2-rhel8-
operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-devfileregistry-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-jwtproxy-rhel8%3Apull%2Cpus
h&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-machineexec-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-plugin-java11-openj9-rhel8%3Apull%2Cpush&scope=repos
itory%3Aocp4-v4.6%2Fcodeready-workspaces-plugin-java11-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-plugin-java8-openj9-rhel8%3Apull%2Cpush&scope=repository%3Aocp4
-v4.6%2Fcodeready-workspaces-plugin-java8-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-plugin-kubernetes-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcoderea
dy-workspaces-plugin-openshift-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-pluginbroker-artifacts-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-wor
kspaces-pluginbroker-metadata-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-pluginregistry-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-s
erver-operator-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-server-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-stacks-cpp-rhel8%3Apull%
2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-stacks-dotnet-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-stacks-golang-rhel8%3Apull%2Cpush&scope=repos
itory%3Aocp4-v4.6%2Fcodeready-workspaces-stacks-java-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-stacks-node-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fco
deready-workspaces-stacks-php-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-stacks-python-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-theia-endpoint-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-theia-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcodeready-workspaces-traefik-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-bridge-marker%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-cluster-network-addons-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-cnv-containernetworking-plugins%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-cnv-must-gather-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-hco-bundle-registry%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-hostpath-provisioner-rhel8-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-hostpath-provisioner-rhel8%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-hyperconverged-cluster-operator%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-kubemacpool%3Apull%2Cpush&scope=repository%3Aocp4-v4.6%2Fcontainer-native-virtualization-kubernetes-nmstate-handler-rhel8%3Apull%2Cpush" 414 170 "-" "-" 0.000 - .

Follows the CLI used by OpenShift (OCP docs)

oc adm catalog mirror \
  registry.redhat.io/redhat/redhat-operator-index:v4.6 \
  harbor.localdomain/ocp4-v4.6 \
  -a ~/pull-secret.json \
  --filter-by-os=linux/amd64

To fix the problem, I've injected a custom Nginx config, essentially playing with the Nginx's buffers

fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;

client_max_body_size 24M;
client_body_buffer_size 128k;

client_header_buffer_size 5120k;
large_client_header_buffers 16 5120k;

Expected behavior and actual behavior: I'd expect Nginx to be tuned out of the box to handle large requests otherwise it may fail as in my case.

Steps to reproduce the problem: Pushing a lot of contents in parallel

See the above linked OCP documentation and CLI used to actually create a mirror of OperatorHub

Versions: Please specify the versions of following systems.

reasonerjt commented 3 years ago

I think the problem is in the implementation of oc, we should ask if it's reasonable to generate a URL like this. You can keep the workaround in your fork if it works for you, but I don't think a server should always accept a URL like this

oliverbutanowitz commented 3 years ago

Hi there, i see the same issue here using Openshift 4.7. Openshift docs are describing the requirements for the mirror-registry to be a "registry that supports Docker v2-2" (https://docs.docker.com/registry/spec/manifest-v2-2/). I didn't find in the harbor docs if Harbor 2.2 is supporting this Docker v2-2? Do you know if it is supported?

reasonerjt commented 3 years ago

@m4r1k I see this is fixed on the oc side? Do you think we still need to keep this open?

m4r1k commented 3 years ago

Closing Harbor issue (and PR) given the problem was indeed in oc https://github.com/openshift/oc/issues/789