production incident: Nginx new port setup is broken

[EvertonSA]

Hi, recently jfrog changed nginx ports from 80 and 443 to 8080 and 8443.

it looks good, but i don't think it was tested. when reaching for a docker remote, jfrog artifactory issues the client back to registry auth, but instead of doing it on 443, it issues back to 8443.

and 8443 is not exposed on the nginx service (only 80 and 443, mapping to 8080 and 8443).

I'm facing a huge production incident and I have opened ticket number 285150.

I tried to set the nginx ports back to 80 and 443 but nginx does not start, it throws permission denied.

this is what jfrog clients see on their kubernetes clusters:

Failed to pull image "jfrog-xxxx.xxxx.com/registry-1.docker.io/xxxx": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "jfrog-xxxx.xxxx.com/registry-1.docker.io/xxxx": failed to resolve reference "jfrog-xxxx.xxxx.com/registry-1.docker.io/xxxx": failed to authorize: failed to fetch oauth token: Post "https://jfrog-xxxx.xxxx.com:8443/artifactory/api/docker/registry-1.docker.io/v2/token": dial tcp 10.xxxx:8443: i/o timeout

we were able to workaround a few client clusters by adding manually the following

but this only works for clients that allow external access to 8443. if they only allow 80 and 443, they still have issues and I have no control over their firewall egress rules.

please fix asap, either by allowing reverting back to 80 and 443, or make artifactory redirect auth requests to the same port the client issued the initial tcp connection.

[EvertonSA]

when trying to set the ports back to the same ports it was using before,

nginx:
    http:
        internalPort: 80
    https:
        internalPort: 443

we get the following error:

Using deprecated password for user _internal. 2024-02-11 09:31:46 [169 entrypoint-nginx.sh] Preparing to run Nginx in Docker 2024-02-11 09:31:46 [14 entrypoint-nginx.sh] Dockerfile for this image can found inside the container. 2024-02-11 09:31:46 [15 entrypoint-nginx.sh] To view the Dockerfile: 'cat /docker/nginx-artifactory-pro/Dockerfile.nginx'. 2024-02-11 09:31:46 [19 entrypoint-nginx.sh] Setting up directories if missing 2024-02-11 09:31:46 [160 entrypoint-nginx.sh] Generating ssl example.key and example.crt 2024-02-11 09:31:46 [83 entrypoint-nginx.sh] Artifactory configuration already in /var/opt/jfrog/nginx/conf.d/artifactory.conf mkdir: can't create directory '/var/opt/jfrog/nginx/logrotate': File exists 2024-02-11 09:31:46 [92 entrypoint-nginx.sh] Adding logrotate to crontab 2024-02-11 09:31:46 [187 entrypoint-nginx.sh] Starting updateConf.sh in the background Using deprecated password for user _internal. 2024-02-11 09:31:46 [192 entrypoint-nginx.sh] Starting nginx daemon... nginx: [emerg] bind() to 0.0.0.0:443 failed (13: Permission denied)

[EvertonSA]

after trying to rollback only nginx tag to 7.71.11,

nginx:
    http:
        internalPort: 80
    https:
        internalPort: 443
    image:
        tag: 7.71.11

we get the following:

Using deprecated password for user _internal.
2024-02-11 10:03:18  [210 entrypoint-nginx.sh] Preparing to run Nginx in Docker
2024-02-11 10:03:18   [14 entrypoint-nginx.sh] Dockerfile for this image can found inside the container.
2024-02-11 10:03:18   [15 entrypoint-nginx.sh] To view the Dockerfile: 'cat /docker/nginx-artifactory-pro/Dockerfile.nginx'.
2024-02-11 10:03:18   [71 entrypoint-nginx.sh] Setting up directories if missing
2024-02-11 10:03:18  [135 entrypoint-nginx.sh] Artifactory configuration already in /var/opt/jfrog/nginx/conf.d/artifactory.conf
2024-02-11 10:03:18   [30 entrypoint-nginx.sh] SSL is set. Setting up SSL certificate and key
2024-02-11 10:03:18   [64 entrypoint-nginx.sh] /var/opt/jfrog/nginx/ssl does not exist. Creating it
2024-02-11 10:03:18   [19 entrypoint-nginx.sh] Copying auto generated SSL keys
2024-02-11 10:03:18   [24 entrypoint-nginx.sh] Copying...
cp: can't create '/var/opt/jfrog/nginx/ssl/example.key': Read-only file system
2024-02-11 10:03:18   [38 38 functions.sh] ERROR: Failed copying /etc/tls/private/example.key to /var/opt/jfrog/nginx/ssl
2024-02-11T10:03:18.757438995Z

[EvertonSA]

for the reference, the good colleagues from support helped me to figure out and it seems to be related to this line https://github.com/jfrog/charts/blob/c4bad2b31c3bb1aa8f0d3fb291d4116257e3be7e/stable/artifactory/values.yaml#L1633

I will leave this issue open, but the i'm closing the ticket. please feel free to close this issue.

[gitta-jfrog]

Hi @EvertonSA
It was a pleasure to work with you and I'm glad we managed to resolve the issue. I will ask R&D to work on this issue and provide a permanent solution soon.

Problem Description:

Following changing Nginx internal Ports to 8080/8443 in Artifactory Helm Charts, the value of Wrong X-JFrog-Override-Base-Url/X-Forwarded-Port which is based on “$server_port” changed when "nginx.service.ssloffload" is false (default)

https://github.com/jfrog/charts/blob/master/stable/artifactory/values.yaml#L1630

        {{- if .Values.nginx.service.ssloffload}}
        proxy_set_header    X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host;
        {{- else }}
        proxy_set_header    X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host:$server_port;
        proxy_set_header    X-Forwarded-Port  $server_port;
        {{- end }}

As a result, clients like Docker will redirect the requests to $host:$server_port while the "server_port" will reflect the internal port (8443/8080) which is not accessible as it is different from the external port (80/443)

What is the expected behavior?

The internal port should not impact the value of these headers.

Workaround:

Add the whole artifactoryConf section to your custom values.yaml and replace the if-else-end section with:

proxy_set_header    X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host;