Closed Nick-Harvey closed 6 years ago
how does your nginx config look like? who is serving the static content nginx or ruby/puma?
Did you set env var RAILS_SERVE_STATIC_FILES=true
if you serve your static content via ruby.
This is probably be a bug because I set the API_ROOT_URL based on root_url
and that returns http://
instead of https://
.
nginx.conf:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "nginx.fullname" . }}
labels:
app: {{ template "name" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
conf: |
# This file is largely based on the one written by @Djelibeybi in:
# https://github.com/Djelibeybi/Portus-On-OracleLinux7/
events {
worker_connections 1024;
}
http {
default_type application/octet-stream;
charset UTF-8;
# Some basic config.
server_tokens off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# On timeouts.
keepalive_timeout 65;
client_header_timeout 240;
client_body_timeout 240;
fastcgi_read_timeout 249;
reset_timedout_connection on;
## Set a variable to help us decide if we need to add the
## 'Docker-Distribution-Api-Version' header.
## The registry always sets this header.
## In the case of nginx performing auth, the header will be unset
## since nginx is auth-ing before proxying.
map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
'' 'registry/2.0';
}
upstream {{ template "portus.fullname" . }} {
least_conn;
server {{ template "portus.fullname" . }}:{{ .Values.portus.service.port }} max_fails=3 fail_timeout=15s;
}
upstream {{ template "registry.fullname" . }}:{{ .Values.registry.service.port }} {
least_conn;
server {{ template "registry.fullname" . }}:{{ .Values.registry.service.port }} max_fails=3 fail_timeout=15s;
}
server {
listen 443 ssl default http2;
listen 80 default;
server_name {{ template "nginx.fullname" . }};
#{{- if .Values.portus.tls.enabled }}
#listen {{ .Values.nginx.service.port }} ssl http2;
##
# SSL
#ssl on;
# Certificates
ssl_certificate /certificates/portus.crt;
ssl_certificate_key /certificates/portus.key;
# Enable session resumption to improve https performance
#
# http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Enables server-side protection from BEAST attacks
# http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html
ssl_prefer_server_ciphers on;
# Disable SSLv3 (enabled by default since nginx 0.8.19)
# since it's less secure than TLS
# http://en.wikipedia.org/wiki/Secure_Sockets_Layer#SSL_3.0
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# Ciphers chosen for forward secrecy and compatibility.
#
# http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
#{{- else }}
#listen {{ .Values.nginx.service.port }} http2;
#{{- end }}
##
# Docker-specific stuff.
proxy_set_header Host $http_host; # required for Docker client sake
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486
# (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
##
# Custom headers.
# Adding HSTS[1] (HTTP Strict Transport Security) to avoid SSL stripping[2].
#
# [1] https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security
# [2] https://en.wikipedia.org/wiki/SSL_stripping#SSL_stripping
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# Don't allow the browser to render the page inside a frame or iframe
# and avoid Clickjacking. More in the following link:
#
# https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
add_header X-Frame-Options DENY;
# Disable content-type sniffing on some browsers.
add_header X-Content-Type-Options nosniff;
# This header enables the Cross-site scripting (XSS) filter built into
# most recent web browsers. It's usually enabled by default anyway, so the
# role of this header is to re-enable the filter for this particular
# website if it was disabled by the user.
add_header X-XSS-Protection "1; mode=block";
# Add header for IE in compatibility mode.
add_header X-UA-Compatible "IE=edge";
# Redirect (most) requests to /v2/* to the Docker Registry
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
## If $docker_distribution_api_version is empty, the header will not be added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "registry.fullname" . }}:{{ .Values.registry.service.port }};
{{- else }}
proxy_pass http://{{ template "registry.fullname" . }}:{{ .Values.registry.service.port }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
# Portus needs to handle /v2/token for authentication
location = /v2/token {
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "portus.fullname" . }};
{{- else }}
proxy_pass http://{{ template "portus.fullname" . }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
# Portus needs to handle /v2/webhooks/events for notifications
location = /v2/webhooks/events {
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "portus.fullname" . }};
{{- else }}
proxy_pass http://{{ template "portus.fullname" . }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
# Portus handles everything else for the UI
location / {
try_files $uri/index.html $uri.html $uri @{{ template "portus.fullname" . }};
# Nginx doesn't support nested If statements, so we
# concatenate compound conditions on the $cors variable
# and process later
# If request comes from allowed subdomain
# (*.foo.bar.com) then we enable CORS
if ($http_origin ~* (https?:\/\/(.[^?]*\.)?foo.bar\com(:[0-9]+)?$)) {
set $cors "1";
}
# OPTIONS indicates a CORS pre-flight request
if ($request_method = 'OPTIONS') {
set $cors "${cors}o";
}
# Append CORS headers to any request from
# allowed CORS domain, except OPTIONS
if ($cors = "1") {
add_header 'Access-Control-Allow-Origin: $http_origin' always;
add_header 'Access-Control-Allow-Credentials: true' always;
proxy_pass http://{{ template "portus.fullname" . }};
}
# OPTIONS (pre-flight) request from allowed
# CORS domain. return response directly
if ($cors = "1o") {
add_header 'Access-Control-Allow-Origin: $http_origin' always;
add_header 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE' always;
add_header 'Access-Control-Allow-Credentials: true' always;
add_header 'Access-Control-Allow-Headers: Origin,Content-Type,Accept' always;
add_header Content-Length 0;
add_header Content-Type text/plain;
return 204;
}
# Requests from non-allowed CORS domains
proxy_pass http://{{ template "portus.fullname" . }};
}
location @{{ template "portus.fullname" . }} {
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "portus.fullname" . }};
{{- else }}
proxy_pass http://{{ template "portus.fullname" . }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
}
}
Portus configmap: RAILS_SERVE_STATIC_FILES: "true"
Is there a workaround I can implement to fix the root_url issue?
@Nick-Harvey you could replace #{root_url}
with https://mydomain.com/
. That is not ideal but it would unblock you until we provide a proper fix. Lemme know if it worked. :)
And thanks for reporting this!
What about The Protocol-relative URL?
so instead of http://
one could just use //
which will automatically select the Protocol. Will that work as well in that case?
@Vad1mo yeah, I already found a solution by using request.env['HTTP_HOST']
. I'll open a PR and wait for @mssola feedback.
Just waiting for @Nick-Harvey feedback to know if that fixes the issue.
The suggested fix did seem to fix the errors on the script, but now it's throwing SSL errors
"OpenSSL::SSL::SSLError: could not stablish connection: SSL error. You can skip this check by clicking on the "Skip remote checks" checkbox."
SSL cert (signed by external authority) is packaged into the app on build in /certificates/portus.crt and /certificates/portus.key
SSL certs are also applied in the values.yaml file in the helm chart. I'll post a scrubbed version of haml here in a second
Helms value file
## Default values for Portus Helm Chart.
## This is a YAML-formatted file.
## Declare variables to be passed into your templates.
## Default values for Portus
##
portus:
replicas: 1
## Image configuration.
##
# image:
# repository: "opensuse/portus"
# tag: "head"
# pullPolicy: "IfNotPresent"
image:
repository: "foo.bar.com:portus"
tag: "latest"
pullPolicy: "Always"
## Service configuration.
##
service:
port: "3000"
## Resource configuration.
##
resources:
requests:
memory: "512Mi"
cpu: "300m"
## Portus database configuration.
##
productionDatabase: "portusdevreg" # name of database
productionUsername: "portus" # database user
## only set productionHost and productionPassword if relying on existing mariadb deployment
##
productionHost: <gcloud SQL server>
productionPassword: <scrubbed>
## Defaults to a random 10-character alphanumeric string if not set
##
password: <scurbbed>
## Defaults to a random 128-character alphanumeric string if not set
##
secretKeyBase: <scrubbed>
## TLS configuration
## the internal host names of the portus, registry and nginx service must be covered by the key/cert in order for TLS to work properly
##
tls:
enabled: true
## must include key if using tls
##
cert: |
<pasted cert (fullchain) is here>
## must include certificate if using tls
##
key: |
<pasted key here>
## Default values for Crono.
##
crono:
replicas: 1
## image configuration.
##
image:
repository: "opensuse/portus"
tag: "latest"
pullPolicy: "IfNotPresent"
## Resource configuration.
##
resources:
requests:
memory: "512Mi"
cpu: "300m"
## Default values for Docker Registry.
##
registry:
replicas: 1
mountPath: "/gcs"
## persistence configuration.
##
persistence:
enabled: false
accessMode: "ReadWriteOnce"
size: "10Gi"
## image configuration.
# #
image:
repository: "library/registry"
tag: "2.5.2"
pullPolicy: "IfNotPresent"
## Service configuration.
##
service:
port: "5000"
debugPort: "5001"
## Resource configuration.
##
resources:
requests:
memory: "512Mi"
cpu: "300m"
## Default values for Nginx.
##
nginx:
replicas: 1
## image configuration.
##
image:
repository: "library/nginx"
tag: "alpine"
pullPolicy: "IfNotPresent"
## Service configuration.
##
service:
## Set to ClusterIP if using ingress, or NodePort if using minikube
##
type: "ClusterIP"
port: "80"
# nodePort:
## in order to access the docker registry from outside of the cluster
## if ingress is enabled set host to the domain you are using
## if NodePort is being used set host to the ip address of a cluster node
##
## ingress configuration.
##
ingress:
enabled: true
## Anntations to be added to the web ingress
##
annotations:
kubernetes.io/ingress.class: "nginx"
#ingress.kubernetes.io/force-ssl-redirect: "true"
kubernetes.io/ssl-redirect: "true"
#kubernetes.io/enable-cors: "true"
#ingress.kubernetes.io/force-ssl-redirect: "true"
#kubernetes.io/tls-acme: "true"
#kubernetes.io/X-Forwarded-Proto: "https"
#kubernetes.io/ingress.global-static-ip-name: devreg-static-ip
## TLS configuration
## the ingress host must be covered by the key/cert in order for TLS to work properly
##
tls:
enabled: true
## Secrets containing SSL key and cert must be manually created in the namespace
##
secretName: "portus-tls"
host: "foo.bar.com"
## Resource configuration.
##
resources:
requests:
memory: "512Mi"
cpu: "300m"
## Default values for Mariadb.
##
mariadb:
## Use Mariadb chart dependency
## Set to false if using your own Mariadb
##
enabled: false
## persistence configuration.
##
persistence:
enabled: true
accessMode: "ReadWriteOnce"
size: "8Gi"
## Configuration values for Mariadb.
## must match Portus database values.
##
mariadbUser: "portus"
mariadbDatabase: "portus"
Since the CORS road I was heading down I went ahead and altered my nginx config back to the original one supplied in the helm chart
new nginx file
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "nginx.fullname" . }}
labels:
app: {{ template "name" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
conf: |
# This file is largely based on the one written by @Djelibeybi in:
# https://github.com/Djelibeybi/Portus-On-OracleLinux7/
events {
worker_connections 1024;
}
http {
default_type application/octet-stream;
charset UTF-8;
# Some basic config.
server_tokens off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# On timeouts.
keepalive_timeout 65;
client_header_timeout 240;
client_body_timeout 240;
fastcgi_read_timeout 249;
reset_timedout_connection on;
## Set a variable to help us decide if we need to add the
## 'Docker-Distribution-Api-Version' header.
## The registry always sets this header.
## In the case of nginx performing auth, the header will be unset
## since nginx is auth-ing before proxying.
map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
'' 'registry/2.0';
}
upstream {{ template "portus.fullname" . }} {
least_conn;
server {{ template "portus.fullname" . }}:{{ .Values.portus.service.port }} max_fails=3 fail_timeout=15s;
}
upstream {{ template "registry.fullname" . }}:{{ .Values.registry.service.port }} {
least_conn;
server {{ template "registry.fullname" . }}:{{ .Values.registry.service.port }} max_fails=3 fail_timeout=15s;
}
server {
listen 443 ssl default http2;
listen 80 default;
server_name {{ template "nginx.fullname" . }};
#{{- if .Values.portus.tls.enabled }}
#listen {{ .Values.nginx.service.port }} ssl http2;
##
# SSL
#ssl on;
# Certificates
ssl_certificate /certificates/portus.crt;
ssl_certificate_key /certificates/portus.key;
# Enable session resumption to improve https performance
#
# http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Enables server-side protection from BEAST attacks
# http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html
ssl_prefer_server_ciphers on;
# Disable SSLv3 (enabled by default since nginx 0.8.19)
# since it's less secure than TLS
# http://en.wikipedia.org/wiki/Secure_Sockets_Layer#SSL_3.0
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# Ciphers chosen for forward secrecy and compatibility.
#
# http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
#{{- else }}
#listen {{ .Values.nginx.service.port }} http2;
#{{- end }}
##
# Docker-specific stuff.
proxy_set_header Host $http_host; # required for Docker client sake
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486
# (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
##
# Custom headers.
# Adding HSTS[1] (HTTP Strict Transport Security) to avoid SSL stripping[2].
#
# [1] https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security
# [2] https://en.wikipedia.org/wiki/SSL_stripping#SSL_stripping
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# Don't allow the browser to render the page inside a frame or iframe
# and avoid Clickjacking. More in the following link:
#
# https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
add_header X-Frame-Options DENY;
# Disable content-type sniffing on some browsers.
add_header X-Content-Type-Options nosniff;
# This header enables the Cross-site scripting (XSS) filter built into
# most recent web browsers. It's usually enabled by default anyway, so the
# role of this header is to re-enable the filter for this particular
# website if it was disabled by the user.
add_header X-XSS-Protection "1; mode=block";
# Add header for IE in compatibility mode.
add_header X-UA-Compatible "IE=edge";
# Redirect (most) requests to /v2/* to the Docker Registry
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
## If $docker_distribution_api_version is empty, the header will not be added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "registry.fullname" . }}:{{ .Values.registry.service.port }};
{{- else }}
proxy_pass http://{{ template "registry.fullname" . }}:{{ .Values.registry.service.port }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
# Portus needs to handle /v2/token for authentication
location = /v2/token {
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "portus.fullname" . }};
{{- else }}
proxy_pass http://{{ template "portus.fullname" . }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
# Portus needs to handle /v2/webhooks/events for notifications
location = /v2/webhooks/events {
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "portus.fullname" . }};
{{- else }}
proxy_pass http://{{ template "portus.fullname" . }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
# Portus handles everything else for the UI
location / {
try_files $uri/index.html $uri.html $uri @{{ template "portus.fullname" . }};
}
location @{{ template "portus.fullname" . }} {
{{- if .Values.portus.tls.enabled }}
proxy_pass https://{{ template "portus.fullname" . }};
{{- else }}
proxy_pass http://{{ template "portus.fullname" . }};
{{- end }}
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
proxy_buffering on;
}
}
}
The suggested fix did seem to fix the errors on the script, but now it's throwing SSL errors
👍
@vitoravelino I emailed the email address listed on your profile for bounty details. just wanted to let you know incase it wasn't something you checked regularly.
@Nick-Harvey Just realized that my email is not working and I don't know for how long. But don't worry about it!
@Nick-Harvey We are closing this issue because it was initially related to the mixed content issue that now it's fixed in master.
Feel free to reopen it or open a new issue if you face any other issue. Thanks!
I've been trying to get to the bottom of this for a better part of a week now and can't seem to squash it. I've even put a bounty out on it on stackoverflow and still no dice.
here's the stackoverflow issue: https://stackoverflow.com/questions/48001934/mixed-content-error-nginx-ingress-in-kubernetes-for-rails-app
Further Info: I'm running the latest master branch - last merge from upstream was a few hours ago - and deploying the application via the helm chart. All pretty standard, except on thing. I've added some custom html into the app (minor stuff like company logo, custom contact information) in case any of our users need to report something. I build the app via the secure compose file in examples, upload the image to gcr.io. However, no matter how I configure it, no matter what ingress annotations I use, I get Mixed Content errors, and if I disable the warning via crome I get preflight errors. SSL certs are loaded into the docker image
Mixed Content error:
Mixed Content: The page at 'https://foo.bar.com/admin/registries/new' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://foo.bar.ai//api/ v1/registries/validate?name=reg2&hostname=reg-foobar-registry%3A5000&external_hostname=&use_ssl=true&force=false&only%5B%5D=hostname'. This request has been blocked; the content must be served over HTTPS.
Tell chrome to load the unsafe scripts and I get the following:
Failed to load http://foo.bar.com//api/v1/registries/validate?name=reg&hostname=&external_hostname=&use_ssl=false&force=false&only%5B%5D=name: Response for preflight is invalid (redirect)
From the looks of things, even though rails is deployed in a prod environment (based on the init script from the official docker portus image)
I'm really trying to get this rolled out for my team (pre-prod environment) and this is the last hurdle. I'm willing to pay $50 (ethereum or bitcoin) to the person who can help me get this thing solved. :)