achave11-ucsc
commented
5 months ago Assignee to provide reproduction/symptoms in description and to make the title more concrete.
Closed achave11-ucsc closed 1 month ago
achave11-ucsc
commented
5 months ago Assignee to provide reproduction/symptoms in description and to make the title more concrete.
achave11-ucsc
commented
5 months ago Spike to diagnose.
achave11-ucsc
commented
5 months ago @hannes-ucsc: "Too much contention on the GitLab instances at the moment. We'll be picking up the spike later."
hannes-ucsc
commented
3 months ago Consider spike now.
dsotirho-ucsc
commented
3 months ago The ciphers configured by our current EC2 AMI are:
Ciphers aes256-ctr,aes192-ctr,aes128-ctrwhile in a newer AMI this value is:
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctrSince chacha20-poly1305@openssh.com isn't listed as a supported cipher…
[ec2-user@ip-172-21-0-99 ~]$ ssh -Q cipher
3des-cbc
aes128-cbc
aes192-cbc
aes256-cbc
rijndael-cbc@lysator.liu.se
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.com
aes256-gcm@openssh.com…and is mentioned as being vulnerable to the Terrapin Attack…
The attack requires an active Man-in-the-Middle attacker that can intercept and modify the connection's traffic at the TCP/IP layer. Additionally, we require the negotiation of either ChaCha20-Poly1305, or any CBC cipher in combination with Encrypt-then-MAC as the connection's encryption mode.
If you feel uncomfortable waiting for your SSH implementation to provide a patch, you can workaround this vulnerability by temporarily disabling the affected chacha20-poly1305@openssh.com encryption and -etm@openssh.com MAC algorithms in the configuration of your SSH server (or client), and use unaffected algorithms like AES-GCM instead.
…I tried using a newer AMI with this cipher removed
Index: terraform/gitlab/gitlab.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
--- a/terraform/gitlab/gitlab.tf.json.template.py (revision d69098293eb65bcdcfed5f7f3090b533a527e0c9)
+++ b/terraform/gitlab/gitlab.tf.json.template.py (date 1717603306686)
@@ -247,12 +247,12 @@
# For instructions on finding the latest CIS-hardened AMI, see
# OPERATOR.rst#upgrading-linux-ami
#
-# CIS Amazon Linux 2 Kernel 4.14 Benchmark v2.0.0.29 - Level 1-4c096026-c6b0-440c-bd2f-6d34904e4fc6
+# CIS Amazon Linux 2 Kernel 4.14 Benchmark - Level 1 - v05 -4c096026-c6b0-440c-bd2f-6d34904e4fc6
#
ami_id = {
# FIXME: Do not update AMI to v3 until issue with OpenSSH daemon is resolved
# https://github.com/DataBiosphere/azul/issues/6082
- 'us-east-1': 'ami-02adfaf34663c8edb'
+ 'us-east-1': 'ami-0889b6cfe6c5e001e'
}
gitlab_mount = '/mnt/gitlab'
@@ -2185,6 +2185,12 @@
'--in-place',
's/curve25519[^,]*,//g',
'/etc/ssh/sshd_config'
+ ],
+ [
+ 'sed',
+ '--in-place',
+ 's/chacha20-poly1305@openssh.com,//g',
+ '/etc/ssh/sshd_config'
],
[
'systemctl',
This resulted in the EC2 instance successfully starting without the Bad SSH2 cipher error, however GitLab failed to start. Notably the gitlab-runner never succeeded while checking for jobs:
[
{
"@timestamp": "2024-06-05 01:00:02.213",
"@message": "Jun 5 01:00:02 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=deuj1fCu #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post \"https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request\": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:49846->127.0.0.11:53: i/o timeout"
},
{
"@timestamp": "2024-06-05 01:00:02.213",
"@message": "Jun 5 01:00:02 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=nJ9KdGyH #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post \"https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request\": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:49846->127.0.0.11:53: i/o timeout"
},
{
"@timestamp": "2024-06-05 01:00:06.445",
"@message": "Jun 5 01:00:02 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=xHQi2x_S #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post \"https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request\": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:49846->127.0.0.11:53: i/o timeout"
},
{
"@timestamp": "2024-06-05 01:00:23.267",
"@message": "Jun 5 01:00:23 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=xHQi2x_S #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post \"https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request\": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:56862->127.0.0.11:53: i/o timeout"
},
{
"@timestamp": "2024-06-05 01:00:23.267",
"@message": "Jun 5 01:00:23 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=nJ9KdGyH #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post \"https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request\": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:56862->127.0.0.11:53: i/o timeout"
},
…
]Assignee to continue spike to test fix
dsotirho-ucsc
commented
3 months ago Adding the --dns option to the docker run of the gitlab-runner did not seem to have an effect.
[root@ip-172-21-0-99 ~]# cat /etc/resolv.conf
options timeout:2 attempts:5
; generated by /usr/sbin/dhclient-script
search ec2.internal
nameserver 172.21.0.2
[root@ip-172-21-0-99 ~]# cat /etc/systemd/system/gitlab-runner.service
[Unit]
Description=GitLab runner service
After=docker.service gitlab-dind.service gitlab.service
Requires=docker.service gitlab-dind.service gitlab.service
[Service]
StandardOutput=null
StandardError=null
TimeoutStartSec=5min
Restart=always
ExecStartPre=-/usr/bin/docker stop gitlab-runner
ExecStartPre=-/usr/bin/docker rm gitlab-runner
ExecStartPre=/usr/bin/docker pull 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-runner:ubuntu-v16.11.1
ExecStart=/usr/bin/docker run --name gitlab-runner --rm --volume /mnt/gitlab/runner/config:/etc/gitlab-runner --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 --dns 172.21.0.2 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-runner:ubuntu-v16.11.1
[Install]
WantedBy=multi-user.target
[root@ip-172-21-0-99 ~]# systemctl status gitlab-runner
● gitlab-runner.service - GitLab runner service
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-06-06 00:10:22 UTC; 43min ago
Process: 5733 ExecStartPre=/usr/bin/docker pull 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-runner:ubuntu-v16.11.1 (code=exited, status=0/SUCCESS)
Process: 5723 ExecStartPre=/usr/bin/docker rm gitlab-runner (code=exited, status=1/FAILURE)
Process: 5683 ExecStartPre=/usr/bin/docker stop gitlab-runner (code=exited, status=1/FAILURE)
Main PID: 6043 (docker)
Tasks: 9
Memory: 10.9M
CGroup: /system.slice/gitlab-runner.service
└─6043 /usr/bin/docker run --name gitlab-runner --rm --volume /mnt/gitlab/runner/config:/etc/gitlab-runner --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 --d...
Jun 06 00:09:55 ip-172-21-0-99.ec2.internal systemd[1]: Starting GitLab runner service...
Jun 06 00:10:22 ip-172-21-0-99.ec2.internal systemd[1]: Started GitLab runner service.
[root@ip-172-21-0-99 ~]# tail -n 3 /var/log/messages
Jun 6 00:53:29 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=deuj1fCu #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post "https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:54704->127.0.0.11:53: i/o timeout
Jun 6 00:53:29 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=xHQi2x_S #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post "https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:54704->127.0.0.11:53: i/o timeout
Jun 6 00:53:29 ip-172-21-0-99 docker: gitlab-runner: #033[0;33mWARNING: Checking for jobs... failed #033[0;m #033[0;33mrunner#033[0;m=nJ9KdGyH #033[0;33mstatus#033[0;m=couldn't execute POST against https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request: Post "https://gitlab.dev.singlecell.gi.ucsc.edu/api/v4/jobs/request": dial tcp: lookup gitlab.dev.singlecell.gi.ucsc.edu on 127.0.0.11:53: read udp 127.0.0.1:54704->127.0.0.11:53: i/o timeoutThe DNS timeout in the gitlab-runner container is a bit of a red herring. Though that's still an issue, the more obvious problem is the fact that the gitlab container isn't reachable at all from outside the instance. Turns out the reason for that is that the new image adds an nftables ruleset while dockerd sets up iptables chains and rules. Since iptables-legacy is used (that's what the docker package depends on), instead of iptables-nft, the two rulesets contradict each other. I tried switching to iptables-nft and that led to the creation only a nftables ruleset (I think), but the container still wasn't reachable. The only solution I could come up with is to remove the nftables ruleset. This will violate a CIS control but we can live with that for now. The iptables ruleset provides sufficient protection, considering that the instance the container run on has no public IP.
The fix for the DNS timeout appears to be passing the --dns flag to the gitlab-dind container in a second place, so that it is passed to the dockerd daemon running inside the container, in addition to the docker client on the host. I don't think it hurts to basically pass --dns to all containers on the host. Passing it to the daemon inside the DinD container should take care of all containers started by the DinD container.
I was able to make everything work in-situ but we need to verify the changes from scratch.
Assignee to try this patch in addition to the currently applied patches.
Index: terraform/gitlab/gitlab.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
--- a/terraform/gitlab/gitlab.tf.json.template.py (revision fb58b01c339f7db17658cf10fc887b67c012e944)
+++ b/terraform/gitlab/gitlab.tf.json.template.py (date 1717985052216)
@@ -189,6 +189,12 @@
split_tunnel = not config.deployment.is_stable
+vpc_dns_servers = [
+ # https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS
+ str(nth(ipaddress.ip_network(vpc_cidr).hosts(), 1)),
+ '169.254.169.253'
+]
+
# The public key of that keypair
#
administrator_key = (
@@ -1233,11 +1239,7 @@
'server_certificate_arn': '${data.aws_acm_certificate.gitlab_vpn.arn}',
'transport_protocol': 'udp',
'split_tunnel': split_tunnel,
- 'dns_servers': [] if split_tunnel else [
- # https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS
- str(nth(ipaddress.ip_network(vpc_cidr).hosts(), 1)),
- '169.254.169.253'
- ],
+ 'dns_servers': [] if split_tunnel else vpc_dns_servers,
'authentication_options': {
'type': 'certificate-authentication',
'root_certificate_chain_arn': '${data.aws_acm_certificate.gitlab_vpn.arn}'
@@ -1741,10 +1743,24 @@
# container have a functional non-localhost
# DNS server and don't fall back to the
# Google ones.
+ #
+ # TBD: It is unclear if mounting resolv.conf
+ # is needed in conjunction with the
+ # --dns flag passed below. The --dns
+ # flag appears to override the use of
+ # the embedded server for containers
+ # launched by the DinD daemon. It's
+ # just not clear if that applies to all
+ # such containers or just those
+ # launched with a custom network.
+ #
'--volume /etc/resolv.conf:/etc/resolv.conf',
f'--volume {gitlab_mount}/docker:/var/lib/docker',
f'--volume {gitlab_mount}/runner/config:/etc/gitlab-runner',
- str(dind_image)
+ str(dind_image),
+ # TODO: Explanatory comment
+ # TODO: Loop over the list
+ f'--dns {vpc_dns_servers[0]}'
),
'[Install]',
'WantedBy=multi-user.target',
@@ -1769,6 +1785,8 @@
'ExecStartPre=-/usr/bin/docker stop gitlab',
'ExecStartPre=-/usr/bin/docker rm gitlab',
'ExecStartPre=/usr/bin/docker pull ' + str(gitlab_image),
+ # TODO: Explanatory comment
+ 'ExecStartPre=nft flush ruleset',
jw(
'ExecStart=/usr/bin/docker',
'run',I was able to make everything work in-situ …
As @dsotirho-ucsc observed, that is not the case. The IT job that I triggered after making the changes failed with
Running with gitlab-runner 16.11.1 (535ced5f)
on Azul #2 xHQi2x_S, system ID: r_HXUshJtjLOSP
Preparing the "docker" executor 00:09
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at tcp://gitlab-dind:2375. Is the docker daemon running? (docker.go:950:0s)
Will be retried in 3s ...
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at tcp://gitlab-dind:2375. Is the docker daemon running? (docker.go:950:0s)
Will be retried in 3s ...
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at tcp://gitlab-dind:2375. Is the docker daemon running? (docker.go:950:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Cannot connect to the Docker daemon at tcp://gitlab-dind:2375. Is the docker daemon running? (docker.go:950:0s)Nevertheless, we'll still proceed as planned. If/when we can reproduce this new issue, I will investigate in situ.
achave11-ucsc
commented
2 months ago @achave11-ucsc to take over from @dsotirho-ucsc, who will provide the complete patch for the experiment.
dsotirho-ucsc
commented
2 months ago Index: terraform/gitlab/gitlab.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
--- a/terraform/gitlab/gitlab.tf.json.template.py (revision 4d8352e6e9bc3a967faaffc0ba00d8f053ffb464)
+++ b/terraform/gitlab/gitlab.tf.json.template.py (date 1718139754437)
@@ -189,6 +189,12 @@
split_tunnel = not config.deployment.is_stable
+vpc_dns_servers = [
+ # https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS
+ str(nth(ipaddress.ip_network(vpc_cidr).hosts(), 1)),
+ '169.254.169.253'
+]
+
# The public key of that keypair
#
administrator_key = (
@@ -247,12 +253,12 @@
# For instructions on finding the latest CIS-hardened AMI, see
# OPERATOR.rst#upgrading-linux-ami
#
-# CIS Amazon Linux 2 Kernel 4.14 Benchmark v2.0.0.29 - Level 1-4c096026-c6b0-440c-bd2f-6d34904e4fc6
+# CIS Amazon Linux 2 Kernel 4.14 Benchmark - Level 1 - v05 -4c096026-c6b0-440c-bd2f-6d34904e4fc6
#
ami_id = {
# FIXME: Do not update AMI to v3 until issue with OpenSSH daemon is resolved
# https://github.com/DataBiosphere/azul/issues/6082
- 'us-east-1': 'ami-02adfaf34663c8edb'
+ 'us-east-1': 'ami-0889b6cfe6c5e001e'
}
gitlab_mount = '/mnt/gitlab'
@@ -1233,11 +1239,7 @@
'server_certificate_arn': '${data.aws_acm_certificate.gitlab_vpn.arn}',
'transport_protocol': 'udp',
'split_tunnel': split_tunnel,
- 'dns_servers': [] if split_tunnel else [
- # https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#AmazonDNS
- str(nth(ipaddress.ip_network(vpc_cidr).hosts(), 1)),
- '169.254.169.253'
- ],
+ 'dns_servers': [] if split_tunnel else vpc_dns_servers,
'authentication_options': {
'type': 'certificate-authentication',
'root_certificate_chain_arn': '${data.aws_acm_certificate.gitlab_vpn.arn}'
@@ -1741,10 +1743,27 @@
# container have a functional non-localhost
# DNS server and don't fall back to the
# Google ones.
+ #
+ # TBD: It is unclear if mounting resolv.conf
+ # is needed in conjunction with the
+ # --dns flag passed below. The --dns
+ # flag appears to override the use of
+ # the embedded server for containers
+ # launched by the DinD daemon. It's
+ # just not clear if that applies to all
+ # such containers or just those
+ # launched with a custom network.
+ #
'--volume /etc/resolv.conf:/etc/resolv.conf',
f'--volume {gitlab_mount}/docker:/var/lib/docker',
f'--volume {gitlab_mount}/runner/config:/etc/gitlab-runner',
- str(dind_image)
+ *[f'--dns {s}' for s in vpc_dns_servers],
+ str(dind_image),
+ # Adding the `--dns` option after the image
+ # argument passes it as a custom daemon flag
+ # to be passed to the dockerd daemon running
+ # inside the container.
+ *[f'--dns {s}' for s in vpc_dns_servers]
),
'[Install]',
'WantedBy=multi-user.target',
@@ -1769,6 +1788,8 @@
'ExecStartPre=-/usr/bin/docker stop gitlab',
'ExecStartPre=-/usr/bin/docker rm gitlab',
'ExecStartPre=/usr/bin/docker pull ' + str(gitlab_image),
+ # TODO: Explanatory comment
+ 'ExecStartPre=nft flush ruleset',
jw(
'ExecStart=/usr/bin/docker',
'run',
@@ -1781,6 +1802,7 @@
f'--volume {gitlab_mount}/config:/etc/gitlab',
f'--volume {gitlab_mount}/logs:/var/log/gitlab',
f'--volume {gitlab_mount}/data:/var/opt/gitlab',
+ *[f'--dns {s}' for s in vpc_dns_servers],
str(gitlab_image)
),
'[Install]',
@@ -1814,6 +1836,7 @@
f'--volume {gitlab_mount}/runner/config:/etc/gitlab-runner',
'--network gitlab-runner-net',
'--env DOCKER_HOST=tcp://gitlab-dind:2375',
+ *[f'--dns {s}' for s in vpc_dns_servers],
str(runner_image)
),
'[Install]',
@@ -1847,6 +1870,7 @@
'--volume /var/run/docker.sock:/var/run/docker.sock',
'--volume /:/scan:ro',
f'--volume {gitlab_mount}/clamav:/var/lib/clamav:rw',
+ *[f'--dns {s}' for s in vpc_dns_servers],
str(clamav_image),
'/bin/sh',
'-c',
@@ -2185,6 +2209,13 @@
'--in-place',
's/curve25519[^,]*,//g',
'/etc/ssh/sshd_config'
+ ],
+ [
+ # OpenSSH fails to start with the chacha20 cipher enabled
+ 'sed',
+ '--in-place',
+ 's/chacha20-poly1305@openssh.com,//g',
+ '/etc/ssh/sshd_config'
],
[
'systemctl',EDIT: @achave11-ucsc
… version v3.0.0.1 of the CIS-hardened image. A CIS support ticket (SUPPORT-33122) was created in an effort to resolve this issue.
The following screenshot exhibits this, emphasize the line
sshd: /etc/ssh/sshd_config line 154: Bad SSH2 cipher spec 'chacha20-poly1305@openssh.com,aes256-gcm@openssh. com,aes128-gcm@openssh.com, aes256-ctr, aes192-ctr, aes128-ctr'