search

caddyserver / ingress

WIP Caddy 2 ingress controller for Kubernetes
Apache License 2.0
620 stars 70 forks source link

Caddy ingress might be hackable? #196

Closed timthelion closed 7 months ago

timthelion commented 7 months ago

Hello,

I'm reaching out to you without clear knowledge of what is going on. A few hours ago I noticed that some of the lets-encrypt renewal notices that I was receiving were for subdomains that were not ours. When I looked into the logs for Caddy ingress, it seems that Caddy is trying to create subdomains that are not ours:

{"level":"info","ts":1704019228.780658,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["mobile.app.vpn.gitlab.dashboard.igmirs-heidelb.dopracenakole.cz"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"timothy.hobbs@auto-mat.cz"}
{"level":"error","ts":1704019229.319874,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"mobile.app.vpn.gitlab.dashboard.igmirs-heidelb.dopracenakole.cz","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for \"dopracenakole.cz\". Retry after 2024-01-01T07:00:00Z: see https://letsencrypt.org/docs/rate-limits/"}
{"level":"error","ts":1704019229.3199372,"logger":"tls.obtain","msg":"will retry","error":"[mobile.app.vpn.gitlab.dashboard.igmirs-heidelb.dopracenakole.cz] Obtain: [mobile.app.vpn.gitlab.dashboard.igmirs-heidelb.dopracenakole.cz] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for \"dopracenakole.cz\". Retry after 2024-01-01T07:00:00Z: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.556682216,"max_duration":2592000}
I1231 10:40:29.947594       1 request.go:665] Waited for 1.192409301s due to client-side throttling, not priority and fairness, request: GET:https://10.245.0.1:443/apis/coordination.k8s.io/v1/namespaces/caddy-system/leases/caddy-lock-issue.cert.wiki.app.support.support.support.cdn.hfgs.dopracenakole.cz
.... tones of similar messages ...
{"level":"error","ts":1704019370.1456294,"logger":"tls.obtain","msg":"will retry","error":"[demo.static.cms.san.ab876a.fr.hostmaster.hfgs.dopracenakole.cz] Obtain: [demo.static.cms.san.ab876a.fr.hostmaster.hfgs.dopracenakole.cz] creating new order: attempt 1: https://acme-staging-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":61.390685831,"max_duration":2592000}
{"level":"error","ts":1704019377.7821412,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.git.phpmyadmin.ns1.mail.phpmyadmin.en-za.dopracenakole.cz","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/"}
{"level":"error","ts":1704019377.7821918,"logger":"tls.obtain","msg":"will retry","error":"[www.git.phpmyadmin.ns1.mail.phpmyadmin.en-za.dopracenakole.cz] Obtain: [www.git.phpmyadmin.ns1.mail.phpmyadmin.en-za.dopracenakole.cz] creating new order: attempt 1: https://acme-staging-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":61.227525444,"max_duration":2592000}
I1231 10:43:03.957629       1 request.go:665] Waited for 1.001353621s due to client-side throttling, not priority and fairness, request: GET:https://10.245.0.1:443/apis/coordination.k8s.io/v1/namespaces/caddy-system/leases/caddy-lock-issue.cert.mobile.app.vpn.gitlab.dashboard.igmirs-heidelb.dopracenakole.cz

I'm not yet sure what is going on. You can find our configs here: https://github.com/auto-mat/k8s/tree/master/manifests/ingress I will update this issue with more information as I discover it.

timthelion commented 7 months ago

So I looked through the hosts here and it seems the k8s config is in tact:

image

There are no extra hosts here. So somehow Caddy is being re-configured directly.

timthelion commented 7 months ago

This is in one of the infected pods:

/ # ps
PID   USER     TIME  COMMAND
    1 root      4d01 /ingress-controller -config-map=caddy-ingress-controller-configma
   31 root      0:00 /bin/sh
   37 root      0:00 ps
/ #
timthelion commented 7 months ago

This is in one of the infected pods:

/ # netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:2019          0.0.0.0:*               LISTEN      1/ingress-controlle
tcp        0      0 :::443                  :::*                    LISTEN      1/ingress-controlle
tcp        0      0 :::80                   :::*                    LISTEN      1/ingress-controlle
tcp        0      0 :::9765                 :::*                    LISTEN      1/ingress-controlle
/ #
timthelion commented 7 months ago 
/ # lsof
1   /ingress-controller 0   /dev/null
1   /ingress-controller 1   pipe:[47445]
1   /ingress-controller 2   pipe:[47446]
1   /ingress-controller 3   socket:[190654291]
1   /ingress-controller 4   anon_inode:[eventpoll]
1   /ingress-controller 5   pipe:[47532]
1   /ingress-controller 6   pipe:[47532]
1   /ingress-controller 7   socket:[47537]
1   /ingress-controller 8   socket:[47544]
1   /ingress-controller 9   socket:[47545]
1   /ingress-controller 10  socket:[47546]
1   /ingress-controller 11  socket:[190633594]
1   /ingress-controller 12  socket:[190604276]
1   /ingress-controller 13  socket:[190602688]
1   /ingress-controller 14  socket:[190660781]
1   /ingress-controller 15  socket:[190660783]
1   /ingress-controller 16  socket:[190608159]
1   /ingress-controller 17  socket:[190603765]
1   /ingress-controller 18  socket:[190634220]
1   /ingress-controller 19  socket:[190655485]
1   /ingress-controller 20  socket:[190633365]
1   /ingress-controller 21  socket:[190656098]
1   /ingress-controller 22  socket:[190652964]
1   /ingress-controller 23  socket:[190600617]
1   /ingress-controller 24  socket:[190609695]
1   /ingress-controller 25  socket:[190660790]
1   /ingress-controller 26  socket:[190631092]
1   /ingress-controller 27  socket:[190598773]
1   /ingress-controller 28  socket:[190613923]
1   /ingress-controller 29  socket:[188766636]
1   /ingress-controller 30  socket:[190656327]
1   /ingress-controller 31  socket:[190611364]
1   /ingress-controller 32  socket:[190598782]
1   /ingress-controller 33  socket:[190633292]
1   /ingress-controller 34  socket:[190613236]
1   /ingress-controller 35  socket:[190608983]
1   /ingress-controller 36  socket:[190603398]
1   /ingress-controller 37  socket:[190610226]
1   /ingress-controller 38  socket:[190635235]
1   /ingress-controller 39  socket:[190635665]
1   /ingress-controller 40  socket:[190635635]
1   /ingress-controller 41  socket:[190634437]
1   /ingress-controller 42  socket:[190616057]
1   /ingress-controller 43  socket:[190618537]
1   /ingress-controller 44  socket:[190632504]
1   /ingress-controller 45  socket:[190654062]
1   /ingress-controller 46  socket:[190651936]
1   /ingress-controller 48  socket:[190604063]
1   /ingress-controller 49  socket:[190615049]
1   /ingress-controller 50  socket:[190604074]
1   /ingress-controller 51  socket:[190606971]
1   /ingress-controller 52  socket:[190656232]
1   /ingress-controller 53  socket:[190654007]
1   /ingress-controller 54  socket:[190658067]
1   /ingress-controller 55  socket:[190634204]
1   /ingress-controller 56  socket:[190633804]
1   /ingress-controller 57  socket:[190604374]
1   /ingress-controller 58  socket:[190659875]
1   /ingress-controller 59  socket:[190611312]
1   /ingress-controller 60  socket:[190602793]
1   /ingress-controller 61  socket:[190606433]
1   /ingress-controller 62  socket:[190634174]
1   /ingress-controller 63  socket:[190607818]
1   /ingress-controller 64  socket:[190649277]
1   /ingress-controller 65  socket:[190606418]
1   /ingress-controller 66  socket:[190630779]
1   /ingress-controller 67  socket:[190607716]
1   /ingress-controller 68  socket:[190635115]
1   /ingress-controller 69  socket:[190613925]
1   /ingress-controller 71  socket:[190634811]
1   /ingress-controller 72  socket:[190613129]
1   /ingress-controller 73  socket:[190634772]
1   /ingress-controller 74  socket:[190615580]
1   /ingress-controller 75  socket:[190653158]
1   /ingress-controller 76  socket:[190615029]
1   /ingress-controller 77  socket:[190635813]
1   /ingress-controller 78  socket:[190611157]
1   /ingress-controller 79  socket:[190651793]
1   /ingress-controller 80  socket:[190652971]
1   /ingress-controller 81  socket:[190634989]
1   /ingress-controller 82  socket:[190651003]
1   /ingress-controller 83  socket:[190601859]
1   /ingress-controller 84  socket:[190614458]
1   /ingress-controller 85  socket:[190654293]
1   /ingress-controller 86  socket:[190608006]
1   /ingress-controller 88  socket:[190617709]
1   /ingress-controller 89  socket:[190618016]
1   /ingress-controller 90  socket:[190651152]
1   /ingress-controller 91  socket:[190653149]
1   /ingress-controller 92  socket:[190655036]
1   /ingress-controller 93  socket:[190652947]
1   /ingress-controller 94  socket:[190653208]
1   /ingress-controller 95  socket:[190654017]
1   /ingress-controller 96  socket:[190602871]
1   /ingress-controller 97  socket:[190601879]
1   /ingress-controller 98  socket:[190655829]
1   /ingress-controller 99  socket:[190611662]
1   /ingress-controller 100 socket:[190604348]
1   /ingress-controller 101 socket:[190614425]
1   /ingress-controller 102 socket:[190606265]
1   /ingress-controller 103 socket:[190633573]
1   /ingress-controller 104 socket:[190631292]
1   /ingress-controller 105 socket:[190605529]
1   /ingress-controller 106 socket:[190656332]
1   /ingress-controller 107 socket:[190630404]
1   /ingress-controller 108 socket:[190604283]
1   /ingress-controller 109 socket:[190649916]
1   /ingress-controller 110 socket:[190606986]
1   /ingress-controller 111 socket:[190635513]
1   /ingress-controller 112 socket:[190658595]
1   /ingress-controller 113 socket:[190633773]
1   /ingress-controller 114 socket:[190602681]
1   /ingress-controller 115 socket:[190634774]
1   /ingress-controller 116 socket:[190610196]
1   /ingress-controller 117 socket:[190635608]
1   /ingress-controller 118 socket:[190614776]
1   /ingress-controller 119 socket:[190616793]
1   /ingress-controller 121 socket:[190603548]
1   /ingress-controller 122 socket:[190617711]
1   /ingress-controller 123 socket:[190603230]
1   /ingress-controller 124 socket:[190614929]
1   /ingress-controller 125 socket:[190634788]
1   /ingress-controller 126 socket:[190614431]
1   /ingress-controller 127 socket:[190634795]
1   /ingress-controller 128 socket:[190654726]
1   /ingress-controller 129 socket:[190604946]
1   /ingress-controller 130 socket:[190652911]
1   /ingress-controller 131 socket:[190618054]
1   /ingress-controller 132 socket:[190657199]
1   /ingress-controller 133 socket:[190601815]
1   /ingress-controller 134 socket:[190604129]
1   /ingress-controller 135 socket:[190654024]
1   /ingress-controller 136 socket:[190658601]
1   /ingress-controller 137 socket:[190650073]
1   /ingress-controller 138 socket:[190630405]
1   /ingress-controller 139 socket:[190654037]
1   /ingress-controller 140 socket:[190633439]
1   /ingress-controller 141 socket:[190653914]
1   /ingress-controller 142 socket:[190601836]
1   /ingress-controller 143 socket:[190654717]
1   /ingress-controller 144 socket:[190633450]
1   /ingress-controller 145 socket:[190615039]
1   /ingress-controller 146 socket:[190613785]
1   /ingress-controller 147 socket:[190633579]
1   /ingress-controller 148 socket:[190607024]
1   /ingress-controller 149 socket:[190650421]
1   /ingress-controller 150 socket:[190614433]
1   /ingress-controller 151 socket:[190649192]
1   /ingress-controller 152 socket:[190654742]
1   /ingress-controller 153 socket:[190649289]
1   /ingress-controller 154 socket:[190608279]
1   /ingress-controller 156 socket:[190610390]
1   /ingress-controller 157 socket:[190649201]
1   /ingress-controller 158 socket:[190603454]
1   /ingress-controller 159 socket:[190597927]
1   /ingress-controller 160 socket:[190612514]
1   /ingress-controller 161 socket:[190635752]
1   /ingress-controller 162 socket:[190603572]
1   /ingress-controller 163 socket:[190635521]
1   /ingress-controller 164 socket:[190633776]
1   /ingress-controller 165 socket:[190654737]
1   /ingress-controller 166 socket:[190635556]
1   /ingress-controller 167 socket:[190613947]
1   /ingress-controller 168 socket:[190654039]
1   /ingress-controller 169 socket:[190609596]
1   /ingress-controller 170 socket:[190635557]
1   /ingress-controller 171 socket:[190604085]
1   /ingress-controller 172 socket:[190651109]
1   /ingress-controller 173 socket:[190658671]
1   /ingress-controller 175 socket:[190635630]
1   /ingress-controller 176 socket:[190658178]
1   /ingress-controller 177 socket:[190613685]
1   /ingress-controller 178 socket:[190652038]
1   /ingress-controller 179 socket:[190658491]
1   /ingress-controller 180 socket:[190657146]
1   /ingress-controller 181 socket:[190650819]
1   /ingress-controller 182 socket:[190653027]
1   /ingress-controller 183 socket:[190610303]
1   /ingress-controller 184 socket:[190608293]
1   /ingress-controller 185 socket:[190657201]
1   /ingress-controller 186 socket:[190651700]
1   /ingress-controller 187 socket:[190652724]
1   /ingress-controller 188 socket:[190604364]
1   /ingress-controller 190 socket:[190601926]
1   /ingress-controller 191 socket:[190654947]
1   /ingress-controller 192 socket:[190654949]
1   /ingress-controller 193 socket:[190651702]
1   /ingress-controller 194 socket:[190635923]
1   /ingress-controller 195 socket:[190658643]
1   /ingress-controller 197 socket:[190613913]
1   /ingress-controller 198 socket:[190651342]
1   /ingress-controller 199 socket:[190603581]
1   /ingress-controller 200 socket:[190658144]
1   /ingress-controller 201 socket:[190658145]
1   /ingress-controller 202 socket:[190658146]
1   /ingress-controller 203 socket:[190609753]
1   /ingress-controller 205 socket:[190632484]
1   /ingress-controller 206 socket:[190611644]
1   /ingress-controller 207 socket:[190611677]
1   /ingress-controller 208 socket:[190601890]
1   /ingress-controller 209 socket:[190614050]
1   /ingress-controller 210 socket:[190651923]
1   /ingress-controller 212 socket:[190633178]
1   /ingress-controller 216 socket:[190649911]
1   /ingress-controller 218 socket:[190635641]
1   /ingress-controller 222 socket:[190635615]
1   /ingress-controller 227 socket:[190601938]
1   /ingress-controller 228 socket:[190609604]
1   /ingress-controller 230 socket:[190597972]
1   /ingress-controller 231 socket:[190598002]
1   /ingress-controller 233 socket:[190603113]
1   /ingress-controller 234 socket:[190598022]
1   /ingress-controller 235 socket:[190598033]
1   /ingress-controller 236 socket:[190605581]
1   /ingress-controller 238 socket:[190604684]
1   /ingress-controller 239 socket:[190658149]
1   /ingress-controller 240 socket:[190605674]
1   /ingress-controller 242 socket:[190602128]
1   /ingress-controller 244 socket:[190614945]
1   /ingress-controller 245 socket:[190608011]
1   /ingress-controller 247 socket:[190656932]
1   /ingress-controller 249 socket:[190610889]
1   /ingress-controller 250 socket:[190653319]
1   /ingress-controller 251 socket:[190607158]
1   /ingress-controller 253 socket:[190608267]
1   /ingress-controller 254 socket:[190616871]
1   /ingress-controller 257 socket:[190608335]
1   /ingress-controller 259 socket:[190613570]
1   /ingress-controller 260 socket:[190611919]
1   /ingress-controller 261 socket:[190612192]
1   /ingress-controller 262 socket:[190612203]
1   /ingress-controller 263 socket:[190616920]
1   /ingress-controller 264 socket:[190618024]
1   /ingress-controller 266 socket:[190613798]
1   /ingress-controller 267 socket:[190610336]
1   /ingress-controller 268 socket:[190613252]
1   /ingress-controller 269 socket:[190618036]
1   /ingress-controller 270 socket:[190618045]
1   /ingress-controller 271 socket:[190618065]
1   /ingress-controller 272 socket:[190610379]
1   /ingress-controller 274 socket:[190610385]
1   /ingress-controller 275 socket:[190634162]
1   /ingress-controller 276 socket:[190633000]
1   /ingress-controller 277 socket:[190618130]
1   /ingress-controller 278 socket:[190631113]
1   /ingress-controller 279 socket:[190614061]
1   /ingress-controller 280 socket:[190614070]
31  /bin/busybox    0   /dev/pts/0
31  /bin/busybox    1   /dev/pts/0
31  /bin/busybox    2   /dev/pts/0
31  /bin/busybox    10  /dev/tty
/ #
timthelion commented 7 months ago 
/ # find . | grep -v /sys | grep -v /proc
.
./run
./run/secrets
./run/secrets/kubernetes.io
./run/secrets/kubernetes.io/serviceaccount
./run/secrets/kubernetes.io/serviceaccount/..data
./run/secrets/kubernetes.io/serviceaccount/..2023_12_31_11_48_51.2405516304
./run/secrets/kubernetes.io/serviceaccount/..2023_12_31_11_48_51.2405516304/ca.crt
./run/secrets/kubernetes.io/serviceaccount/..2023_12_31_11_48_51.2405516304/token
./run/secrets/kubernetes.io/serviceaccount/..2023_12_31_11_48_51.2405516304/namespace
./run/secrets/kubernetes.io/serviceaccount/namespace
./run/secrets/kubernetes.io/serviceaccount/ca.crt
./run/secrets/kubernetes.io/serviceaccount/token
./srv
./opt
./usr
./usr/share
./usr/share/apk
./usr/share/apk/keys
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-61666e3f.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616ac3bc.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616ae350.rsa.pub
./usr/share/apk/keys/armv7
./usr/share/apk/keys/armv7/alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub
./usr/share/apk/keys/armv7/alpine-devel@lists.alpinelinux.org-616adfeb.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616abc23.rsa.pub
./usr/share/apk/keys/aarch64
./usr/share/apk/keys/aarch64/alpine-devel@lists.alpinelinux.org-616ae350.rsa.pub
./usr/share/apk/keys/aarch64/alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub
./usr/share/apk/keys/ppc64le
./usr/share/apk/keys/ppc64le/alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub
./usr/share/apk/keys/ppc64le/alpine-devel@lists.alpinelinux.org-616abc23.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616a9724.rsa.pub
./usr/share/apk/keys/riscv64
./usr/share/apk/keys/riscv64/alpine-devel@lists.alpinelinux.org-60ac2099.rsa.pub
./usr/share/apk/keys/riscv64/alpine-devel@lists.alpinelinux.org-616db30d.rsa.pub
./usr/share/apk/keys/s390x
./usr/share/apk/keys/s390x/alpine-devel@lists.alpinelinux.org-616ac3bc.rsa.pub
./usr/share/apk/keys/s390x/alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-5e69ca50.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub
./usr/share/apk/keys/x86_64
./usr/share/apk/keys/x86_64/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
./usr/share/apk/keys/x86_64/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
./usr/share/apk/keys/x86_64/alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
./usr/share/apk/keys/armhf
./usr/share/apk/keys/armhf/alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub
./usr/share/apk/keys/armhf/alpine-devel@lists.alpinelinux.org-616a9724.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-60ac2099.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616db30d.rsa.pub
./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-616adfeb.rsa.pub
./usr/share/apk/keys/mips64
./usr/share/apk/keys/mips64/alpine-devel@lists.alpinelinux.org-5e69ca50.rsa.pub
./usr/share/apk/keys/x86
./usr/share/apk/keys/x86/alpine-devel@lists.alpinelinux.org-61666e3f.rsa.pub
./usr/share/apk/keys/x86/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
./usr/share/apk/keys/x86/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
./usr/share/udhcpc
./usr/share/udhcpc/default.script
./usr/share/misc
./usr/share/man
./usr/share/ca-certificates
./usr/share/ca-certificates/mozilla
./usr/share/ca-certificates/mozilla/NAVER_Global_Root_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/NetLock_Arany_=Class_Gold=_Főtanúsítvány.crt
./usr/share/ca-certificates/mozilla/GlobalSign_Root_CA_-_R6.crt
./usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA.crt
./usr/share/ca-certificates/mozilla/DigiCert_High_Assurance_EV_Root_CA.crt
./usr/share/ca-certificates/mozilla/Starfield_Class_2_CA.crt
./usr/share/ca-certificates/mozilla/Certigna.crt
./usr/share/ca-certificates/mozilla/Certum_EC-384_CA.crt
./usr/share/ca-certificates/mozilla/emSign_ECC_Root_CA_-_C3.crt
./usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
./usr/share/ca-certificates/mozilla/D-TRUST_EV_Root_CA_1_2020.crt
./usr/share/ca-certificates/mozilla/SZAFIR_ROOT_CA2.crt
./usr/share/ca-certificates/mozilla/GTS_Root_R2.crt
./usr/share/ca-certificates/mozilla/SwissSign_Gold_CA_-_G2.crt
./usr/share/ca-certificates/mozilla/GlobalSign_ECC_Root_CA_-_R4.crt
./usr/share/ca-certificates/mozilla/GlobalSign_Root_CA.crt
./usr/share/ca-certificates/mozilla/GTS_Root_R3.crt
./usr/share/ca-certificates/mozilla/e-Szigno_Root_CA_2017.crt
./usr/share/ca-certificates/mozilla/SSL.com_Root_Certification_Authority_RSA.crt
./usr/share/ca-certificates/mozilla/vTrus_ECC_Root_CA.crt
./usr/share/ca-certificates/mozilla/Certainly_Root_E1.crt
./usr/share/ca-certificates/mozilla/Telia_Root_CA_v2.crt
./usr/share/ca-certificates/mozilla/TeliaSonera_Root_CA_v1.crt
./usr/share/ca-certificates/mozilla/Izenpe.com.crt
./usr/share/ca-certificates/mozilla/Baltimore_CyberTrust_Root.crt
./usr/share/ca-certificates/mozilla/TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.crt
./usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_1_G3.crt
./usr/share/ca-certificates/mozilla/AffirmTrust_Premium.crt
./usr/share/ca-certificates/mozilla/ACCVRAIZ1.crt
./usr/share/ca-certificates/mozilla/Certainly_Root_R1.crt
./usr/share/ca-certificates/mozilla/certSIGN_Root_CA_G2.crt
./usr/share/ca-certificates/mozilla/Amazon_Root_CA_1.crt
./usr/share/ca-certificates/mozilla/Certigna_Root_CA.crt
./usr/share/ca-certificates/mozilla/Go_Daddy_Root_Certificate_Authority_-_G2.crt
./usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_2.crt
./usr/share/ca-certificates/mozilla/DigiCert_Assured_ID_Root_G3.crt
./usr/share/ca-certificates/mozilla/DigiCert_Global_Root_CA.crt
./usr/share/ca-certificates/mozilla/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.crt
./usr/share/ca-certificates/mozilla/OISTE_WISeKey_Global_Root_GC_CA.crt
./usr/share/ca-certificates/mozilla/Buypass_Class_2_Root_CA.crt
./usr/share/ca-certificates/mozilla/Trustwave_Global_ECC_P384_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/D-TRUST_Root_Class_3_CA_2_EV_2009.crt
./usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_3.crt
./usr/share/ca-certificates/mozilla/COMODO_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority_-_EC1.crt
./usr/share/ca-certificates/mozilla/TrustCor_ECA-1.crt
./usr/share/ca-certificates/mozilla/GTS_Root_R1.crt
./usr/share/ca-certificates/mozilla/Comodo_AAA_Services_root.crt
./usr/share/ca-certificates/mozilla/vTrus_Root_CA.crt
./usr/share/ca-certificates/mozilla/TunTrust_Root_CA.crt
./usr/share/ca-certificates/mozilla/Trustwave_Global_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/GlobalSign_ECC_Root_CA_-_R5.crt
./usr/share/ca-certificates/mozilla/USERTrust_ECC_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/Hellenic_Academic_and_Research_Institutions_RootCA_2015.crt
./usr/share/ca-certificates/mozilla/AffirmTrust_Networking.crt
./usr/share/ca-certificates/mozilla/BJCA_Global_Root_CA1.crt
./usr/share/ca-certificates/mozilla/Microsec_e-Szigno_Root_CA_2009.crt
./usr/share/ca-certificates/mozilla/emSign_Root_CA_-_C1.crt
./usr/share/ca-certificates/mozilla/TrustCor_RootCert_CA-1.crt
./usr/share/ca-certificates/mozilla/HiPKI_Root_CA_-_G1.crt
./usr/share/ca-certificates/mozilla/Certum_Trusted_Network_CA_2.crt
./usr/share/ca-certificates/mozilla/SecureSign_RootCA11.crt
./usr/share/ca-certificates/mozilla/Buypass_Class_3_Root_CA.crt
./usr/share/ca-certificates/mozilla/Entrust.net_Premium_2048_Secure_Server_CA.crt
./usr/share/ca-certificates/mozilla/AC_RAIZ_FNMT-RCM_SERVIDORES_SEGUROS.crt
./usr/share/ca-certificates/mozilla/certSIGN_ROOT_CA.crt
./usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority_-_G2.crt
./usr/share/ca-certificates/mozilla/emSign_Root_CA_-_G1.crt
./usr/share/ca-certificates/mozilla/SecureTrust_CA.crt
./usr/share/ca-certificates/mozilla/E-Tugra_Global_Root_CA_ECC_v3.crt
./usr/share/ca-certificates/mozilla/emSign_ECC_Root_CA_-_G3.crt
./usr/share/ca-certificates/mozilla/D-TRUST_BR_Root_CA_1_2020.crt
./usr/share/ca-certificates/mozilla/Security_Communication_ECC_RootCA1.crt
./usr/share/ca-certificates/mozilla/Trustwave_Global_ECC_P256_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/UCA_Extended_Validation_Root.crt
./usr/share/ca-certificates/mozilla/Hongkong_Post_Root_CA_1.crt
./usr/share/ca-certificates/mozilla/DigiCert_Global_Root_G2.crt
./usr/share/ca-certificates/mozilla/COMODO_ECC_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/GlobalSign_Root_CA_-_R3.crt
./usr/share/ca-certificates/mozilla/IdenTrust_Public_Sector_Root_CA_1.crt
./usr/share/ca-certificates/mozilla/Actalis_Authentication_Root_CA.crt
./usr/share/ca-certificates/mozilla/USERTrust_RSA_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/Security_Communication_RootCA2.crt
./usr/share/ca-certificates/mozilla/SwissSign_Silver_CA_-_G2.crt
./usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority_-_G4.crt
./usr/share/ca-certificates/mozilla/Starfield_Services_Root_Certificate_Authority_-_G2.crt
./usr/share/ca-certificates/mozilla/GLOBALTRUST_2020.crt
./usr/share/ca-certificates/mozilla/DigiCert_Assured_ID_Root_G2.crt
./usr/share/ca-certificates/mozilla/Certum_Trusted_Root_CA.crt
./usr/share/ca-certificates/mozilla/ePKI_Root_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/Security_Communication_RootCA3.crt
./usr/share/ca-certificates/mozilla/D-TRUST_Root_Class_3_CA_2_2009.crt
./usr/share/ca-certificates/mozilla/Amazon_Root_CA_4.crt
./usr/share/ca-certificates/mozilla/Security_Communication_Root_CA.crt
./usr/share/ca-certificates/mozilla/ANF_Secure_Server_Root_CA.crt
./usr/share/ca-certificates/mozilla/SSL.com_EV_Root_Certification_Authority_RSA_R2.crt
./usr/share/ca-certificates/mozilla/Secure_Global_CA.crt
./usr/share/ca-certificates/mozilla/Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.crt
./usr/share/ca-certificates/mozilla/OISTE_WISeKey_Global_Root_GB_CA.crt
./usr/share/ca-certificates/mozilla/DigiCert_Assured_ID_Root_CA.crt
./usr/share/ca-certificates/mozilla/HARICA_TLS_RSA_Root_CA_2021.crt
./usr/share/ca-certificates/mozilla/XRamp_Global_CA_Root.crt
./usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/Microsoft_ECC_Root_Certificate_Authority_2017.crt
./usr/share/ca-certificates/mozilla/GlobalSign_Root_R46.crt
./usr/share/ca-certificates/mozilla/SSL.com_Root_Certification_Authority_ECC.crt
./usr/share/ca-certificates/mozilla/CFCA_EV_ROOT.crt
./usr/share/ca-certificates/mozilla/GDCA_TrustAUTH_R5_ROOT.crt
./usr/share/ca-certificates/mozilla/T-TeleSec_GlobalRoot_Class_3.crt
./usr/share/ca-certificates/mozilla/Atos_TrustedRoot_2011.crt
./usr/share/ca-certificates/mozilla/AffirmTrust_Commercial.crt
./usr/share/ca-certificates/mozilla/Microsoft_RSA_Root_Certificate_Authority_2017.crt
./usr/share/ca-certificates/mozilla/Hongkong_Post_Root_CA_3.crt
./usr/share/ca-certificates/mozilla/E-Tugra_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/IdenTrust_Commercial_Root_CA_1.crt
./usr/share/ca-certificates/mozilla/TWCA_Root_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/UCA_Global_G2_Root.crt
./usr/share/ca-certificates/mozilla/DigiCert_TLS_RSA4096_Root_G5.crt
./usr/share/ca-certificates/mozilla/COMODO_RSA_Certification_Authority.crt
./usr/share/ca-certificates/mozilla/E-Tugra_Global_Root_CA_RSA_v3.crt
./usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_2_G3.crt
./usr/share/ca-certificates/mozilla/HARICA_TLS_ECC_Root_CA_2021.crt
./usr/share/ca-certificates/mozilla/AC_RAIZ_FNMT-RCM.crt
./usr/share/ca-certificates/mozilla/CA_Disig_Root_R2.crt
./usr/share/ca-certificates/mozilla/SSL.com_EV_Root_Certification_Authority_ECC.crt
./usr/share/ca-certificates/mozilla/Starfield_Root_Certificate_Authority_-_G2.crt
./usr/share/ca-certificates/mozilla/DigiCert_Global_Root_G3.crt
./usr/share/ca-certificates/mozilla/Go_Daddy_Class_2_CA.crt
./usr/share/ca-certificates/mozilla/AffirmTrust_Premium_ECC.crt
./usr/share/ca-certificates/mozilla/QuoVadis_Root_CA_3_G3.crt
./usr/share/ca-certificates/mozilla/Amazon_Root_CA_2.crt
./usr/share/ca-certificates/mozilla/GlobalSign_Root_E46.crt
./usr/share/ca-certificates/mozilla/BJCA_Global_Root_CA2.crt
./usr/share/ca-certificates/mozilla/ISRG_Root_X2.crt
./usr/share/ca-certificates/mozilla/T-TeleSec_GlobalRoot_Class_2.crt
./usr/share/ca-certificates/mozilla/DigiCert_Trusted_Root_G4.crt
./usr/share/ca-certificates/mozilla/TrustCor_RootCert_CA-2.crt
./usr/share/ca-certificates/mozilla/Amazon_Root_CA_3.crt
./usr/share/ca-certificates/mozilla/GTS_Root_R4.crt
./usr/share/ca-certificates/mozilla/TWCA_Global_Root_CA.crt
./usr/share/ca-certificates/mozilla/DigiCert_TLS_ECC_P384_Root_G5.crt
./usr/bin
./usr/bin/uuencode
./usr/bin/time
./usr/bin/lzma
./usr/bin/nmeter
./usr/bin/[
./usr/bin/whoami
./usr/bin/free
./usr/bin/od
./usr/bin/traceroute
./usr/bin/setsid
./usr/bin/md5sum
./usr/bin/hostid
./usr/bin/head
./usr/bin/nl
./usr/bin/fuser
./usr/bin/logger
./usr/bin/realpath
./usr/bin/sum
./usr/bin/lsof
./usr/bin/last
./usr/bin/dirname
./usr/bin/less
./usr/bin/nsenter
./usr/bin/dc
./usr/bin/pkill
./usr/bin/lsusb
./usr/bin/cmp
./usr/bin/vlock
./usr/bin/pscan
./usr/bin/unlzop
./usr/bin/seq
./usr/bin/tac
./usr/bin/beep
./usr/bin/dos2unix
./usr/bin/readlink
./usr/bin/tr
./usr/bin/mkpasswd
./usr/bin/passwd
./usr/bin/paste
./usr/bin/flock
./usr/bin/pstree
./usr/bin/volname
./usr/bin/killall
./usr/bin/uudecode
./usr/bin/cksum
./usr/bin/ssl_client
./usr/bin/bzip2
./usr/bin/xxd
./usr/bin/cpio
./usr/bin/unexpand
./usr/bin/shred
./usr/bin/scanelf
./usr/bin/ipcs
./usr/bin/fallocate
./usr/bin/pmap
./usr/bin/basename
./usr/bin/vi
./usr/bin/xargs
./usr/bin/unlzma
./usr/bin/microcom
./usr/bin/wget
./usr/bin/id
./usr/bin/unix2dos
./usr/bin/unxz
./usr/bin/expand
./usr/bin/sha3sum
./usr/bin/eject
./usr/bin/tty
./usr/bin/truncate
./usr/bin/udhcpc6
./usr/bin/resize
./usr/bin/openvt
./usr/bin/lzopcat
./usr/bin/pwdx
./usr/bin/clear
./usr/bin/lzcat
./usr/bin/uptime
./usr/bin/cal
./usr/bin/mkfifo
./usr/bin/which
./usr/bin/whois
./usr/bin/ttysize
./usr/bin/wc
./usr/bin/chvt
./usr/bin/hd
./usr/bin/awk
./usr/bin/tee
./usr/bin/bc
./usr/bin/diff
./usr/bin/du
./usr/bin/reset
./usr/bin/xzcat
./usr/bin/unzip
./usr/bin/top
./usr/bin/hexdump
./usr/bin/showkey
./usr/bin/unlink
./usr/bin/getconf
./usr/bin/crontab
./usr/bin/groups
./usr/bin/getent
./usr/bin/cryptpw
./usr/bin/sha512sum
./usr/bin/shuf
./usr/bin/mesg
./usr/bin/nc
./usr/bin/factor
./usr/bin/expr
./usr/bin/nproc
./usr/bin/deallocvt
./usr/bin/sort
./usr/bin/nohup
./usr/bin/printf
./usr/bin/timeout
./usr/bin/ipcrm
./usr/bin/comm
./usr/bin/yes
./usr/bin/[[
./usr/bin/tail
./usr/bin/split
./usr/bin/traceroute6
./usr/bin/uniq
./usr/bin/iconv
./usr/bin/fold
./usr/bin/strings
./usr/bin/pgrep
./usr/bin/find
./usr/bin/sha256sum
./usr/bin/ldd
./usr/bin/who
./usr/bin/sha1sum
./usr/bin/install
./usr/bin/renice
./usr/bin/bzcat
./usr/bin/nslookup
./usr/bin/env
./usr/bin/bunzip2
./usr/bin/blkdiscard
./usr/bin/setkeycodes
./usr/bin/unshare
./usr/bin/cut
./usr/bin/test
./usr/bin/c_rehash
./usr/bin/curl
./usr/lib
./usr/lib/modules-load.d
./usr/lib/libcrypto.so.1.1
./usr/lib/libssl.so.1.1
./usr/lib/engines-1.1
./usr/lib/engines-1.1/afalg.so
./usr/lib/engines-1.1/capi.so
./usr/lib/engines-1.1/padlock.so
./usr/lib/libcurl.so.4.8.0
./usr/lib/libnghttp2.so.14
./usr/lib/libbrotlidec.so.1
./usr/lib/libbrotlicommon.so.1.0.9
./usr/lib/libcurl.so.4
./usr/lib/libnghttp2.so.14.21.2
./usr/lib/libbrotlienc.so.1.0.9
./usr/lib/libbrotlicommon.so.1
./usr/lib/libbrotlidec.so.1.0.9
./usr/lib/libbrotlienc.so.1
./usr/local
./usr/local/share
./usr/local/share/ca-certificates
./usr/local/bin
./usr/local/lib
./usr/sbin
./usr/sbin/fbset
./usr/sbin/addgroup
./usr/sbin/chpasswd
./usr/sbin/rdev
./usr/sbin/arping
./usr/sbin/nanddump
./usr/sbin/loadfont
./usr/sbin/killall5
./usr/sbin/nandwrite
./usr/sbin/adduser
./usr/sbin/rdate
./usr/sbin/ntpd
./usr/sbin/setlogcons
./usr/sbin/chroot
./usr/sbin/setfont
./usr/sbin/delgroup
./usr/sbin/brctl
./usr/sbin/rfkill
./usr/sbin/nbd-client
./usr/sbin/readahead
./usr/sbin/add-shell
./usr/sbin/ether-wake
./usr/sbin/crond
./usr/sbin/deluser
./usr/sbin/partprobe
./usr/sbin/remove-shell
./usr/sbin/sendmail
./usr/sbin/update-ca-certificates
./bin
./bin/chgrp
./bin/lzop
./bin/iostat
./bin/setserial
./bin/arch
./bin/busybox
./bin/chmod
./bin/ionice
./bin/mkdir
./bin/touch
./bin/nice
./bin/ls
./bin/makemime
./bin/sleep
./bin/printenv
./bin/netstat
./bin/kbd_mode
./bin/login
./bin/getopt
./bin/uname
./bin/hostname
./bin/run-parts
./bin/bbconfig
./bin/base64
./bin/sync
./bin/true
./bin/usleep
./bin/kill
./bin/fatattr
./bin/ps
./bin/dmesg
./bin/pwd
./bin/reformime
./bin/tar
./bin/cat
./bin/rmdir
./bin/stat
./bin/rev
./bin/su
./bin/pidof
./bin/mountpoint
./bin/ln
./bin/egrep
./bin/mknod
./bin/more
./bin/date
./bin/stty
./bin/ed
./bin/ping
./bin/umount
./bin/cp
./bin/fdflush
./bin/link
./bin/ipcalc
./bin/echo
./bin/chattr
./bin/fgrep
./bin/sh
./bin/ping6
./bin/mpstat
./bin/dumpkmap
./bin/grep
./bin/dd
./bin/mv
./bin/gzip
./bin/rm
./bin/watch
./bin/lsattr
./bin/false
./bin/fsync
./bin/df
./bin/sed
./bin/setpriv
./bin/dnsdomainname
./bin/chown
./bin/linux64
./bin/gunzip
./bin/pipe_progress
./bin/ash
./bin/mount
./bin/linux32
./bin/zcat
./bin/mktemp
./tmp
./etc
./etc/services
./etc/profile.d
./etc/profile.d/README
./etc/profile.d/locale.sh
./etc/profile.d/color_prompt.sh.disabled
./etc/opt
./etc/crontabs
./etc/crontabs/root
./etc/os-release
./etc/inittab
./etc/init.d
./etc/apk
./etc/apk/arch
./etc/apk/keys
./etc/apk/keys/alpine-devel@lists.alpinelinux.org-61666e3f.rsa.pub
./etc/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub
./etc/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub
./etc/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub
./etc/apk/keys/alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
./etc/apk/protected_paths.d
./etc/apk/protected_paths.d/ca-certificates.list
./etc/apk/repositories
./etc/apk/world
./etc/hostname
./etc/passwd
./etc/profile
./etc/modules-load.d
./etc/udhcpd.conf
./etc/conf.d
./etc/shadow
./etc/modprobe.d
./etc/modprobe.d/aliases.conf
./etc/modprobe.d/i386.conf
./etc/modprobe.d/blacklist.conf
./etc/modprobe.d/kms.conf
./etc/hosts
./etc/network
./etc/network/if-pre-up.d
./etc/network/if-pre-down.d
./etc/network/if-post-down.d
./etc/network/if-post-up.d
./etc/network/if-down.d
./etc/network/if-up.d
./etc/network/if-up.d/dad
./etc/issue
./etc/alpine-release
./etc/motd
./etc/mtab
./etc/secfixes.d
./etc/secfixes.d/alpine
./etc/protocols
./etc/modules
./etc/securetty
./etc/group
./etc/fstab
./etc/ssl
./etc/ssl/openssl.cnf
./etc/ssl/cert.pem
./etc/ssl/ct_log_list.cnf
./etc/ssl/certs
./etc/ssl/certs/ca-certificates.crt
./etc/ssl/certs/ca-cert-BJCA_Global_Root_CA2.pem
./etc/ssl/certs/ca-cert-DigiCert_Assured_ID_Root_G2.pem
./etc/ssl/certs/ca-cert-Atos_TrustedRoot_2011.pem
./etc/ssl/certs/ca-cert-e-Szigno_Root_CA_2017.pem
./etc/ssl/certs/7719f463.0
./etc/ssl/certs/6fa5da56.0
./etc/ssl/certs/1e08bfd1.0
./etc/ssl/certs/b7a5b843.0
./etc/ssl/certs/ca-cert-COMODO_Certification_Authority.pem
./etc/ssl/certs/e35234b1.0
./etc/ssl/certs/7a3adc42.0
./etc/ssl/certs/ca-cert-AC_RAIZ_FNMT-RCM_SERVIDORES_SEGUROS.pem
./etc/ssl/certs/626dceaf.0
./etc/ssl/certs/9482e63a.0
./etc/ssl/certs/ca-cert-HiPKI_Root_CA_-_G1.pem
./etc/ssl/certs/ca-cert-GlobalSign_Root_CA_-_R3.pem
./etc/ssl/certs/ca-cert-D-TRUST_BR_Root_CA_1_2020.pem
./etc/ssl/certs/5cd81ad7.0
./etc/ssl/certs/ca-cert-QuoVadis_Root_CA_2.pem
./etc/ssl/certs/ca-cert-TWCA_Root_Certification_Authority.pem
./etc/ssl/certs/06dc52d5.0
./etc/ssl/certs/ca-cert-ePKI_Root_Certification_Authority.pem
./etc/ssl/certs/ca-cert-DigiCert_Global_Root_CA.pem
./etc/ssl/certs/ca-cert-Certainly_Root_E1.pem
./etc/ssl/certs/106f3e4d.0
./etc/ssl/certs/1001acf7.0
./etc/ssl/certs/18856ac4.0
./etc/ssl/certs/ca-cert-GLOBALTRUST_2020.pem
./etc/ssl/certs/68dd7389.0
./etc/ssl/certs/3e44d2f7.0
./etc/ssl/certs/ca-cert-vTrus_ECC_Root_CA.pem
./etc/ssl/certs/ef954a4e.0
./etc/ssl/certs/706f604c.0
./etc/ssl/certs/ca-cert-emSign_ECC_Root_CA_-_C3.pem
./etc/ssl/certs/244b5494.0
./etc/ssl/certs/ca-cert-Amazon_Root_CA_2.pem
./etc/ssl/certs/de6d66f3.0
./etc/ssl/certs/ca-cert-DigiCert_Assured_ID_Root_G3.pem
./etc/ssl/certs/5f618aec.0
./etc/ssl/certs/f387163d.0
./etc/ssl/certs/930ac5d2.0
./etc/ssl/certs/ce5e74ef.0
./etc/ssl/certs/ca-cert-TWCA_Global_Root_CA.pem
./etc/ssl/certs/b433981b.0
./etc/ssl/certs/8f103249.0
./etc/ssl/certs/ca-cert-ACCVRAIZ1.pem
./etc/ssl/certs/ca-cert-Entrust_Root_Certification_Authority.pem
./etc/ssl/certs/f0c70a8d.0
./etc/ssl/certs/ca-cert-GTS_Root_R2.pem
./etc/ssl/certs/57bcb2da.0
./etc/ssl/certs/ca-cert-USERTrust_ECC_Certification_Authority.pem
./etc/ssl/certs/0b9bc432.0
./etc/ssl/certs/ca-cert-DigiCert_Global_Root_G3.pem
./etc/ssl/certs/dd8e9d41.0
./etc/ssl/certs/ca-cert-HARICA_TLS_ECC_Root_CA_2021.pem
./etc/ssl/certs/002c0b4f.0
./etc/ssl/certs/ca-cert-Hongkong_Post_Root_CA_1.pem
./etc/ssl/certs/ca-cert-SSL.com_Root_Certification_Authority_RSA.pem
./etc/ssl/certs/ca-cert-UCA_Global_G2_Root.pem
./etc/ssl/certs/ca-cert-HARICA_TLS_RSA_Root_CA_2021.pem
./etc/ssl/certs/ca-cert-certSIGN_Root_CA_G2.pem
./etc/ssl/certs/4042bcee.0
./etc/ssl/certs/ca-cert-Entrust.net_Premium_2048_Secure_Server_CA.pem
./etc/ssl/certs/3bde41ac.0
./etc/ssl/certs/ca-cert-Telia_Root_CA_v2.pem
./etc/ssl/certs/5443e9e3.0
./etc/ssl/certs/eed8c118.0
./etc/ssl/certs/ca-cert-Trustwave_Global_ECC_P384_Certification_Authority.pem
./etc/ssl/certs/ca-cert-DigiCert_Global_Root_G2.pem
./etc/ssl/certs/fa5da96b.0
./etc/ssl/certs/ecccd8db.0
./etc/ssl/certs/ca-cert-Security_Communication_ECC_RootCA1.pem
./etc/ssl/certs/d4dae3dd.0
./etc/ssl/certs/0a775a30.0
./etc/ssl/certs/ca-cert-GlobalSign_Root_CA.pem
./etc/ssl/certs/3e359ba6.0
./etc/ssl/certs/ca-cert-Trustwave_Global_Certification_Authority.pem
./etc/ssl/certs/d7e8dc79.0
./etc/ssl/certs/ca-cert-GlobalSign_ECC_Root_CA_-_R5.pem
./etc/ssl/certs/062cdee6.0
./etc/ssl/certs/ca-cert-E-Tugra_Certification_Authority.pem
./etc/ssl/certs/a3418fda.0
./etc/ssl/certs/ca-cert-TrustCor_RootCert_CA-2.pem
./etc/ssl/certs/90c5a3c8.0
./etc/ssl/certs/ca-cert-Certigna.pem
./etc/ssl/certs/14bc7599.0
./etc/ssl/certs/ca-cert-emSign_Root_CA_-_G1.pem
./etc/ssl/certs/ca-cert-USERTrust_RSA_Certification_Authority.pem
./etc/ssl/certs/0bf05006.0
./etc/ssl/certs/f39fc864.0
./etc/ssl/certs/ca-cert-XRamp_Global_CA_Root.pem
./etc/ssl/certs/ca-cert-Starfield_Root_Certificate_Authority_-_G2.pem
./etc/ssl/certs/9d04f354.0
./etc/ssl/certs/ca-cert-DigiCert_Assured_ID_Root_CA.pem
./etc/ssl/certs/ca-cert-AffirmTrust_Networking.pem
./etc/ssl/certs/40547a79.0
./etc/ssl/certs/0179095f.0
./etc/ssl/certs/9c8dfbd4.0
./etc/ssl/certs/ca-cert-TrustCor_RootCert_CA-1.pem
./etc/ssl/certs/0f5dc4f3.0
./etc/ssl/certs/ca-cert-TunTrust_Root_CA.pem
./etc/ssl/certs/ca-cert-Go_Daddy_Root_Certificate_Authority_-_G2.pem
./etc/ssl/certs/3513523f.0
./etc/ssl/certs/75d1b2ed.0
./etc/ssl/certs/ca-cert-emSign_ECC_Root_CA_-_G3.pem
./etc/ssl/certs/ca-cert-GTS_Root_R1.pem
./etc/ssl/certs/9ef4a08a.0
./etc/ssl/certs/4b718d9b.0
./etc/ssl/certs/dc4d6a89.0
./etc/ssl/certs/fd64f3fc.0
./etc/ssl/certs/ca-cert-TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.pem
./etc/ssl/certs/09789157.0
./etc/ssl/certs/ca-cert-Security_Communication_RootCA3.pem
./etc/ssl/certs/f30dd6ad.0
./etc/ssl/certs/ca-cert-Amazon_Root_CA_3.pem
./etc/ssl/certs/e73d606e.0
./etc/ssl/certs/d6325660.0
./etc/ssl/certs/66445960.0
./etc/ssl/certs/ca-cert-Secure_Global_CA.pem
./etc/ssl/certs/ca-cert-NAVER_Global_Root_Certification_Authority.pem
./etc/ssl/certs/ca-cert-Certainly_Root_R1.pem
./etc/ssl/certs/9f727ac7.0
./etc/ssl/certs/8cb5ee0f.0
./etc/ssl/certs/ca-cert-TeliaSonera_Root_CA_v1.pem
./etc/ssl/certs/3fb36b73.0
./etc/ssl/certs/ca-cert-T-TeleSec_GlobalRoot_Class_2.pem
./etc/ssl/certs/40193066.0
./etc/ssl/certs/ca-cert-SSL.com_EV_Root_Certification_Authority_RSA_R2.pem
./etc/ssl/certs/ca-cert-AffirmTrust_Premium.pem
./etc/ssl/certs/7a780d93.0
./etc/ssl/certs/ca-cert-GTS_Root_R3.pem
./etc/ssl/certs/ca-cert-Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
./etc/ssl/certs/ca-cert-QuoVadis_Root_CA_2_G3.pem
./etc/ssl/certs/e868b802.0
./etc/ssl/certs/3e45d192.0
./etc/ssl/certs/ca-cert-Security_Communication_Root_CA.pem
./etc/ssl/certs/f081611a.0
./etc/ssl/certs/8508e720.0
./etc/ssl/certs/ca-cert-Trustwave_Global_ECC_P256_Certification_Authority.pem
./etc/ssl/certs/e18bfb83.0
./etc/ssl/certs/aee5f10d.0
./etc/ssl/certs/ca-cert-Starfield_Services_Root_Certificate_Authority_-_G2.pem
./etc/ssl/certs/ca-cert-DigiCert_High_Assurance_EV_Root_CA.pem
./etc/ssl/certs/d52c538d.0
./etc/ssl/certs/76faf6c0.0
./etc/ssl/certs/b81b93f0.0
./etc/ssl/certs/f3377b1b.0
./etc/ssl/certs/ca-cert-Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem
./etc/ssl/certs/cd58d51e.0
./etc/ssl/certs/ca-cert-TrustCor_ECA-1.pem
./etc/ssl/certs/bf53fb88.0
./etc/ssl/certs/ca-cert-DigiCert_TLS_ECC_P384_Root_G5.pem
./etc/ssl/certs/8d89cda1.0
./etc/ssl/certs/6b99d060.0
./etc/ssl/certs/5a7722fb.0
./etc/ssl/certs/ca-cert-SwissSign_Silver_CA_-_G2.pem
./etc/ssl/certs/7aaf71c0.0
./etc/ssl/certs/ca-cert-Buypass_Class_2_Root_CA.pem
./etc/ssl/certs/ca-cert-Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem
./etc/ssl/certs/607986c7.0
./etc/ssl/certs/ca-cert-Comodo_AAA_Services_root.pem
./etc/ssl/certs/5273a94c.0
./etc/ssl/certs/cbf06781.0
./etc/ssl/certs/ca-cert-certSIGN_ROOT_CA.pem
./etc/ssl/certs/ca-cert-IdenTrust_Commercial_Root_CA_1.pem
./etc/ssl/certs/b0e59380.0
./etc/ssl/certs/ca-cert-GlobalSign_Root_CA_-_R6.pem
./etc/ssl/certs/ca-cert-Certum_Trusted_Network_CA_2.pem
./etc/ssl/certs/e8de2f56.0
./etc/ssl/certs/ca-cert-SSL.com_EV_Root_Certification_Authority_ECC.pem
./etc/ssl/certs/ca-cert-GlobalSign_Root_E46.pem
./etc/ssl/certs/a94d09e5.0
./etc/ssl/certs/ca-cert-D-TRUST_EV_Root_CA_1_2020.pem
./etc/ssl/certs/ca-cert-Amazon_Root_CA_1.pem
./etc/ssl/certs/5e98733a.0
./etc/ssl/certs/ca-cert-CA_Disig_Root_R2.pem
./etc/ssl/certs/32888f65.0
./etc/ssl/certs/1d3472b9.0
./etc/ssl/certs/ca-cert-T-TeleSec_GlobalRoot_Class_3.pem
./etc/ssl/certs/ca-cert-QuoVadis_Root_CA_3_G3.pem
./etc/ssl/certs/988a38cb.0
./etc/ssl/certs/ca-cert-SwissSign_Gold_CA_-_G2.pem
./etc/ssl/certs/ca-cert-Microsoft_ECC_Root_Certificate_Authority_2017.pem
./etc/ssl/certs/ca-cert-E-Tugra_Global_Root_CA_ECC_v3.pem
./etc/ssl/certs/cd8c0d63.0
./etc/ssl/certs/f51bb24c.0
./etc/ssl/certs/4f316efb.0
./etc/ssl/certs/5860aaa6.0
./etc/ssl/certs/ca-cert-GlobalSign_Root_R46.pem
./etc/ssl/certs/2b349938.0
./etc/ssl/certs/d887a5bb.0
./etc/ssl/certs/ca-cert-SZAFIR_ROOT_CA2.pem
./etc/ssl/certs/8160b96c.0
./etc/ssl/certs/ca-cert-D-TRUST_Root_Class_3_CA_2_2009.pem
./etc/ssl/certs/ca-cert-NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem
./etc/ssl/certs/9846683b.0
./etc/ssl/certs/ca-cert-Izenpe.com.pem
./etc/ssl/certs/feffd413.0
./etc/ssl/certs/ca-cert-OISTE_WISeKey_Global_Root_GB_CA.pem
./etc/ssl/certs/ca6e4ad9.0
./etc/ssl/certs/653b494a.0
./etc/ssl/certs/5f15c80c.0
./etc/ssl/certs/ca-cert-Buypass_Class_3_Root_CA.pem
./etc/ssl/certs/ca-cert-Entrust_Root_Certification_Authority_-_G2.pem
./etc/ssl/certs/ca-cert-Entrust_Root_Certification_Authority_-_EC1.pem
./etc/ssl/certs/02265526.0
./etc/ssl/certs/ca-cert-QuoVadis_Root_CA_3.pem
./etc/ssl/certs/ca-cert-ISRG_Root_X2.pem
./etc/ssl/certs/7f3d5d1d.0
./etc/ssl/certs/ca-cert-Baltimore_CyberTrust_Root.pem
./etc/ssl/certs/ed858448.0
./etc/ssl/certs/ca-cert-GlobalSign_ECC_Root_CA_-_R4.pem
./etc/ssl/certs/5931b5bc.0
./etc/ssl/certs/749e9e03.0
./etc/ssl/certs/ff34af3f.0
./etc/ssl/certs/9b5697b0.0
./etc/ssl/certs/6d41d539.0
./etc/ssl/certs/ca-cert-Actalis_Authentication_Root_CA.pem
./etc/ssl/certs/cc450945.0
./etc/ssl/certs/ca-cert-Starfield_Class_2_CA.pem
./etc/ssl/certs/ca-cert-GTS_Root_R4.pem
./etc/ssl/certs/ca-cert-Microsoft_RSA_Root_Certificate_Authority_2017.pem
./etc/ssl/certs/ca-cert-Microsec_e-Szigno_Root_CA_2009.pem
./etc/ssl/certs/ca-cert-Certigna_Root_CA.pem
./etc/ssl/certs/ca-cert-Amazon_Root_CA_4.pem
./etc/ssl/certs/ca-cert-DigiCert_Trusted_Root_G4.pem
./etc/ssl/certs/ca-cert-E-Tugra_Global_Root_CA_RSA_v3.pem
./etc/ssl/certs/ca-cert-emSign_Root_CA_-_C1.pem
./etc/ssl/certs/773e07ad.0
./etc/ssl/certs/e36a6752.0
./etc/ssl/certs/ca-cert-vTrus_Root_CA.pem
./etc/ssl/certs/ca-cert-SecureTrust_CA.pem
./etc/ssl/certs/ee64a828.0
./etc/ssl/certs/ca-cert-AC_RAIZ_FNMT-RCM.pem
./etc/ssl/certs/ca-cert-CFCA_EV_ROOT.pem
./etc/ssl/certs/ca-cert-QuoVadis_Root_CA_1_G3.pem
./etc/ssl/certs/ca-cert-D-TRUST_Root_Class_3_CA_2_EV_2009.pem
./etc/ssl/certs/0b1b94ef.0
./etc/ssl/certs/ca-cert-ISRG_Root_X1.pem
./etc/ssl/certs/406c9bb1.0
./etc/ssl/certs/ca-cert-IdenTrust_Public_Sector_Root_CA_1.pem
./etc/ssl/certs/ca-cert-Go_Daddy_Class_2_CA.pem
./etc/ssl/certs/fe8a2cd8.0
./etc/ssl/certs/b1159c4c.0
./etc/ssl/certs/fc5a8f99.0
./etc/ssl/certs/ca-cert-Certum_Trusted_Root_CA.pem
./etc/ssl/certs/b66938e9.0
./etc/ssl/certs/ca-cert-GDCA_TrustAUTH_R5_ROOT.pem
./etc/ssl/certs/ca-cert-UCA_Extended_Validation_Root.pem
./etc/ssl/certs/ca-cert-AffirmTrust_Commercial.pem
./etc/ssl/certs/ca-cert-DigiCert_TLS_RSA4096_Root_G5.pem
./etc/ssl/certs/2923b3f9.0
./etc/ssl/certs/ca-cert-BJCA_Global_Root_CA1.pem
./etc/ssl/certs/0f6fa695.0
./etc/ssl/certs/48bec511.0
./etc/ssl/certs/8d86cdd1.0
./etc/ssl/certs/ca-cert-Hongkong_Post_Root_CA_3.pem
./etc/ssl/certs/064e0aa9.0
./etc/ssl/certs/c28a8a30.0
./etc/ssl/certs/08063a00.0
./etc/ssl/certs/c01eb047.0
./etc/ssl/certs/ca-cert-SSL.com_Root_Certification_Authority_ECC.pem
./etc/ssl/certs/4bfab552.0
./etc/ssl/certs/ca-cert-COMODO_ECC_Certification_Authority.pem
./etc/ssl/certs/ca-cert-COMODO_RSA_Certification_Authority.pem
./etc/ssl/certs/ca-cert-OISTE_WISeKey_Global_Root_GC_CA.pem
./etc/ssl/certs/ca-cert-ANF_Secure_Server_Root_CA.pem
./etc/ssl/certs/e113c810.0
./etc/ssl/certs/ca-cert-Certum_EC-384_CA.pem
./etc/ssl/certs/5ad8a5d6.0
./etc/ssl/certs/1e09d511.0
./etc/ssl/certs/ca-cert-Certum_Trusted_Network_CA.pem
./etc/ssl/certs/ca-cert-AffirmTrust_Premium_ECC.pem
./etc/ssl/certs/b727005e.0
./etc/ssl/certs/5d3033c5.0
./etc/ssl/certs/2ae6433e.0
./etc/ssl/certs/93bc0acc.0
./etc/ssl/certs/ca-cert-Security_Communication_RootCA2.pem
./etc/ssl/certs/ca-cert-SecureSign_RootCA11.pem
./etc/ssl/certs/f249de83.0
./etc/ssl/certs/ca-cert-Entrust_Root_Certification_Authority_-_G4.pem
./etc/ssl/certs/54657681.0
./etc/ssl/ct_log_list.cnf.dist
./etc/ssl/openssl.cnf.dist
./etc/ssl/private
./etc/ssl/misc
./etc/ssl/misc/CA.pl
./etc/ssl/misc/tsget
./etc/ssl/misc/tsget.pl
./etc/periodic
./etc/periodic/monthly
./etc/periodic/15min
./etc/periodic/hourly
./etc/periodic/daily
./etc/periodic/weekly
./etc/shells
./etc/logrotate.d
./etc/logrotate.d/acpid
./etc/resolv.conf
./etc/ca-certificates.conf
./etc/ca-certificates
./etc/ca-certificates/update.d
./etc/ca-certificates/update.d/certhash
./home
./lib
./lib/libc.musl-x86_64.so.1
./lib/libapk.so.3.12.0
./lib/apk
./lib/apk/db
./lib/apk/db/triggers
./lib/apk/db/scripts.tar
./lib/apk/db/installed
./lib/apk/db/lock
./lib/modules-load.d
./lib/mdev
./lib/libcrypto.so.1.1
./lib/ld-musl-x86_64.so.1
./lib/libz.so.1.2.12
./lib/firmware
./lib/libssl.so.1.1
./lib/libz.so.1
./media
./media/usb
./media/floppy
./media/cdrom
./var
./var/run
./var/opt
./var/empty
./var/tmp
./var/log
./var/mail
./var/lib
./var/lib/apk
./var/lib/udhcpd
./var/lib/misc
./var/cache
./var/cache/apk
./var/cache/apk/APKINDEX.af244049.tar.gz
./var/cache/apk/APKINDEX.77a9a2bb.tar.gz
./var/cache/misc
./var/lock
./var/lock/subsys
./var/local
./var/spool
./var/spool/mail
./var/spool/cron
./var/spool/cron/crontabs
./root
./root/.config
./root/.config/caddy
./root/.config/caddy/autosave.json
./root/.ash_history
./mnt
./dev
./dev/core
./dev/stderr
./dev/stdout
./dev/stdin
./dev/fd
./dev/ptmx
./dev/urandom
./dev/zero
./dev/tty
./dev/full
./dev/random
./dev/null
./dev/shm
./dev/termination-log
./dev/mqueue
./dev/pts
./dev/pts/0
./dev/pts/ptmx
./sbin
./sbin/slattach
./sbin/fsck
./sbin/iprule
./sbin/mkmntdirs
./sbin/blockdev
./sbin/apk
./sbin/iptunnel
./sbin/raidautorun
./sbin/logread
./sbin/init
./sbin/poweroff
./sbin/udhcpc
./sbin/ldconfig
./sbin/inotifyd
./sbin/vconfig
./sbin/reboot
./sbin/loadkmap
./sbin/switch_root
./sbin/iproute
./sbin/halt
./sbin/iplink
./sbin/ip
./sbin/depmod
./sbin/ifconfig
./sbin/swapoff
./sbin/hwclock
./sbin/ifenslave
./sbin/ipneigh
./sbin/swapon
./sbin/mdev
./sbin/ipaddr
./sbin/findfs
./sbin/acpid
./sbin/klogd
./sbin/fbsplash
./sbin/blkid
./sbin/modinfo
./sbin/pivot_root
./sbin/losetup
./sbin/mkdosfs
./sbin/modprobe
./sbin/arp
./sbin/setconsole
./sbin/adjtimex
./sbin/rmmod
./sbin/fstrim
./sbin/tunctl
./sbin/lsmod
./sbin/nologin
./sbin/ifdown
./sbin/mkfs.vfat
./sbin/insmod
./sbin/ifup
./sbin/fdisk
./sbin/route
./sbin/watchdog
./sbin/getty
./sbin/mkswap
./sbin/nameif
./ingress-controller
/ #

Weird thing is that everything seems so clean...

francislavoie commented 7 months ago

Looks like you have on-demand enabled, but I don't see an ask endpoint configured. If that's the case, then any TLS connection with a hostname that resolves to your server will cause Caddy to attempt to issue a certificate. (The ask endpoint limits the allowed domains to ones approved by your server). This is a known attack vector, and in the latest versions of Caddy the ask endpoint is required.

I'm not sure what version you're using, and I don't know much about ingress config though, because it only supports a subset of Caddy's config and I don't use it myself.

timthelion commented 7 months ago

Any idea what the purpose of this attack could be? Is it just a DOS by someone who hates us?

francislavoie commented 7 months ago

It could be malicious, it could just be a bot scanning for possible hostname patterns. Couldn't say.

But you should not enable on-demand if you know the domains you need ahead of time. And you should definitely never enable it without an ask endpoint.

timthelion commented 7 months ago

OK, we've disabled it. Thank you for your help :) . We certainly get a huge amount of bot traffic. Mostly just random nonsense like looking for wpadmin (we run django apps).