search

github / policy-controller

Temporary GitHub managed Sigstore Policy Controller fork
Apache License 2.0
2 stars 1 forks source link

controller crashes with segmentation violation error when retrying to run an admissible pod #177

Closed falcorocks closed 1 month ago

falcorocks commented 3 months ago

Hello!

I think I found a bug. I tried to reproduce it using the upstream controller, but was not able to, so I'm opening this here. I have documented in detail my experimental setup here. TLDR is:

$ kubectl run falcorocks-learns --namespace=test --image=ghcr.io/falcorocks/learns@sha256:4ac65f23061de2faef157760fa2125c954b5b064bc25e10655e90bd92bc3b354
pod/falcorocks-learns created

$ kubectl run falcorocks-learns --namespace=test --image=ghcr.io/falcorocks/learns@sha256:4ac65f23061de2faef157760fa2125c954b5b064bc25e10655e90bd92bc3b354
Error from server (InternalError): Internal error occurred: failed calling webhook "policy.sigstore.dev": failed to call webhook: Post "https://webhook.github-policy-controller.svc:443/validations?timeout=10s": EOF

full logs from the crashed controller

{"level":"info","ts":1722955214.0927722,"logger":"fallback","caller":"webhook/main.go:132","msg":"Initializing TUF root from  => https://tuf-repo-cdn.sigstore.dev"}
main.go:149: {
  "gitVersion": "v0.10.0-github5",
  "gitCommit": "c294fd1281ad4b7e861f96fb3cb719b08deb4d62",
  "gitTreeState": "clean",
  "buildDate": "2024-07-09T19:35:00Z",
  "goVersion": "go1.22.0",
  "compiler": "gc",
  "platform": "linux/amd64"
}
main.go:228: Registering 3 clients
main.go:229: Registering 4 informer factories
main.go:230: Registering 7 informers
main.go:231: Registering 8 controllers
{"level":"info","ts":"2024-08-06T14:40:14.457Z","logger":"policy-controller","caller":"profiling/server.go:65","msg":"Profiling enabled: false","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.461Z","logger":"policy-controller","caller":"leaderelection/context.go:47","msg":"Running with Standard leader election","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.494Z","logger":"policy-controller","caller":"sharedmain/main.go:283","msg":"Starting configuration manager...","commit":"c294fd1"}
{"level":"info","ts":1722955214.595121,"logger":"fallback","caller":"injection/injection.go:63","msg":"Starting informers..."}
{"level":"info","ts":"2024-08-06T14:40:14.610Z","logger":"policy-controller","caller":"clusterimagepolicy/controller.go:102","msg":"Doing a global resync on ClusterImagePolicies due to ConfigMap changing or resync period.","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.611Z","logger":"policy-controller","caller":"trustroot/controller.go:67","msg":"Doing a global resync on TrustRoot due to ConfigMap changing or resync period.","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.696Z","logger":"policy-controller","caller":"webhook/webhook.go:218","msg":"Informers have been synced, unblocking admission webhooks.","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.696Z","logger":"policy-controller","caller":"sharedmain/main.go:311","msg":"Starting controllers...","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.696Z","logger":"policy-controller","caller":"injection/health_check.go:43","msg":"Probes server listening on port 8080","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.696Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.resource-conversion.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_e802eb03-8c8d-4852-98e5-d81905cca32b\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.696Z","logger":"policy-controller.resource-conversion","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.resource-conversion","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.webhookcertificates.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_249cf0b4-0b0d-44c0-be61-2d4de53ce9b6\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.WebhookCertificates","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.WebhookCertificates","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.policy.sigstore.dev-validating.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_a8cd42b6-4e50-4bb0-b747-15a13279d1eb\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.policy.sigstore.dev","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.policy.sigstore.dev","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.policy.sigstore.dev-mutating.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_8e32d06a-e062-4aed-a232-a73c93f4a67b\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.policy.sigstore.dev","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.697Z","logger":"policy-controller.policy.sigstore.dev","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.github.com.sigstore.policy-controller.pkg.reconciler.trustroot.reconciler.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_ecea11c5-bb67-4d79-bace-eb51210c39cb\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1","knative.dev/controller":"github.com.sigstore.policy-controller.pkg.reconciler.trustroot.Reconciler","knative.dev/kind":"policy.sigstore.dev.TrustRoot"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1","knative.dev/controller":"github.com.sigstore.policy-controller.pkg.reconciler.trustroot.Reconciler","knative.dev/kind":"policy.sigstore.dev.TrustRoot"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.github.com.sigstore.policy-controller.pkg.reconciler.clusterimagepolicy.reconciler.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_015fc30c-3df1-4068-a845-dfed63098d58\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1","knative.dev/controller":"github.com.sigstore.policy-controller.pkg.reconciler.clusterimagepolicy.Reconciler","knative.dev/kind":"policy.sigstore.dev.ClusterImagePolicy"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1","knative.dev/controller":"github.com.sigstore.policy-controller.pkg.reconciler.clusterimagepolicy.Reconciler","knative.dev/kind":"policy.sigstore.dev.ClusterImagePolicy"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.validating.clusterimagepolicy.sigstore.dev.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_749a928e-16dc-4998-b911-3116c8a49047\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller.validating.clusterimagepolicy.sigstore.dev","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller.validating.clusterimagepolicy.sigstore.dev","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1"}
I0806 14:40:14.698656       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.resource-conversion.00-of-01...
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller","caller":"leaderelection/context.go:149","msg":"policy-controller.defaulting.clusterimagepolicy.sigstore.dev.00-of-01 will run in leader-elected mode with id \"github-policy-controller-webhook-8496984cbc-cjchv_cc7e277c-f8e7-4c7a-9097-320dd3580fd7\"","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller.defaulting.clusterimagepolicy.sigstore.dev","caller":"controller/controller.go:486","msg":"Starting controller and workers","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:14.698Z","logger":"policy-controller.defaulting.clusterimagepolicy.sigstore.dev","caller":"controller/controller.go:496","msg":"Started workers","commit":"c294fd1"}
I0806 14:40:14.699204       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.webhookcertificates.00-of-01...
I0806 14:40:14.699532       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.policy.sigstore.dev-validating.00-of-01...
I0806 14:40:14.699795       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.policy.sigstore.dev-mutating.00-of-01...
I0806 14:40:14.700144       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.github.com.sigstore.policy-controller.pkg.reconciler.trustroot.reconciler.00-of-01...
I0806 14:40:14.700335       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.github.com.sigstore.policy-controller.pkg.reconciler.clusterimagepolicy.reconciler.00-of-01...
I0806 14:40:14.700597       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.validating.clusterimagepolicy.sigstore.dev.00-of-01...
I0806 14:40:14.700956       1 leaderelection.go:250] attempting to acquire leader lease github-policy-controller/policy-controller.defaulting.clusterimagepolicy.sigstore.dev.00-of-01...
{"level":"info","ts":"2024-08-06T14:40:45.530Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000928480), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":]string{\"application/json, */*\"}, \"Accept-Encoding\":]string{\"gzip\"}, \"Content-Length\":]string{\"2628\"}, \"Content-Type\":]string{\"application/json\"}, \"User-Agent\":]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc0008dd000), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:2628, TransferEncoding:]string(nil), Close:false, Host:\"webhook.github-policy-controller.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"192.168.65.3:63154\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00086e210), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0008bf9f0), pat:(*http.pattern)(0xc0007f6120), matches:]string(nil), otherValues:map[string]string(nil)}","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:45.536Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: null","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"test","knative.dev/name":"falcorocks-learns","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"docker-for-desktop"}
{"level":"info","ts":"2024-08-06T14:40:45.536Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"test","knative.dev/name":"falcorocks-learns","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"docker-for-desktop","admissionreview/uid":"fde10b05-3a28-4b55-9930-7edba851bd67","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2024-08-06T14:40:45.540Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000ab0240), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":]string{\"application/json, */*\"}, \"Accept-Encoding\":]string{\"gzip\"}, \"Content-Length\":]string{\"2732\"}, \"Content-Type\":]string{\"application/json\"}, \"User-Agent\":]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc0009b01c0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:2732, TransferEncoding:]string(nil), Close:false, Host:\"webhook.github-policy-controller.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"192.168.65.3:63162\", RequestURI:\"/validations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00086e370), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0009ba000), pat:(*http.pattern)(0xc0007f60c0), matches:]string(nil), otherValues:map[string]string(nil)}","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:52.096Z","logger":"policy-controller","caller":"webhook/validator.go:1266","msg":"Validated 1 policies for image ghcr.io/falcorocks/learns@sha256:4ac65f23061de2faef157760fa2125c954b5b064bc25e10655e90bd92bc3b354","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"test","knative.dev/name":"falcorocks-learns","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"docker-for-desktop"}
{"level":"info","ts":"2024-08-06T14:40:52.096Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"test","knative.dev/name":"falcorocks-learns","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"docker-for-desktop","admissionreview/uid":"b91be04c-cbc1-48c4-bb52-f2fac4b976b8","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2024-08-06T14:40:56.339Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000c065a0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":]string{\"application/json, */*\"}, \"Accept-Encoding\":]string{\"gzip\"}, \"Content-Length\":]string{\"2628\"}, \"Content-Type\":]string{\"application/json\"}, \"User-Agent\":]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc001094d40), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:2628, TransferEncoding:]string(nil), Close:false, Host:\"webhook.github-policy-controller.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"192.168.65.3:63154\", RequestURI:\"/mutations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00086e210), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc0010cb270), pat:(*http.pattern)(0xc0007f6120), matches:]string(nil), otherValues:map[string]string(nil)}","commit":"c294fd1"}
{"level":"info","ts":"2024-08-06T14:40:56.396Z","logger":"policy-controller","caller":"defaulting/defaulting.go:158","msg":"Kind: \"/v1, Kind=Pod\" PatchBytes: null","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"test","knative.dev/name":"falcorocks-learns","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"docker-for-desktop"}
{"level":"info","ts":"2024-08-06T14:40:56.396Z","logger":"policy-controller","caller":"webhook/admission.go:151","msg":"remote admission controller audit annotations=map[string]string(nil)","commit":"c294fd1","knative.dev/kind":"/v1, Kind=Pod","knative.dev/namespace":"test","knative.dev/name":"falcorocks-learns","knative.dev/operation":"CREATE","knative.dev/resource":"/v1, Resource=pods","knative.dev/subresource":"","knative.dev/userinfo":"docker-for-desktop","admissionreview/uid":"0bfaa010-f026-49a1-b9bd-c52676f9c594","admissionreview/allowed":true,"admissionreview/result":"nil"}
{"level":"info","ts":"2024-08-06T14:40:56.405Z","logger":"policy-controller","caller":"webhook/admission.go:93","msg":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000c74480), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":]string{\"application/json, */*\"}, \"Accept-Encoding\":]string{\"gzip\"}, \"Content-Length\":]string{\"2732\"}, \"Content-Type\":]string{\"application/json\"}, \"User-Agent\":]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc001095cc0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:2732, TransferEncoding:]string(nil), Close:false, Host:\"webhook.github-policy-controller.svc:443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"192.168.65.3:63162\", RequestURI:\"/validations?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00086e370), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc000e78190), pat:(*http.pattern)(0xc0007f60c0), matches:]string(nil), otherValues:map[string]string(nil)}","commit":"c294fd1"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1c3d260]

goroutine 842 [running]:
github.com/sigstore/sigstore-go/pkg/root.(*TrustedRoot).RekorLogs(0xc0008a8700?)
    github.com/sigstore/sigstore-go@v0.3.0/pkg/root/trusted_root.go:74
github.com/sigstore/sigstore-go/pkg/verify.VerifyArtifactTransparencyLog({0x347da30, 0xc0008a8700}, {0x3477fe0, 0x0}, 0x1, 0x1, 0x0)
    github.com/sigstore/sigstore-go@v0.3.0/pkg/verify/tlog.go:85 +0x289
github.com/sigstore/sigstore-go/pkg/verify.(*SignedEntityVerifier).VerifyTransparencyLogInclusion(0xc000bc3e90?, {0x347da30?, 0xc0008a8700?})
    github.com/sigstore/sigstore-go@v0.3.0/pkg/verify/signed_entity.go:618 +0x65
github.com/sigstore/sigstore-go/pkg/verify.(*SignedEntityVerifier).Verify(0xc000115880, {0x347da30, 0xc0008a8700}, {0xc000bc3e90?, {0xc0000a3d58?, 0x10?, 0xc0000a4808?}})
    github.com/sigstore/sigstore-go@v0.3.0/pkg/verify/signed_entity.go:475 +0xdb
github.com/sigstore/policy-controller/pkg/webhook.VerifiedBundles({0x3477fa0, 0xc000e78410}, {0x3477fe0?, 0x0?}, {0xc0010f7b90, 0x2, 0x2}, {0xc0000a3d58, 0x1, 0x1}, ...)
    github.com/sigstore/policy-controller/pkg/webhook/bundle.go:116 +0x28d
github.com/sigstore/policy-controller/pkg/webhook.ValidatePolicyAttestationsForAuthorityWithBundle({0x34750f8, 0xc000e4de00}, {0x3477fa0, 0xc000e78410}, {{0xc00054c150, 0xb}, 0x0, 0xc0004d8b00, 0x0, {0x0, ...}, ...}, ...)
    github.com/sigstore/policy-controller/pkg/webhook/validator.go:1028 +0x669
github.com/sigstore/policy-controller/pkg/webhook.ValidatePolicy.func1()
    github.com/sigstore/policy-controller/pkg/webhook/validator.go:542 +0x44d
created by github.com/sigstore/policy-controller/pkg/webhook.ValidatePolicy in goroutine 840
    github.com/sigstore/policy-controller/pkg/webhook/validator.go:516 +0x1e5
Stream closed EOF for github-policy-controller/github-policy-controller-webhook-8496984cbc-cjchv (policy-controller-webhook)
codysoyland commented 3 months ago

Thank you @falcorocks for the bug report! I will try to reproduce this locally, and I'll let you know what I figure out.

codysoyland commented 1 month ago

@falcorocks Apologies for the delay. I was able to reproduce this today and have a fix that I will release later this week. Again, I appreciate the detailed report!