TestEnterpriseSearch* fails for 8.16.0-SNAPSHOT

[pebrc]

=== RUN   TestEnterpriseSearchTLSDisabledVersionUpgradeToLatest8x/EnterpriseSearch_Pods_should_eventually_be_ready#01
Retries (30m0s timeout): ......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
    step.go:51: 
            Error Trace:    /go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94
            Error:          Received unexpected error:
                            pod test-ent-notls-v-up-8x-5wpr-ent-7976f98988-sqdnz was not upgraded (yet?) to match the expected specification
            Test:           TestEnterpriseSearchTLSDisabledVersionUpgradeToLatest8x/EnterpriseSearch_Pods_should_eventually_be_ready#01
{"log.level":"error","@timestamp":"2024-08-19T02:19:50.395Z","message":"continuing with additional tests","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","error":"test EnterpriseSearch Pods should eventually be ready failed","error.stack_trace":"github.com/elastic/cloud-on-k8s/v2/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:52\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/test.RunMutations\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/run_mutation.go:39\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/ent.TestEnterpriseSearchTLSDisabledVersionUpgradeToLatest8x\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/ent/ent_test.go:109\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1689"}
{"log.level":"info","@timestamp":"2024-08-19T02:19:50.395Z","log.logger":"e2e","message":"Running eck-diagnostics","service.version":"0.0.0-SNAPSHOT+00000000","service.type":"eck","ecs.version":"1.4.0","cluster":"eck-e2e-stack-8-16-0-sna-gubv-641","test":"TestEnterpriseSearchTLSDisabledVersionUpgradeToLatest8x","step":"EnterpriseSearch Pods should eventually be ready"}
I0819 02:20:05.860619   24092 request.go:697] Waited for 1.043763776s due to client-side throttling, not priority and fairness, request: GET:https://10.58.240.1:443/apis/networking.gke.io/v1beta1
    --- FAIL: TestEnterpriseSearchTLSDisabledVersionUpgradeToLatest8x/EnterpriseSearch_Pods_should_eventually_be_ready#01 (1800.00s)

Exception: Unexpected exception while running Enterprise Search: NoMethodError: undefined method `actastic_class' for #<AddAppSearchGatedFormFields:0x14a5e48e>
/usr/share/enterprise-search/lib/war/gems/gems/activemodel-6.1.7.7/lib/active_model/attribute_methods.rb:469:in `method_missing': undefined method `actastic_class' for #<AddAppSearchGatedFormFields:0x14a5e48e> (NoMethodError)
    from /usr/share/enterprise-search/lib/war/shared_togo/db/migrate/

[markjhoy]

@pebrc - is this error reproducible, or was this a one-time issue? If it's reproducible, can you point me at how I can test and / or see additional logs? Thanks!

[pebrc]

I would say so it has been failing since Aug 15th in each nightly test. I will share a link to diagnostics from our build system out of band.

[markjhoy]

We have a potential patch PR ready to go to hopefully fix this. See the OOB thread for details.