search

cessda / cessda.cvs.two

Apache License 2.0
0 stars 2 forks source link

CVS application is unavailable - 503 error #454

Closed cessda-bitbucket-importer closed 1 year ago

cessda-bitbucket-importer commented 1 year ago

Original report on BitBucket by John Shepherdson (GitHub: john-shepherdson).

As at 2022-11-09 17:08:09 CET

cvs-frontend-0 pod is unstable. Many restarts. Logs say:
Wrapped by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vocabularyServiceImpl' defined in file [/app/classes/eu/cessda/cvs/service/impl/VocabularyServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 10; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vocabularyEditorSearchRepository': Invocation of init method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository]: Constructor threw exception; nested exception is java.lang.IllegalArgumentException: mapper [titleLt.Key] of different type, current_type [keyword], merged_type [icu_collation_keyword]I deleted the cvs-frontend-0 pod. I was reinstantiated but failed with same error.

I deleted the cvs-frontend-0 pod. It was automatically reinstantiated but failed with same error.

Status is:

back-off 5m0s restarting failed container=frontend pod=cvs-frontend-0_cvs(f85fb86f-88a0-476a-aca3-266582b03ba4): CrashLoopBackOff

cessda-bitbucket-importer commented 1 year ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

Back up at 2022-11-10 09:48:08 CET

cessda-bitbucket-importer commented 1 year ago

Original comment by John Shepherdson (GitHub: john-shepherdson).

Cause appears to be erroneous use of image tag master-latest on latest development build.

Investigating how tag came to be used.

cessda-bitbucket-importer commented 1 year ago

Original comment by Matthew Morris (GitHub: matthew-morris-cessda).

The current theory is that this was caused by a manual deployment to production using the cessda.cvs.deploy job in Jenkins, rather than the cessda.cvs.deploy.prod job.

The cessda.cvs.deploy job should be restricted so it can only be triggered by upstream jobs, preventing further cases of this issue.

matthew-morris-cessda commented 1 year ago

This incident has been resolved