jhipster / generator-jhipster

JHipster is a development platform to quickly generate, develop, & deploy modern web applications & microservice architectures.
https://www.jhipster.tech
Apache License 2.0
21.4k stars 4.02k forks source link

ElasticSearch Exception on Newly Generated Application #24206

Open user-0209 opened 9 months ago

user-0209 commented 9 months ago
Overview of the issue

In a newly generated application, Elastic Search is causing an "Internal server error" by throwing an exception. Once an entity instance has been updated, it can be searched. On all other (not yet updated) entities of the same type, ElasticSearch doesn't find anything. On other entity types, where no entity yet has been updated, the Internal Server Error/ElasticSearch Exception still occurs.

Stack Traces and Messages
Application Message when no entity has been updated:
 
2023-11-13T21:32:20.676+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Enter: searchAS() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:32:20.676+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : REST request to search for a page of AS for query Kutch 
2023-11-13T21:32:20.677+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Enter: search() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:32:20.677+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Request to search for a page of AS for query Kutch 
2023-11-13T21:32:20.830+01:00 ERROR 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Exception in search() with cause = 'co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed' and exception = '[es/search] failed: [search_phase_execution_exception] all shards failed' 

org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:102)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateException(ElasticsearchExceptionTranslator.java:63)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:625)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.doSearch(ElasticsearchTemplate.java:341)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.search(ElasticsearchTemplate.java:334)
    at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:492)
    at com.mycompany.myapp.repository.search.ASearchRepositoryInternalImpl.search(ASearchRepository.java:53)
    at com.mycompany.myapp.repository.search.ASearchRepositoryInternalImpl.search(ASearchRepository.java:48)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:237)
    at com.mycompany.myapp.repository.search.$Proxy212.search(Unknown Source)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:288)
....
...
Caused by: co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
    at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:334)
    at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:154)
    at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1882)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.lambda$doSearch$14(ElasticsearchTemplate.java:341)
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:623)
    ... 246 common frames omitted

2023-11-13T21:32:20.879+01:00  WARN 262279 --- [  XNIO-1 task-7] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed] 
Messages upon successful search on an previously updated entity:
2023-11-13T21:34:07.349+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Enter: searchAS() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:34:07.350+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : REST request to search for a page of AS for query Kutch 
2023-11-13T21:34:07.350+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Enter: search() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:34:07.350+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Request to search for a page of AS for query Kutch 
2023-11-13T21:34:07.632+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Exit: search() with result = Page 1 of 1 containing com.mycompany.myapp.service.dto.ADTO instances 
2023-11-13T21:34:07.633+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Exit: searchAS() with result = <200 OK OK,[ADTO{id=1, firstName='Kenya', lastName='Kutch'}],[X-Total-Count:"1", Link:"; rel="last",; rel="first""]> 
Messages upon failed search on a not-yet updated entity:
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Enter: searchAS() with argument[s] = [Reta, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : REST request to search for a page of AS for query Reta 
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Enter: search() with argument[s] = [Reta, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Request to search for a page of AS for query Reta 
2023-11-13T21:35:13.364+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Exit: search() with result = Page 1 of 0 containing UNKNOWN instances 
2023-11-13T21:35:13.365+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Exit: searchAS() with result = <200 OK OK,[],[X-Total-Count:"0", Link:"; rel="last",; rel="first""]> 
Motivation for or Use Case

A newly generated application should work out of the box. Apparently ElasticSearch has to build datastructures that do not exist on a new application. Also, initial indexing does not happen on fake data. All of this creates much irritation because you are not told, what the problem is.

Reproduce the error

A sample JDL is given below. JDL is in the otherwise empty application folder.

  1. jhipster jdl application.jdl --force
  2. in a second terminal window: docker compose -f src/main/docker/elasticsearch.yml up
  3. ./mvnw clean && ./mvnw
  4. browse to http://localhost:8080
  5. sign in
  6. navigate to Entities/A
  7. put the last name of the first instance (in my case "Kutch") into the search box and search: first sequence of messages appears
  8. clear the search, select the edit button on that entity. Click save.
  9. Again put the last name into the search box: search is successfull with the second sequence of messages
  10. put the last name of the second instance into the search box and search: not successfull, third sequence of messages
  11. navigate to Entities/B: put a search term. Again Internal Server Error happens, repeat 8. und 9.
Related issues

https://github.com/jhipster/generator-jhipster/issues/22603

Suggest a Fix
  1. The ElasticSearch exception should not just lead to "Internal server error", but be more specific.
  2. When the app is initializing the database (mvn clean), it should also initialize ElasticSearch (important during development cycles).
JHipster Version(s)
JHipster configuration
.yo-rc.json file
{
  "generator-jhipster": {
    "applicationIndex": 0,
    "applicationType": "monolith",
    "authenticationType": "jwt",
    "baseName": "espaginationconflict",
    "buildTool": "maven",
    "cacheProvider": "ehcache",
    "clientFramework": "angular",
    "creationTimestamp": 1699883023373,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "devServerPort": 4200,
    "enableHibernateCache": true,
    "enableSwaggerCodegen": true,
    "enableTranslation": true,
    "entities": [
      "A",
      "B",
      "C"
    ],
    "jhipsterVersion": "8.0.0",
    "languages": [
      "en",
      "de",
      "fr"
    ],
    "lastLiquibaseTimestamp": 1699883203000,
    "messageBroker": false,
    "microfrontends": [],
    "nativeLanguage": "en",
    "packageFolder": "com/mycompany/myapp",
    "packageName": "com.mycompany.myapp",
    "prodDatabaseType": "mariadb",
    "reactive": false,
    "searchEngine": "elasticsearch",
    "serviceDiscoveryType": false,
    "testFrameworks": [
      "gatling",
      "cucumber"
    ],
    "websocket": "spring-websocket",
    "withAdminUi": true
  }
}
JDL definitions
    application {
  config {
    jhipsterVersion "8.0.0"
    creationTimestamp 1699883023373
    baseName espaginationconflict
    packageName com.mycompany.myapp
    applicationType monolith
    buildTool maven
//    jhiPrefix jhi
//  entitySuffix false  // do not enable, will cause complation errors
//    dtoSuffix DTO
    authenticationType jwt
    withAdminUi true
    enableSwaggerCodegen true
    databaseType sql
    devDatabaseType h2Disk
    prodDatabaseType mariadb
    searchEngine elasticsearch
    websocket spring-websocket
    cacheProvider ehcache
    enableHibernateCache true
    clientFramework angular
    enableTranslation true
    nativeLanguage en
    languages [en, de, fr]
    testFrameworks [gatling, cucumber]
    messageBroker false
    serviceDiscoveryType false
    microfrontends []
    reactive false
  }
    entities *
}
entity A {
    firstName String
    lastName String
}
entity B {
    shortName String
    longName String
}
entity C {
    shortIdent String
    longIdent String
}
service  all with serviceImpl
dto      all with mapstruct
filter   all
paginate all with pagination
search   all with elasticsearch
  
Entity configuration(s) entityName.json files generated in the .jhipster directory
Browsers and Operating System

Ubuntu 22.04 Firefox: 119.0.1

Environment and Tools

openjdk version "17.0.8.1" 2023-08-24 OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu122.04) OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu122.04, mixed mode, sharing) git version 2.34.1 node: v20.9.0 npm: 10.1.0 Docker version 24.0.7, build afdd53b

aturyng commented 9 months ago

I tried to generate an app with the configuration below. I don't get an error anymore (probably because the pagination is turned off), but the search simply delivers no results. I turned the faker off and added my own entries. I also tried to add "generator-jhipster-es-entity-reindexer", but apparently it didn't help.

@user-0209 So you still don't have a solution? At least some manual fix after the generation?

{
  "generator-jhipster": {
    "applicationType": "monolith",
    "authenticationType": "oauth2",
    "baseName": "kitakeiler",
    "buildTool": "gradle",
    "cacheProvider": "ehcache",
    "clientFramework": "angular",
    "clientTestFrameworks": ["cypress"],
    "clientTheme": "journal",
    "clientThemeVariant": "primary",
    "creationTimestamp": 1699909007890,
    "cypressAudit": false,
    "cypressCoverage": false,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "devServerPort": 4200,
    "enableGradleEnterprise": null,
    "enableHibernateCache": true,
    "enableSwaggerCodegen": false,
    "enableTranslation": true,
    "entities": [
      "AcquisitionTarget",
      "Agent",
      "ChangeEvent",
      "ColoredCategory",
      "ContactInfo",
      "Country",
      "Employee",
      "Facility",
      "FacilityType",
      "Guidelines",
      "Notification",
      "StatusAdvancement",
      "StatusCampaign",
      "StatusInterest"
    ],
    "gradleEnterpriseHost": null,
    "jhipsterVersion": "8.0.0",
    "languages": ["de", "en"],
    "lastLiquibaseTimestamp": 1699909847000,
    "messageBroker": false,
    "microfrontend": null,
    "microfrontends": [],
    "nativeLanguage": "de",
    "otherModules": [
      {
        "name": "generator-jhipster-es-entity-reindexer",
        "version": "1.0.1"
      }
    ],
    "packageName": "com.company",
    "prodDatabaseType": "postgresql",
    "reactive": false,
    "searchEngine": "elasticsearch",
    "serverPort": null,
    "serverSideOptions": ["searchEngine:elasticsearch"],
    "serviceDiscoveryType": false,
    "skipUserManagement": true,
    "testFrameworks": ["gatling", "cypress"],
    "websocket": false,
    "withAdminUi": true
  }
}
user-0209 commented 9 months ago

I don't have a solution at this moment. I'm trying to make https://github.com/Ebsan/generator-jhipster-es-entity-reindexer work. I'm pretty sure its just about rebuilding the index.

github-actions[bot] commented 9 months ago

JHipster has completed the sample check .yo-rc.json: valid Entities JDL: Application: successfully generated Frontend check: success Backend check: success E2E check: success

This check uses jhipster info output from the issue description to generate the sample. Bug report that does not contain this information will be marked as invalid.

user-0209 commented 9 months ago

Meanwhile I undestand that in version 8 jhipster adopted the ES module standard. Also, some exports might have changed and that breaks not only https://github.com/Ebsan/generator-jhipster-es-entity-reindexer but also most of the other generators not immediately part of the jhipster repository. However, I need this reindexing functionality and would go forward to update the generator. Can somebody provide a pointer to a sample generator that is up to speed with jhipster 8 and not part of the jhipster repository? Or, point me to documentation about creating a generator from scratch? That would be helpful.

DanielFran commented 9 months ago

@user-0209 I believe you can use https://github.com/jhipster/generator-jhipster-entity-audit as an example. Updated documentation is available here, any feedback is welcome

aturyng commented 9 months ago

@user-0209 I found this sample with elasticsearch on version 8. https://github.com/jhipster/jhipster-sample-app-elasticsearch

Maybe it is of any to you.

user-0209 commented 9 months ago

@user-0209 I found this sample with elasticsearch on version 8. https://github.com/jhipster/jhipster-sample-app-elasticsearch

Maybe it is of any to you.

Thank you for responding. In fact, I find it amazing, how fast I'm getting responses in this Jhipster context. However, although the project has been upgraded to 8.0.0, it exhibits the same weired behaviour as my newly generated application. In fact, it is even more strange: The BankAccount and Label entities can be searched without throwing exception, but matching entities would not be found. The Operation entity is throwing an exception. In all cases, after just updating an entity (click edit, then save), that specific entity instance can be found, all others of the same type still not. I guess, there has been a difference, how these three entities have been generated. Altogether, this makes it very difficult to get into a strictly repeatable development/testing flow. That's why I'm so insisting on that. I hope, I can achieve something with the blueprint link, which appears to open another box.... Anyways, I'll try to solve this and report back.

mshima commented 9 months ago

Our elasticsearch tests needs improvements to avoid regressions like this one.

aturyng commented 9 months ago

By the way, if someone is interested, I positively tested running the backend generated with the latest version 7 and the frontend with version 8. The search works. You only need to change the GET mappings in all *Resource.java files from @GetMapping("/_search/RESOURCE") to @GetMapping("/RESOURCE/_search"), which is really not hard to do.

This way you get Angular 16, which is nice.

aturyng commented 8 months ago

@user-0209 Any progress in the meanwhile?

user-0209 commented 8 months ago

@aturyng: The problem is not ES as such. Once the index for the entity is in place, things work (for the entity instances that have been updated). The problem is with the preloaded fake-data. Not sure, if I'm old-fashioned, but I want defined, repeatable test data in the development cycle. Earlier (I think version 6), I had one of those re-indexing extensions in place and it worked. However, and here is the point: since few versions so many things have changed that all these blueprints don't work and apparently, nobody is taking the effort, to update them (guess why). So I embarked on porting one of the reindexing modules to a proper blueprint, but I have to say, it is very time-consuming due to a complete lack of documentation of those jHipster base classes, but not only. The blueprints chapter can only serve as a teaser, re-eingineering the base classes is necessary. In summary, I'd highly appreciate JSDoc for the baseclasses and in the blueprints documentation experts should say, when (scenario) you want which base class. Amongst the core-team maintained official blueprints there should be an (almost) empty blueprint that is up to speed with the respective JHipster version.

Having said this, I'm still much in favour of getting the application generated, because otherwise you'd end in another tunnel. Thanks to the core team. Let me know, if I'm missing some documentation.

Nenzyz commented 8 months ago

The issue seems to be with new elasticsearch. As a workaround I've used elasticsearch version from Jhipster 7 and it works very good. Version 7.17.4 –> OK, version 8.7.1 -> NOT OK. I'm pretty sure protocol changed on ES side, since Jhipster 7 with ES v8.7.1 also gets an error.

aturyng commented 8 months ago

@Nenzyz Thank you for your input! Unfortunately, when I simply change the version in the docker container from 8.7.1 to 7.17.4 -- it does not solve the problem. Have you done more than this? Can you maybe share your relevant config snippets with us?

Nenzyz commented 8 months ago

@Nenzyz Thank you for your input! Unfortunately, when I simply change the version in the docker container from 8.7.1 to 7.17.4 -- it does not solve the problem. Have you done more than this? Can you maybe share your relevant config snippets with us?

The only change from the freshly created project was changes in src/main/docker/elasticsearch.yml version, and for sure restart of the container. It would be nice to know if the error you are getting the same as reporter has or something else?

What doesn't work still is reindexing of existing entities, but this is unrelated to this issue.

(I won't post in the old issue, since this one is referenced in the old)

Nenzyz commented 8 months ago

I think I know what happened here, give me some time I'll get a reason and a way to solve it.

aturyng commented 7 months ago

Unfortunately, jhipster 8.1.0 does not solve the problem. I get exactly the same error.

nyenloop commented 2 months ago

The same error with me, anyone can help to fix it?

Tcharl commented 2 months ago

Hi,

Not a real fix, but the jhipster-reindexer module can be used to resync entities in search engine

nyenloop commented 2 months ago

@Tcharl thank you for your reply. I faced this error with jhipster-reindexer. I have never used jhipster modules. Could you let me know how to use it?

$ yo jhipster-es-entity-reindexer Error jhipster-es-entity-reindexer

Cannot find module 'C:\Users\Nyen.npm-packages\node_modules\generator-jhipster-es-entity-reindexer\node_modules\generator-jhipster\dist\generators\generator-base\index.js' TypeError [ERR_INVALID_ARG_TYPE]: The "code" argument must be of type number. Received type string ('MODULE_NOT_FOUND')

Tcharl commented 2 months ago

It's kind of an old one: the class has to be adapted a bit