datahub-project / datahub

The Metadata Platform for your Data and AI Stack
https://datahubproject.io
Apache License 2.0
9.88k stars 2.93k forks source link

Connection reset by peer error when no interaction about 30 mins #3796

Closed amosannn closed 1 year ago

amosannn commented 2 years ago

Describe the bug If there is no interaction with the datahub-frontend for 30 minutes, ERROR 500 will appear. Then we need to return to the home page to refresh the page and wait for a while to recover.

image

The error message of datahub-gms shows Connection reset by peer

Caused by: com.linkedin.metadata.dao.exception.ESQueryException: Search query failed:
    at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:62)
    at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:89)
    at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:66)
    at com.linkedin.entity.client.JavaEntityClient.search(JavaEntityClient.java:239)
    at com.linkedin.datahub.graphql.resolvers.search.SearchResolver.lambda$get$0(SearchResolver.java:47)
    ... 2 common frames omitted
Caused by: java.io.IOException: Connection reset by peer
    at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:854)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
    at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:57)
    ... 6 common frames omitted

To Reproduce Steps to reproduce the behavior:

  1. Do not interact with datahub for 30 minutes
  2. Refresh page
  3. image

Desktop (please complete the following information):

Additional context We was deployed datahub by helm, and the command was as following:

kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub  -n datahub
kubectl create secret generic neo4j-secrets --from-literal=neo4j-password=datahub -n datahub

helm repo add datahub https://helm.datahubproject.io/
helm install prerequisites datahub/datahub-prerequisites
helm install datahub datahub/datahub  -n datahub

Maybe this error comes from elasticsearch: https://github.com/elastic/elasticsearch/issues/65213

weixi62961 commented 2 years ago

@amosannn hello, this bug hasn't beed fixed now. Do you have a workaround about it? Thanks

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

github-actions[bot] commented 1 year ago

This issue was closed because it has been inactive for 30 days since being marked as stale.