search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
207 stars 83 forks source link

Snowstorm experiencing outages #155

Closed kaboyles closed 4 years ago

kaboyles commented 4 years ago

We are experiencing outagages on our snowstorm server. Logs attached and below, any insight will be appreciated.

We have two load balanced nodes with mirrored snowstorm servers connected to mirrored elasticsearch

Thanks!

snowstorm error.txt

mime type error snowstorm.txt

Logs:

Sep 17 13:08:28 q-kot-sm0001.qa.ld.nhn.no java[10503]: 2020-09-17 13:08:28.073 INFO 10503 — [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Topping up IdCache(0 - 02) by 4000 Sep 17 13:08:28 q-kot-sm0001.qa.ld.nhn.no java[10503]: 2020-09-17 13:08:28.076 ERROR 10503 — [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Failed to top-up IdCache(0 - 02) with 4000 identifiers ***these are the first lines

lines 1-62/668739 0% ***so it has shown up quite a few times

Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: 2020-09-18 11:09:58.982 INFO 10503 — [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Topping up IdCache(0 - 00) by 1000 Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: 2020-09-18 11:09:58.983 ERROR 10503 — [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Failed to top-up IdCache(0 - 00) with 1000 identifiers Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.apache.http.util.Asserts.check(Asserts.java:46) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:344) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:326) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestClient.performRequest(RestClient.java:192) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:428) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:414) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.springframework.data.elasticsearch.rest.ESRestHighLevelClient.performRequestAndParseEntity(ESRestHighLevelClient.java:47) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:355) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.springframework.data.elasticsearch.rest.ElasticsearchRestClient.doExecute(ElasticsearchRestClient.java:214) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.doSearch(ElasticsearchTemplate.java:979) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.queryForPage(ElasticsearchTemplate.java:280) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.queryForPage(ElasticsearchTemplate.java:275) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.queryForList(ElasticsearchTemplate.java:302) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.snomed.snowstorm.core.data.services.identifier.LocalRandomIdentifierSource.findExistingIdentifiersInAnyBranch(LocalRandomIdentifierSource.java:83) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.snomed.snowstorm.core.data.services.identifier.LocalRandomIdentifierSource.reserveIds(LocalRandomIdentifierSource.java:56) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.snomed.snowstorm.core.data.services.identifier.IdentifierCacheManager.topUp(IdentifierCacheManager.java:112) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.snomed.snowstorm.core.data.services.identifier.IdentifierCacheManager.checkTopUpRequired(IdentifierCacheManager.java:95) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at org.snomed.snowstorm.core.data.services.identifier.IdentifierCacheManager.run(IdentifierCacheManager.java:60) Sep 18 11:09:58 q-kot-sm0001.qa.ld.nhn.no java[10503]: at java.lang.Thread.run(Thread.java:748)

kaicode commented 4 years ago

Hi @kaboyles,

Thanks for the logs. This message itself, although looks bad, can safely be ignored if you are not using the Snowstorm instance for authoring / creating SNOMED CT content. The log shows that Snowstorm is attempting to generate some SNOMED identifiers and check their uniqueness in case someone starts authoring.

I would expect this to run and then just go quiet because the identifiers won't be needed. Is Snowstorm able to load content and responding normally? Are you able to load the MAIN branch on your server? For example https://snowstorm.ihtsdotools.org/snowstorm/snomed-ct/branches/MAIN (public read-only server).

cogliostro commented 4 years ago

The server wasn't responding as expected when we saw these errors. We suspect it has something to do with snowstorm starting before elasticsearch. Could this result in errors like these? Should snowstorm start working once elasticsearch starts up, or would it need to be restarted?

kaboyles commented 4 years ago

Hi Kai, thanks for your help! We are using Snowstorm for authoring/creating content for our extension. I haven't seen the exact lgos, but believe this started back on September 6th.

kaicode commented 4 years ago

@cogliostro Elasticsearch must be started and be online before Snowstorm is started. If Elasticsearch is down Snowstorm will need to be restarted once Elasticsearch is up again. I believe this will solve your issue.

kaicode commented 4 years ago

Closing for now as I think the question has been answered but please get in touch if you are still experiencing problems.

kaboyles commented 4 years ago

Okay to close.

So far it looks like it was the order at start up.

Thanks, Kim

From: Kai Kewley notifications@github.com Sent: Tuesday, September 22, 2020 4:33 PM To: IHTSDO/snowstorm snowstorm@noreply.github.com Cc: Kim Ann Boyles Kim.Ann.Boyles@ehelse.no; Mention mention@noreply.github.com Subject: Re: [IHTSDO/snowstorm] Snowstorm experiencing outages (#155)

Closing for now as I think the question has been answered but please get in touch if you are still experiencing problems.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/IHTSDO/snowstorm/issues/155#issuecomment-696761518, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOQOIWVFTF5ALEOETVHUFY3SHCYSTANCNFSM4RSDG2IQ.