rorydavidson
commented
4 years ago Hi @BillyYarosh-WellSky. Could you please tell us what version/edition you are trying to import?
Closed BillyYarosh-WellSky closed 4 years ago
rorydavidson
commented
4 years ago Hi @BillyYarosh-WellSky. Could you please tell us what version/edition you are trying to import?
BillyYarosh-WellSky
commented
4 years ago The most recent from this month. SnomedCT_USEditionRF2_PRODUCTION_20200301T120000Z
Adding some of the backtrace as well.
snowstorm | 2020-05-19 17:16:58.902 WARN 1 --- [/O dispatcher 1] org.elasticsearch.client.RestClient : request [GET http://es:9200/concept/concept/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&search_type=dfs_query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-6.8.3-0c48c0e "Deprecated: the number of terms [290354] used in the Terms Query request has exceeded the allowed maximum of [65536]. This maximum can be set by changing the [index.max_terms_count] index level setting."]
elasticsearch | [2020-05-19T17:16:58,999][WARN ][o.e.d.i.q.TermsQueryBuilder] [3fo_RQU] Deprecated: the number of terms [290354] used in the Terms Query request has exceeded the allowed maximum of [65536]. This maximum can be set by changing the [index.max_terms_count] index level setting.
snowstorm | 2020-05-19 17:16:59.912 WARN 1 --- [/O dispatcher 1] org.elasticsearch.client.RestClient : request [GET http://es:9200/concept/concept/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&scroll=60000ms&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-6.8.3-0c48c0e "Deprecated: the number of terms [290354] used in the Terms Query request has exceeded the allowed maximum of [65536]. This maximum can be set by changing the [index.max_terms_count] index level setting."]
snowstorm | 2020-05-19 17:17:15.281 WARN 1 --- [pool-2-thread-3] o.s.s.c.d.s.SemanticIndexUpdateService : The following concepts have been referred to in relationships but are missing or inactive: {157265008, 2837008, 63479002, 329497002, 65502005, 158094009, 156073000, 268460000, 238951005, 52988006, 302835009, 149213005, 193462001, 133858001, 113218004, 230456007, 187633009, 270516002, 240197007, 253549006, 106763003, 81701001, 150062003, 107280007, 234975001, 109993000, 156072005, 203045001, 128623006, 203082005, 255637000, 333032003, 201101007, 161891005, 239107007, 195042002, 155441006, 200722003, 150617003, 11953005, 195742007, 284009009, 54819005, 21267007, 282831004, 273945008, 154283005, 273951003, 1033009, 255632006, 234350007, 354034000, 366979004, 324936002, 241047004, 156370009, 211364004, 193982009, 109501002, 123250005, 155597006, 105472004, 156410002, 17775009, 122456005, 273948005, 154826009, 134209002}
elasticsearch | [2020-05-19T17:17:15,458][WARN ][o.e.d.i.q.TermsQueryBuilder] [3fo_RQU] Deprecated: the number of terms [287731] used in the Terms Query request has exceeded the allowed maximum of [65536]. This maximum can be set by changing the [index.max_terms_count] index level setting.
snowstorm | 2020-05-19 17:17:15.526 WARN 1 --- [/O dispatcher 1] org.elasticsearch.client.RestClient : request [GET http://es:9200/semantic/queryconcept/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&scroll=60000ms&search_type=query_then_fetch&batched_reduce_size=512] returned 1 warnings: [299 Elasticsearch-6.8.3-0c48c0e "Deprecated: the number of terms [287731] used in the Terms Query request has exceeded the allowed maximum of [65536]. This maximum can be set by changing the [index.max_terms_count] index level setting."]
snowstorm | 2020-05-19 17:17:15.528 INFO 1 --- [pool-2-thread-3] o.snomed.snowstorm.core.util.TimerUtil : Timer TC index inferred: Collect existingDescendants from QueryConcept. took 17.577 secondsThanks... and what was the import config that you used ?
BillyYarosh-WellSky
commented
4 years ago POST /imports/
{
"branchPath": "MAIN",
"createCodeSystemVersion": true,
"type": "FULL"
}POST /imports/{id}/archive.
I will note that there was also a lock on MAIN I had to remove. I had a bad file fail the day before. Destroyed the environment and rebuilt. After rebuilding I was surprised to see a lock. I assumed it was some sort of state issue and cleared the lock. This error occurred after importing again. It got pretty far along before failing.
kaicode
commented
4 years ago Hi @BillyYarosh-WellSky, thanks for reaching out.
It sounds like the deployment is in an inconsistent state. The MAIN branch will not have a lock if the environment is clean, it sounds like the indices were kept in your Elasticsearch cluster.
I recommend dropping and recreating the indicies before starting a new clean import. Use the argument --delete-indices when starting Snowstorm to have it do this.
See https://github.com/IHTSDO/snowstorm/blob/master/docs/loading-snomed.md#command-line be careful not to leave that flag hanging around! 😄
Please do let us know how you get on. Cheers.
BillyYarosh-WellSky
commented
4 years ago OK, I'll take a look at this again today. Will report back.
BillyYarosh-WellSky
commented
4 years ago By deleting and re-creating indices, I was able to complete an import.
Hi team, very new to SNOWMED CT and attempting to run an import job. I run into an error on a specific concept. It looks like a rollback due to a too large request entity. Running this through docker compose currently.