elasticsearch timeout when importing data

[hpfast]

So the 'index already exists' error is harmless as per https://github.com/histograph/api/issues/72, but what about this timeout I subsequently get from elasticsearch? Anything known about this? I'm still looking for something wrong with my installation..

$ tail -f .pm2/logs/histograph-core-error-0.log                                                                                                                                                                                                       [12/12]
{ [Error: [index_already_exists_exception] already exists, with: {"index":"cshapes"}]
  status: '400',
  message: '[index_already_exists_exception] already exists, with: {"index":"cshapes"}',
  path: '/cshapes',
  query: {},
  body: '{"settings":{"number_of_shards":5,"number_of_replicas":0,"analysis":{"analyzer":{"lowercase":{"type":"custom","filter":" lowercase","tokenizer":"keyword"}}}},"mappings":{"_default_":{"properties":{"geometry":{"precision":"1m","tree":"quadtree","type":"geo_shape"
},"uri":{"index":"not_analyzed","type":"string"},"id":{"index":"not_analyzed","store":true,"type":"string"},"type":{"index":"not_analyzed","type":"string"},"name":{"fields":{"analyzed":{"index":"analyzed","store":true,"type":"string"},"exact":{"analyzer":"lowercase","sto
re":true,"type":"string"}},"type":"string"},"dataset":{"index":"not_analyzed","type":"string"},"validSince":{"format":"date_optional_time","type":"date"},"validUntil":{"format":"date_optional_time","type":"date"}}}}}',
  statusCode: 400,
  response: '{"error":{"root_cause":[{"type":"index_already_exists_exception","reason":"already exists","index":"cshapes"}],"type":"index_already_exists_exception","reason":"already exists","index":"cshapes"},"status":400}',
  toString: [Function],
  toJSON: [Function] } '[index_already_exists_exception] already exists, with: {"index":"cshapes"}'
Error: Request Timeout after 30000ms
    at null.<anonymous> (/opt/histograph/core/node_modules/elasticsearch/src/lib/transport.js:303:15)
    at Timer.listOnTimeout (timers.js:119:15)

[hpfast]

I found out things ARE getting into ES, so that's hopeful. I suspect it's an environment issue i.e. my server installation. Will keep looking. Any suggestions appreciated.

[hpfast]

just a memory problem I think ...

[wires]

hi I've also seen this before on AWS and locally, both were indeed memory related. ES timeout's are not OK, this does lead to data loss, we don't have fallback code for this (that retries, etc)... You can use X-Force header when uploading to API, that will not perform a diff and reindex everything into ES (amongst other things)