RichJackson / cogstack

Database - Elasticsearch realtime mapping. With NLP goodiness.
Apache License 2.0
7 stars 2 forks source link

[Improvement] Add options to write a dummy placeholder for documents that failed processor step #33

Open hkkenneth opened 7 years ago

hkkenneth commented 7 years ago

When this option is on, even when a non-optional processor fails, it will trigger a record to be written to elastic search.

However, note that we possibly cannot catch failure in reader and writer in the same way...

hkkenneth commented 7 years ago

Further explanation:

Why cannot do this in reader: if reading an item fails, we know nothing about this item, nothing about document ID, patient ID etc.

Why cannot do this in writer: if writing to ElasticSearch has problem, writing the dummy placeholder to ES will not succeed...