Same index getting deleted twice?

[mrdavidlaing]

Poking around http://logsearch.cityindextest5.co.uk/#/dashboard/elasticsearch/logsearch-purge-bot

I've noticed that the same index seems to get deleted more than once.

Consider the following two log sequences:

At 2013-10-25T23:27:38.355Z

checking index logstash-2013.09.12
deleting 63701 records
deleted emtpy index logstash-2013.09.12

At 2013-10-27T09:48:59.330Z

checking index logstash-2013.09.12
deleting 113459 records
deleted emtpy index logstash-2013.09.12

Two items of weirdness here:

1. A previously deleted index seems to reappear and be deleted again
2. The record count is different in each case.

[mrdavidlaing]

If you look for "old" records immediately after a purge, they are definately gone:

http://logsearch.cityindextest5.co.uk/#dashboard/temp/q8g9cq0wTHKV2AqX0sBlmw

The only thing I can think is that the deleted records are being restored from somewhere at some point, but I can't fathom where.

@bitpusher - Am I correct in saying that the purgebot is using the Delete by Query API?

@dpb587 - Any idea how records deleted in this fashion could automagically come back?

[mrdavidlaing]

@dpb587 - For completeness, the DELETE call is being issued via the tunnel:

ssh -v -N -L 127.0.0.1:9200:elasticsearch.srv-int.logsearch.cityindextest5.co.uk:9200 ubuntu@redis.logsearch.cityindextest5.co.uk -i ....

Is it possible that we're only deleting records from one of the elasticsearch nodes, and these are being "recovered" from the other one somehow?

[dpb587]

I don't believe this is related to #211; note the relatively small number of documents getting deleted.

I believe the purge bot is actually "working as intended" and is, in fact, related to @fandrei's testing of repeatedly importing test data from prior dates. Purge bot just regularly deletes them (hopefully after @fandrei's done looking at the results).

[sopel]

Closed as Won't Fix due to project being retired to the CityIndex Attic.