Closed sukantasaha closed 9 years ago
Guys any update ?
Thanks for #188 @dadoonet , but why do you think this happens in the first place? Is it possible that there is a bug in snapshot creation that makes it work with files that are still being modified?
@otisg @imotov might know if this could happen. AFAIK, it tries to copy immutable segments so I don't think this could happen.
PR #188 only tries to secure copy over the wire by checking bytes written on both side. But you are right if there is a bug in snapshot creation, this won't fix it.
Which version of elasticsearch was the snapshot created with?
We are using es version : elasticsearch 1.4.1 cloud-aws : elasticsearch-cloud-aws-2.4.0
On Thu, Mar 5, 2015 at 2:40 AM, Igor Motov notifications@github.com wrote:
Which version of elasticsearch was the snapshot created with?
— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/186#issuecomment-77248111 .
@sukantasaha I understand that 1.4.1 is the version that you are currently using. But I am trying to figure out which version of elasticsearch this index was first snapshotted with. You said that you are snapshotting on a regular basis. So, judging from the name, the index was created in June 2014. Is this when you created the first snapshot of this index or was it later on? If this index was first snapshotted back then and you were staying with the latest (more or less) available version of elasticsearch we can assume that this index was first snapshotted by elasticsearch v1.1 or v1.2. Was this the case?
This cluster was created on Thu, Jul 2014 with elasticsearch-1.2.1 and may be we migrated this index . So we snapshot on regular basis, Later when we updated the es version to elasticsearch-1.4.1-SNAPSHOT we created a new repository and there we created the snapshot.
On Thu, Mar 5, 2015 at 10:33 AM, Igor Motov notifications@github.com wrote:
@sukantasaha https://github.com/sukantasaha I understand that 1.4.1 is the version that you are currently using. But I am trying to figure out which version of elasticsearch this index was first snapshotted with. You said that you are snapshotting on a regular basis. So, judging from the name, the index was created in June 2014. Is this when you created the first snapshot of this index or was it later on? If this index was first snapshotted back then and you were staying with the latest (more or less) available version of elasticsearch we can assume that this index was first snapshotted by elasticsearch v1.1 or v1.2. Was this the case?
— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/186#issuecomment-77307099 .
@sukantasaha what did you mean by "may be we migrated this index"? Could it be possible that this index was originally created with older version of elasticsearch? I am just wondering if you are hitting elasticsearch/elasticsearch#9140. Would it be possible for you as an experiment to restore this index using a newer version of elasticsearch (v1.4.3 or higher)?
We did the same test, Launched a new cluster with 1.4.3 version and there we tried to restore that index only and we saw the same error like index shard got corrupted
If you want I can do the same again and show you
On Sat, Mar 7, 2015 at 8:26 AM, Igor Motov notifications@github.com wrote:
@sukantasaha https://github.com/sukantasaha what did you mean by "may be we migrated this index"? Could it be possible that this index was originally created with older version of elasticsearch? I am just wondering if you are hitting elasticsearch/elasticsearch#9140 https://github.com/elasticsearch/elasticsearch/issues/9140. Would it be possible for you as an experiment to restore this index using a newer version of elasticsearch (v1.4.3 or higher)?
— Reply to this email directly or view it on GitHub https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/186#issuecomment-77669835 .
May I know that when S3 plugin has been extended by MD5 checksum and in which version , so that no more problems with snapshot restore because we get to know if snapshot was really successful or not
MD5 could be added in 2.4.2, so it's not there yet.
When can we expect ?
No ETA yet. And the PR has not been merged. Which means that if you want to build it yourself you need to checkout branch es-1.4, back port the PR in and build.
Is this fix now in es-1.5?
No. It's not.
Hi
in all our elasticsearch cluster we use this elasticsearch-cloud-aws plugin to create the snapshots on s3 on a regular basis.
Some times we saw the shard got corrupted for an index in our elasticsearch log. So we try to restore it from backup and while restoring it from backup again we see the same exception in logs which is follows
Even if we back to an older snapshot we found the same exception.
So what we did was we download all the segments files from s3 merge it and there we found some segments were corrupted by using
org.apache.lucene.index.CheckIndex
with-fix
We fixed it but we loose 5gb data.We shared this problem with aws team, here what they suggested:
Can you guys please have a look into this issue and suggest something
Thanks