WARCFileWriter throws IOException if file already exists

[habernal]

Method createSegment() should create a new segment (file) and not override the existing one; however, this is not the case on S3.

This should be updated

FSDataOutputStream fsStream = (progress == null) ?
                fs.create(path, false) :
                fs.create(path, progress);

as fs.create(path, false) sometimes throws

Error: java.io.IOException: File already exists:s3://ukp-research-data/c4corpus/cc-phase1out-2016-07/part-r-00000.seg-00000.warc.gz at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.create(S3NativeFileSystem.java:634) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:912) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:893) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:790) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.create(EmrFileSystem.java:182) at de.tudarmstadt.ukp.dkpro.c4corpus.hadoop.io.WARCFileWriter.createSegment(WARCFileWriter.java:152) at
...

[habernal]

Hopefully fixed by 4cc9a81 Hard to reproduce on small data, reopen if fails on CommonCrawl in the future.

[habernal]

This should be investigated more in detail. Two scenarios:

• failure is an intended behavior of two co-occurring reducers (work with same data to speed up parallelization)
  • but reducers are normally killed by the container (which is not a failure): Speculation: attempt_1457075559415_0002_r_000001_0 succeeded first!
• it's a bug which occurs only on EMR implementation of FileSytem

[habernal]

Fix confirmed; ran on entire CommonCrawl without a single failure.