ARC parser infinite loop reading body

[sebastian-nagel]

On certain ARC files the parser may run into an infinite loop. So far, I've found the following ARC files which reproducibly cause the hang-up when running the "validate" tool:

• IAH-20080430204825-00000-blackbook-truncated.arc - part of ukwa/webarchive-test-suite and also used by jwat as test resource. Note: when parsing the gzipped variant (also part of the test suite) the parser complains about an "invalid ARC trailer". The stack during the hang-up:

  "main" #1 prio=5 os_prio=0 tid=0x00007f0e8400b800 nid=0x38450 runnable [0x00007f0e8908a000]
  java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.NativeThread.current(Native Method)
      at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
      at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:155)
      - locked <0x000000071ab05d30> (a java.lang.Object)
      at org.netpreserve.jwarc.LengthedBody.read(LengthedBody.java:76)
      - locked <0x000000071ac21f30> (a org.netpreserve.jwarc.LengthedBody$Seekable)
      at org.netpreserve.jwarc.LengthedBody$Seekable$1.read(LengthedBody.java:236)
      at org.netpreserve.jwarc.LengthedBody.read(LengthedBody.java:76)
      - locked <0x000000071ac28738> (a org.netpreserve.jwarc.LengthedBody$Seekable)
      at org.netpreserve.jwarc.tools.ValidateTool.readBody(ValidateTool.java:83)
      at org.netpreserve.jwarc.tools.ValidateTool.validateCapture(ValidateTool.java:159)
      at org.netpreserve.jwarc.tools.ValidateTool.validate(ValidateTool.java:182)
      at org.netpreserve.jwarc.tools.ValidateTool.main(ValidateTool.java:283)
      at org.netpreserve.jwarc.tools.WarcTool.main(WarcTool.java:49)

• the gzipped ARC 1266352769711_14.arc.gz (Common Crawl 2010):

  "main" #1 prio=5 os_prio=0 tid=0x00007f414400b800 nid=0x38c12 runnable [0x00007f414a6b8000]
  java.lang.Thread.State: RUNNABLE
      at java.util.zip.Inflater.inflateBytes(Native Method)
      at java.util.zip.Inflater.inflate(Inflater.java:259)
      - locked <0x000000071ab5fe10> (a java.util.zip.ZStreamRef)
      at org.netpreserve.jwarc.GunzipChannel.read(GunzipChannel.java:59)
      at org.netpreserve.jwarc.LengthedBody.read(LengthedBody.java:76)
      - locked <0x000000071abb4330> (a org.netpreserve.jwarc.LengthedBody)
      at org.netpreserve.jwarc.LengthedBody$1.read(LengthedBody.java:138)
      at org.netpreserve.jwarc.LengthedBody.read(LengthedBody.java:76)
      - locked <0x000000071abf55d8> (a org.netpreserve.jwarc.LengthedBody)
      at org.netpreserve.jwarc.tools.ValidateTool.readBody(ValidateTool.java:83)
      at org.netpreserve.jwarc.tools.ValidateTool.validateCapture(ValidateTool.java:159)
      at org.netpreserve.jwarc.tools.ValidateTool.validate(ValidateTool.java:182)
      at org.netpreserve.jwarc.tools.ValidateTool.main(ValidateTool.java:283)
      at org.netpreserve.jwarc.tools.WarcTool.main(WarcTool.java:49)

[ato]

Fix for the infinite loop released as 0.16.5.

The invalid ARC trailer warning on 1266352769711_14.arc.gz I think is correct. The length field for the filedesc:// record seems to be off by one, whereas the subsequent records it's correct. Although I suppose an argument could be made that the definition of the length field in the version block in the spec as "the rest of the version block" is vague and could possibly be interpreted to include the first newline or the trailer newline even though this is inconsistent the response records where it's clearly defined to exclude these.

The ARCs in our collection both those generated by Heritrix and others supplied by IA are all consistent with jwarc's definition of the length.