Using the iterator obtained from a WARCReader obtained from an InputStream opened on a compressed WARC, using GZIPMembersInputStream, the InputStream is closed by the final call to hasNext(). If an uncompressed WARC is used in a similar context, the stream is not closed. The close is due to the use of SequenceInputStream in OpenJDK7GZIPInputStream (stack trace below).
(This matters to us because we maintain a use counter for each WARC in a pool, which is decremented by the close() method of a wrapper stream. The extra close causes the counter to be decremented twice. We've worked around it by wrapping the stream we pass to GZIPMembersInputStream in an IgnoreCloseInputStream (a FilterInputStream that has an empty close() method).)
We believe that open streams that are passed to library classes should not, in general, be closed by the library, and in any case the behavior should be consistent whether the WARC is compressed or not.
I'm happy to submit a PR, but I'm not certain the close performed by the SequenceInputStream is never needed. I.e., is this path ever involved in processing multiple (effectively concatenated) WARCs? My proposed fix is simply to wrap the stream ('in') passed to SequenceInputStream in an IgnoreCloseInputStream, which is inelegant but does minimal violence to the existing code.
at org.lockss.util.CloseCallbackInputStream.close(CloseCallbackInputStream.java:59)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at java.io.SequenceInputStream.nextStream(SequenceInputStream.java:106)
at java.io.SequenceInputStream.read(SequenceInputStream.java:168)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:59)
at org.archive.util.zip.OpenJDK7GZIPInputStream.readUByte(OpenJDK7GZIPInputStream.java:268)
at org.archive.util.zip.OpenJDK7GZIPInputStream.readUShort(OpenJDK7GZIPInputStream.java:260)
at org.archive.util.zip.OpenJDK7GZIPInputStream.readHeader(OpenJDK7GZIPInputStream.java:169)
at org.archive.util.zip.GZIPMembersInputStream.readTrailer(GZIPMembersInputStream.java:138)
at org.archive.util.zip.OpenJDK7GZIPInputStream.read(OpenJDK7GZIPInputStream.java:124)
at org.archive.util.zip.GZIPMembersInputStream.read(GZIPMembersInputStream.java:113)
at org.archive.util.zip.OpenJDK7InflaterInputStream.read(OpenJDK7InflaterInputStream.java:126)
at org.lockss.laaws.rs.io.storage.warc.WarcArtifactDataStore$CompressedWARCReader.gotoEOR(WarcArtifactDataStore.java:2778)
at org.archive.io.ArchiveReader.cleanupCurrentRecord(ArchiveReader.java:176)
at org.archive.io.ArchiveReader$ArchiveRecordIterator.hasNext(ArchiveReader.java:449)
Using the iterator obtained from a WARCReader obtained from an InputStream opened on a compressed WARC, using GZIPMembersInputStream, the InputStream is closed by the final call to hasNext(). If an uncompressed WARC is used in a similar context, the stream is not closed. The close is due to the use of SequenceInputStream in OpenJDK7GZIPInputStream (stack trace below).
(This matters to us because we maintain a use counter for each WARC in a pool, which is decremented by the close() method of a wrapper stream. The extra close causes the counter to be decremented twice. We've worked around it by wrapping the stream we pass to GZIPMembersInputStream in an IgnoreCloseInputStream (a FilterInputStream that has an empty close() method).)
We believe that open streams that are passed to library classes should not, in general, be closed by the library, and in any case the behavior should be consistent whether the WARC is compressed or not.
I'm happy to submit a PR, but I'm not certain the close performed by the SequenceInputStream is never needed. I.e., is this path ever involved in processing multiple (effectively concatenated) WARCs? My proposed fix is simply to wrap the stream ('in') passed to SequenceInputStream in an IgnoreCloseInputStream, which is inelegant but does minimal violence to the existing code.