clj-commons / byte-streams

A Rosetta stone for JVM byte representations
417 stars 33 forks source link

`convert` fails on large pipedinputstream from `byte-transforms/compress` #7

Closed dgrnbrg closed 10 years ago

dgrnbrg commented 10 years ago

I discovered today that when I use byte-transforms/compress with :gzip on a 175MB pre-compressed file (so the compress should mostly just waste time), and then I try to do convert to a seq of 50k byte buffers, I get a StackOverflowException. Here's a stack trace excerpt:

java.lang.StackOverflowError
  java.lang.Class/getSuperclass(Class.java:-2)
  clojure.core$super_chain/invoke(core_deftype.clj:485)
  clojure.core$super_chain/invoke(core_deftype.clj:485)
  clojure.core$super_chain/invoke(core_deftype.clj:485)
  clojure.core$find_protocol_impl/invoke(core_deftype.clj:499)
  clojure.core$satisfies_QMARK_/invoke(core_deftype.clj:531)
  byte_streams$fn__759$fn__772$fn__773/invoke(:270)
  clojure.core.protocols$fn__6034/invoke(protocols.clj:143)
  clojure.core.protocols$fn__6005$G__6000__6014/invoke(protocols.clj:19)
  clojure.core.protocols$seq_reduce/invoke(protocols.clj:31)
  clojure.core.protocols$fn__6026/invoke(protocols.clj:54)
  clojure.core.protocols$fn__5979$G__5974__5992/invoke(protocols.clj:13)
  clojure.core$reduce/invoke(core.clj:6177)
  byte_streams$fn__759$fn__772/invoke(:266)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
  clojure.lang.LazySeq/sval(LazySeq.java:42)
  clojure.lang.LazySeq/seq(LazySeq.java:60)
  clojure.lang.RT/seq(RT.java:484)
  clojure.core$seq/invoke(core.clj:133)
  clojure.core$empty_QMARK_/invoke(core.clj:5595)
  byte_streams$closeable_seq/invoke(:213)
  byte_streams$fn__759$fn__772/invoke(:278)
  byte_streams$convert/invoke(:331)
  byte_streams$eval921$fn__923$fn__925/invoke(:534)
ztellman commented 10 years ago

I tried to reproduce this: https://github.com/ztellman/byte-streams/blob/master/test/byte_streams_test.clj#L83-L90, but failed. Is there something I'm missing?

ztellman commented 10 years ago

I forgot about this, but still don't have a good way to reproduce it. Any ideas?