Error when loading a big journal file

[Suhail]

This happens when trying to read a large journal file for whatever reason:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at net.lag.kestrel.QItem$.unpack(QItem.scala:38) at net.lag.kestrel.Journal.readJournalEntry(Journal.scala:254) at net.lag.kestrel.Journal$$anonfun$fillReadBehind$1.apply(Journal.scala:177) at net.lag.kestrel.Journal$$anonfun$fillReadBehind$1.apply(Journal.scala:171) at scala.Option.foreach(Option.scala:94) at net.lag.kestrel.Journal.fillReadBehind(Journal.scala:171) Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

[robey]

how much memory did you give kestrel with -Xmx? how big is the journal file?

[Suhail]

Journal is massive: 140G and the memory I gave was around 7k MB

[Suhail]

It's odd that it didn't rotate at the default 160MB level set by my config.

[robey]

it should rotate at 16MB if the queue is empty, or at 160MB if there is less than 16MB of stuff in it, but if you always have a lot of stuff in the queue, it won't be able to rotate the journal. was this queue backing up?

[Suhail]

yes it was; basically we lost data due to it....you can't restart kestrel in that instance.

[robey]

you can, if you give it more memory (-Xmx) or lower "max_memory_size" which is the amount of a queue it will try to keep in memory (default 128MB).

[Suhail]

the journal file got to 140G and I only had 8G of memory...still failed even when I raised it.

[robey]

:( try turning down the max_memory_size to keep it from keeping much of the journal in memory. [but it will still take a really long time to read 140GB.]

[Suhail]

It's max_memory_size = 134217728 which is 128 MB =\

[robey]

try making it even smaller if that's a bottleneck. (it will use that much memory per queue.)

[Suhail]

There's only two queues though. I can try though...

[robey]

closing old issue.