Heap space not released until OOM?

[christopherfrieler]

Hello,

I'm experiencing increasing memory consumption of a notebook until I end up with a "java.lang.OutOfMemoryError: Java heap space", althoug I'm just re-execution the only cell the notebook.

Here is my example using Kotlin Dataframe:

import org.jetbrains.kotlinx.dataframe.DataFrame
import org.jetbrains.kotlinx.dataframe.io.readCSV

val data = DataFrame.readCSV("mydata.csv")

The .csv-file is about 260MB big, but I guess that shouldn't matter, just influence how long it takes to be out of memory.

As I re-execute this cell again and again I can see the memory consumption of the kernel's Java process increase until it hits the memory limit I configured and I end up with the OutOfOfMemoryError. While the kernel keeps the content of data for subsequent cells, I would expect it to release any old "version" of that variable when I re-execute the cell. Hence, the memory consumption should not increase with every execution.

Seems like a bug to me. Or am I doing it wrong? Or just misunderstanding how it's supposed to work?

[zakurdaevm]

Hello,

Thank you for providing a clear description. It was easy to understand and reproduce.

In short, if you only re-execute a cell (without a kernel restart), it doesn’t release any old "version" of that variable (and doesn’t release memory allocated to the previous ones).

More specifically: we store instances for all executed snippets and may reference these values even if they are hidden with the variables defined later. That's why we have hard references to all objects that were defined as variables in all the snippets.

And as a result, repeatedly executing this cell will lead to an OutOfMemoryError over time.

Also we already have a bit similar task (for improvement UX in similar cases) in a Kotlin Notebook project in our YouTrack https://youtrack.jetbrains.com/issue/KTNB-553/Show-notification-with-action-in-the-case-of-java.lang.OutOfMemoryError-Java-heap-space (and you can vote for it)

Did I answer your question? Has it become clearer now?

[christopherfrieler]

Thank you for reprocessing and the clarification, at least I know, that I was not doing it wrong. Unfortunately, restarting the kernel is not a good option for me, because this way I'd loose data loaded in earlier cells and would have to execute the entire notebook from the start.

And I still see no reason, why the kernel should keep the data from previous executions, because it would not be accessible anymore once a variable is overridden. Are there plans to change this?

[zakurdaevm]

.... because it would not be accessible anymore once a variable is overridden.

it's not like that. Even in this case you have access to the previous values of a variable

Even in the same cell (after re-executing):

It means basically this:

... we store instances for all executed snippets and may reference these values even if they are hidden with the variables defined later.

And about:

Are there plans to change this?

We don't have such plans, at least at the current moment, but in general this question is quite complicated. For example, imagine that you have code like this:

val x = 3 // first cell
---
fun f() = x * 3 // second cell
---
val x = "str" // third cell
---
f() // What should this cell produce now?

In the current architecture it's clear as long as we store all values. If we override values instead of hiding them, it's not so clear.

But let me ask a question: why do you need to re-execute a cell with a dataset again and again? Could you please describe your use case here?

[christopherfrieler]

Ah, thanks. I didn't know that I can access these varaibles via this@....

I must admit, that don't know, what f() would or should do in your last example. Maybe this should not even compile to prevent me from writing code I don't understand. ;-) Well, seriously, I would have expected all symbols to exist once per kernel and the last cell to write it "wins". But I don't have much experience with jupyter, so if that's the way it is supposed to work, I happily learn something new.

Re-executing cells usually happens while I write my notebook, i.e. I change the code and try it until it works. And when I run into the OOM-Error, I have to restart the kernel and execute the notebook from the beginning. Together with kotlin-dataframe's high memory consumption (https://github.com/Kotlin/dataframe/issues/141), this happens frequently.

[zakurdaevm]

....Well, seriously, I would have expected all symbols to exist once per kernel and the last cell to write it "wins"....

in general it sounds nice, but this question is discussable.

...I happily learn something new.

and thank you for your questions! It let us to get new point of view and very useful for future improvements.

Re-executing cells usually happens while I write my notebook, i.e. I change the code and try it until it works....

This part is quite interesting. As a partially workaround you can try to read file in a separate cell and rest part of code in other cells.

val data = DataFrame.readCSV("mydata.csv")

It should help in case if you don't often change the original dataset (file) But, it wouldn't help if you need to update your original file.

Another option: if you use IntelliJ IDEA (Ultimate Edition) and plugin Kotlin Notebook you can set heap size manually via settings and it also should help a bit (if you have enough memory)

[christopherfrieler]

Thanks for the tips. I already increased the memory limit when iI first encountered the OOM-Error. And splitting the code into smaller cells would help, too. But all this only postpones the problem. With a growing notebook I guess I would always hit the memory limit sooner or later. So some mechanism to release memory should be implemented. If access to values from former executions of cells is a valuable feature (actually, I don't feel the need for that), a way to throw away data explicitly would be helpful, so users can actively decide that they don't need that data anymore.

[zakurdaevm]

thank you for your ideas! I agree that may be we can somehow improve UX/UI here.

I filed an issue in our tracker (Kotlin Notebook) and you can vote for it if you wish.