JoaquinAmatRodrigo / skforecast

Time series forecasting with machine learning models
https://skforecast.org
BSD 3-Clause "New" or "Revised" License
992 stars 113 forks source link

Deep Learning - Fatal Python error: Aborted #730

Closed AVPokrovsky closed 1 week ago

AVPokrovsky commented 1 week ago

Hi, my program executes forecaster.fit(series=data_m) in a loop for ForecasterRnn. After 1908 iterations error appears. It seems kind of Windows fatal error. Listing is attached. The error does not appear for other forecasters, only for Rnn. I guessed it is because of recursion and added sys.setrecursionlimit(10000), but it does not help.
LSTM.txt

JoaquinAmatRodrigo commented 1 week ago

@FernandoCarazoMelo

FernandoCarazoMelo commented 1 week ago

Hi @AVPokrovsky ,

Given that this issue is specific to the RNN forecaster and not others, it might be related to the way RNNs handle memory and state across iterations.

RNNs can be memory-intensive, and repeated training without proper memory management can lead to memory leaks or exhaustion. Since you are using TensorFlow, ensure that you are freeing up memory explicitly if possible

Try it. If that doesn't work, to help better diagnose and resolve the problem effectively, I need a bit more information from you:

Environment Details: Operating System and version Python version TensorFlow version Batch Size

Minimal code snippet that reproduces the error would be extremely helpful. This allows me to understand the context better and try to replicate the issue on my end.

AVPokrovsky commented 1 week ago

Hi, I added in the loop gc.collect() and have an impression that it was resolved, but not sure because tried on different data and different rnn parameters. Anyway, it passed 2661 iterations. More than in first case.

FernandoCarazoMelo commented 1 week ago

That's great. Let us know if you have any other issues with it. @JoaquinAmatRodrigo , we can close this issue.

AVPokrovsky commented 1 week ago

Hi, unfortunately I was wrong. The issue still exists..

Environment Details: Operating System and version Windows 11 Home edition Python version 3.10.4 TensorFlow version 2.15.1 Batch Size 128