Overflow problem

[aishangcengloua]

When I run the command

python3 poisson_model_fit_predict.py \ -data_dir ../data/cycle_length_data \ -model_name generative_poisson_with_skipped_cycles,

without modifying your code, I encounter the issue of exploding loss. Could you please explain why this is happening?

../src/prediction\poisson_with_skipped_cycles_models.py:30: RuntimeWarning: overflow encountered in int_scalars
  max_memory_in_bytes=32*np.power(1024,3) # 32 GiB
../src/prediction\generalized_poisson_with_skipped_cycles_models.py:33: RuntimeWarning: overflow encountered in int_scalars
  max_memory_in_bytes=32*np.power(1024,3) # 32 GiB
        Epoch 0/1000.0 with per-batch average loss=2801.45994388847
        Epoch 10/1000.0 with per-batch average loss=2773.1706068836725
        Epoch 20/1000.0 with per-batch average loss=2765.133135504414
    Model trained after 23 epochs with per-batch average loss=138223.4883536642
Could not create generative_poisson_with_skipped_cycles model with error cannot convert float infinity to integer
FINISHED!

[iurteaga]

Thanks for reaching out.

From the provided output, it is unclear where the "float infinity to integer" error is raised.

Could you rerun it with pdb and provide more detailed information on the issue?

Thanks in advance!

[aishangcengloua]

@iurteaga Thank you very much for your prompt and patient response.

I have discovered that the issue mentioned above was due to an overflow that occurred during the calculation of the maximum memory. The line max_memory_in_bytes = 32 * np.power(1024, 3) # 32 GiB was modified to max_memory_in_bytes = 32.0 * np.power(1024, 3) # 32 GiB to address the overflow problem.

I look forward to your guidance. Additionally, I have learned a lot from the work of your team, and I am extremely grateful to all of you!