fast=True killing the kernel

[kthyng]

When I try to plot with fast=True after a run, something like this,

ocean_drift_sim.plot(linecolor='age_seconds', fast=True)

the kernel dies midway. It has happened in multiple different scenarios with two different implementations of ROMS model output. I dug into the code and tracked it to

https://github.com/OpenDrift/opendrift/blob/15546beb529f0d94629dc971a00b6edcb80dbf2b/opendrift/readers/reader_global_landmask.py#L105-L107

I examined all the inputs and don't see anything weird and can't think of a reason this would cause an issue.

Any idea why this would be happening? Thanks.

[gauteh]

Sounds like you might run out of memory?

[knutfrode]

"Fast" mainly means that a global raster landmask is used, faster than GSHHG polygons (depending on level) but using more memory. However, there s also an option to use landmask from ROMS when plotting: https://opendrift.github.io/gallery/example_model_landmask.html#sphx-glr-gallery-example-model-landmask-py

[gauteh]

By the way, I see you have many similar projects dealing with tracking data. We have recently tried to create a generic tool for drifter and trajectory data (simulated or from experiments) to work with this data in xarray+python, for CF-compliant data, which might be of interest to you: https://github.com/OpenDrift/trajan it doesn't have a lot of functionality yet, but it handles interpolation and gridding which are often first steps in analysing data.

[kthyng]

@gauteh Yes I guess it is often memory when the kernel just dies isn't it? Seems weird because the simulation is small (100 drifters) and short (4 timesteps) and that a landmask would be that much memory for my relatively robust laptop.

@knutfrode I tried using the plotting approach in that example to hide the landmask and the model mask and indeed it didn't crash with fast=True though the plotting also didn't seem to be any faster than without the flag.

@gauteh Thanks for sharing that package with me, I have indeed done quite a bit of Lagrangian modeling over time and I will check it out.

[gauteh]

Loading or using the landmark the first time requires a bit of memory. Maybe it works differently in a script? Hard to know how much memory notebooks use. Maybe the notebook keeps more of the animation in memory as well.

[gauteh]

Did you figure out this one?

[kthyng]

No I've been avoiding using the fast=True input. But, I am hoping to try animations and plotting outside of a notebook today to see what will work for me. I'll report back if I get to it today, otherwise in January.

[kthyng]

@gauteh Sorry for the delay. Finally tried plotting with fast=True in an ipython kernal (not Jupyter notebook) and have a printed message as it aborts (which I haven't looked into yet):

[gauteh]

Looks like there might be some compilation of different libraries. Did you install roaring-landmask via conda or pip?

[kthyng]

pip apparently.

[gauteh]

Maybe it works better with conda, if there's a supported build for your platform (should be).

tor. 25. jan. 2024, 17:15 skrev Kristen Thyng @.***>:

pip apparently.

— Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/1197#issuecomment-1910532141, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN36YDPLDC2D6P3Q3P2KLYQKAKVAVCNFSM6AAAAABALXRHXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQGUZTEMJUGE . You are receiving this because you were mentioned.Message ID: @.***>

[kthyng]

Ah just realized maybe you are talking about the landmask because I used that in the plot command. Please note that any plot command with plot=True will do the same:

[gauteh]

Yes. But the landmask is used in plot if you set fast=True. So, if possible install everything with conda/mamba, because it looks like maybe some C-dependencies of gdal or similar are linked against different versions of a third library.

tor. 25. jan. 2024, 17:32 skrev Kristen Thyng @.***>:

Ah just realized maybe you are talking about the landmask because I used that in the plot command. Please note that any plot command with plot=True will do the same: image.png (view on web) https://github.com/OpenDrift/opendrift/assets/3487237/e88e2132-8e80-46d0-9140-fda6ff0d8add

— Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/1197#issuecomment-1910564102, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN36Z3ADE7M7RUPOCRWRTYQKCIDAVCNFSM6AAAAABALXRHXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQGU3DIMJQGI . You are receiving this because you were mentioned.Message ID: @.***>

[kthyng]

Ah ok got it, I'll report back.

[kthyng]

Same result with roaring-landmask from conda-forge unfortunately.

[gauteh]

Can you post the output from pip list -v and conda list -v ?

[kthyng]

Yes, see attached.

piplist.txt condalist.txt

[kthyng]

Aha! I just went to update opendrift to the new release and realized I had installed it from PyPI for some reason. I have uninstalled the older pip version and installed the newer conda-forge version and now the plot works! For reference, here are my new pip and conda lists. Thank you for your guidance @gauteh.

piplist.txt condalist.txt

[gauteh]

Great! Good to hear!