Error - Windows fatal exception: access violation

[rsbragio]

Hi, I am having the following error in PyGnome in a long run simulation (around 100 days):

Windows fatal exception: access violation Main thread: Current thread 0x0000099c (most recent call first): File "C:...\anaconda3\envs\gnome\lib\site-packages\pygnome-1.1.0-py3.8-win-amd64.egg\gnome\movers\movers.py", line 388 in get_move File "C:...\anaconda3\envs\gnome\lib\site-packages\pygnome-1.1.0-py3.8-win-amd64.egg\gnome\model.py", line 916 in move_elements File "C:...\anaconda3\envs\gnome\lib\site-packages\pygnome-1.1.0-py3.8-win-amd64.egg\gnome\model.py", line 1109 in step File "C:...\anaconda3\envs\gnome\lib\site-packages\pygnome-1.1.0-py3.8-win-amd64.egg\gnome\model.py", line 1163 in next File "c:...\run_pygnome.py", line 116 in File "C:...\anaconda3\envs\gnome\lib\site-packages\spyder_kernels\py3compat.py", line 356 in compat_exec File "C:...\anaconda3\envs\gnome\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 482 in exec_code File "C:...\anaconda3\envs\gnome\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 623 in _exec_file File "C:...\anaconda3\envs\gnome\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 532 in runfile File "C:...\AppData\Local\Temp\ipykernel_5352\2737097386.py", line 1 in <cell line: 1> Restarting kernel...

It seems that the error is related to the time step and the duration in the following command that I have used in PyGnome:

model = gs.Model(start_time=start_time, duration=gs.days(30), time_step=gs.minutes(15), map=mymap, uncertain=False, cache_enabled=True)

If I change the duration and time step, the code works well. However, by increasing the duration, it crashes. For example, the following combinations of duration and time step are close to the limit that works well: duration = 20 days; time step = 15 min duration = 30 days; time step = 30 min duration = 100 days; time step = 60 min

I would like to use the default time step of 15 min. However, if I use more than 20 days, the error occurs. I would like to ask if there is some setting that I am missing that could solve this problem.

Thanks

[ChrisBarker-NOAA]

A couple thoughts:

1) I"d disable the cache -- it turns out it's not very useful, and it's certainly not being used in this case:

model = gs.Model(start_time=start_time,
duration=gs.days(30),
time_step=gs.minutes(15),
map=mymap,
uncertain=False,
cache_enabled=False). # False is the default

Secons, from the failure, it looks likeyou are runing this inside / from Spyder.

Try running it on it's own:

python the_name_of_your_acript.py

That will either not have the problem, or should give perhaps a more helpful error.

But Access Violation does not sound good!

If the problem persists, post you entire script.

[rsbragio]

Hi Chris, Thanks very much for the suggestions. I have applied them and the simulation improved slightly. However, I have noticed what the probable mistake was. As I had problems in working in PyGnome with a current file over 2GB, I have concatenated them by depth such that the final current file did not extrapolate this size. To do that, I have used only HYCOM files up to 25m depth. I have used the command below to set the 3D diffusion:

random_mover_3d = gs.RandomMover3D(vertical_diffusion_coef_above_ml=5,vertical_diffusion_coef_below_ml=0.11,mixed_layer_depth=10, horizontal_diffusion_coef_above_ml=100000,horizontal_diffusion_coef_below_ml=126)

with the default values of horizontal and vertical diffusion. I suppose that the problem occurred when some particles go beyond the maximum HYCOM file depth that I used (25m). By setting vertical_diffusion_coef_below_ml=0, the simulation run ok, as all particles restrained from going below ML=10m.

In my simulation, the oil flowed from in depths that goes from the suface to approximately 20m deep. That is why I have used only up to 25m depth in the current files. Hence, I wonder if there is any way in PyGnome to restrain the particle depth or to disregard particles that go beyond a certain depth. The only way I have realized was to set a zero value to the vertical diffusion below ML. But I would like to maintain the ML value as 10m and disregard particles that goes below 25m. I appreciate if there is any suggestion in this issue.

Best regards

[ChrisBarker-NOAA]

Hmm -- what it should do is give you zero velocity below the maximum depth. But the handling of depth does need some attention, so this may be an error in teh code.

@jay-hennen: one more test case to make sure we get right.

BTW: you may. be able to use netcdf files over 2GB if you use netCDF4, rather than netCDF3 (though I'm only guessing that that's the issue here)

[jay-hennen]

movers.py:338 is a call to a CyMover.get_move, so I think this bug is from the RandomMover3D breaking down for some reason. Would explain the AccessViolation too.

[rsbragio]

Thanks,

My first trial was with netCDF4. I used the formats netCDF4 and netCDF4-CLASSIC, but in both, I had the following error:

Traceback (most recent call last): File "C:\Users\Ricardo\anaconda3\envs\gnome\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec exec(code, globals, locals) File "c:...\run_pygnome.py", line 64, in c_mover = GridCurrentMover(netcdf_file) File "C:\Users\Ricardo\anaconda3\envs\gnome\lib\site-packages\pygnome-1.1.0-py3.8-win-amd64.egg\gnome\movers\current_movers.py", line 512, in init self.mover.text_read(filename, topology_file) File "gnome\cy_gnome\cy_gridcurrent_mover.pyx", line 60, in gnome.cy_gnome.cy_gridcurrent_mover.CyGridCurrentMover.text_read OSError: GridCurrentMover_c.TextRead returned an error.

After getting this error, I returned to netCDF3, as I thought that PyGnome did not stand the netCDF4 format files.

[jay-hennen]

GridCurrentMover is not-yet-officially-but-IMO-should-be deprecated. It does not handle depth at all to begin with, and is meant to handle legacy netCDF files in a GNOME specific format. Try the following replacement:

from gnome.movers.py_current_movers import PyCurrentMover
c_mover = PyCurrentMover.from_netCDF(filename=netcdf_file)

No guarantees it will work 100% since 3D support is still a little brittle, but give it a try and we will be happy to help with any problems that arise.

[rsbragio]

Hi, I used the PyCurrentMover commands that you have suggested and it did not return an error message while processing the data. I tested with the HYCOM current files in netCDF4 and netCDF4_CLASSIC formats. Although there was no error message, the current velocities did not seem to be computed in the model, as the oil spill moved only due to the diffusion and was not advected. Also, the simulation ran very slowly. Previously, with GridCurrent Mover, it took around 5 minutes and with PyCurrentMover, I took approximately 12 hours. Maybe I have some wrong issue in the setup. So, I am sending it below. Thanks.

=============================================================

INICIAL SETUP

import gnome.scripting as gs import shutil from pathlib import Path import numpy as np import os from datetime import datetime, timedelta from gnome import scripting as gs from gnome.basic_types import datetime_value_2d from gnome.utilities.remote_data import get_datafile from gnome.model import Model from gnome.maps import MapFromBNA from gnome.environment import Wind, Tide from gnome.environment import GridCurrent from gnome.spill import point_line_release_spill from gnome.movers import RandomMover, WindMover, CatsMover, RandomMover3D from gnome.movers import GridCurrentMover from gnome.outputters import Renderer, NetCDFOutput import netCDF4 as nc from nco import Nco nco = Nco() from netCDF4 import MFDataset from gnome.movers.py_current_movers import PyCurrentMover

=============================================================

DEFINE BASE DIRECTORY

base_dir = Path(file).parent

=============================================================

LOAD MAP

(refloat in hours / no refloat = -1)

mapfile = gs.get_datafile(base_dir / 'CoastBrazil.bna') mymap = gs.MapFromBNA(mapfile, refloat_halflife=-1) #hours

=============================================================

SIMULATION BEGIN AND END TIME

start_time="2019-07-15 00:00" model = gs.Model(start_time=start_time, duration=gs.days(30), time_step=gs.minutes(15), map=mymap, uncertain=False, cache_enabled=False)

=============================================================

MOVERS

current - HYCOM

current_dir = './Currents' netcdf_file = os.path.join(current_dir, 'HYCOMcombined_CLASSIC.nc4') c_mover = PyCurrentMover.from_netCDF(netcdf_file)

c_mover = GridCurrentMover(netcdf_file) #command disabled

model.movers += c_mover

random walk diffusion (vertical)

random_mover_3d = gs.RandomMover3D(vertical_diffusion_coef_above_ml=5,vertical_diffusion_coef_below_ml=0.11,mixed_layer_depth=10, horizontal_diffusion_coef_above_ml=100000,horizontal_diffusion_coef_below_ml=126) #diffusion coefficients in cm2/s, MLD in meters model.movers += random_mover_3d

=============================================================

SPILL

spill = gs.surface_point_line_spill(release_time=start_time, start_position=(-26, -10, 0), num_elements=1000) model.spills += spill

=============================================================

OUTPUTTERS

images_dir = './sp001'

a) IMAGE FILES

renderer = gs.Renderer(output_dir=images_dir, map_filename=mapfile, size = (1280,1024), output_timestep=gs.hours(24)) model.outputters += renderer

b) NETCDF FILE

netcdf_file2 = os.path.join(images_dir, 'sp001.nc') gs.remove_netcdf(netcdf_file2) nc_outputter = gs.NetCDFOutput(netcdf_file2, which_data='standard', output_timestep=gs.hours(24)) model.outputters += nc_outputter

=============================================================

VIEW DATA

x=[] y=[] for step in model: positions = model.get_spill_property('positions') x.append(positions[:,0]) y.append(positions[:,1])

=============================================================

SAVE MODEL

gnome_model = os.path.join(images_dir, 'sp001.gnome')

model.save(gnome_model)

[jay-hennen]

It gave no errors but also took 12 hours? Very strange!

Can you post a ncdump -h of the file you are using? May be some hints in there.

[AmyMacFadyen]

One thing I would check is that the longitude format matches in the map file and the HYCOM currents. For example, if the HYCOM is in 0-->360 convention but the map is in -180-->180 the spill will be "outside" the mover grid.

[rsbragio]

Regarding the map file, it is in the -180 to 180 convention. Although I have used the same convention when I downloaded the HYCOM current files (nc4 format), I read the longitude variables of the downloaded file and they are in the 0 to 360 convention.

I also checked some combinations in the input of the PyGnome case, with the following results:

a) HYCOM in nc3 format up to 2GB (generated from the original nc4 current file), command GridCurrentMover: It worked well and fast with the spill at the surface (vertical diffusion set to zero) . It seems that the longitude in the current file is recognized correctly. To run a subsurface spill up to the maximum depth of the HYCOM files (25m, limited to the 2GB), I set the vertical diffusion to be zero for ML below the maximum depth of the HYCOM current file. It seems to work well in this way. However, the ML was set to a large depth (in the case 25m), which is not adequate as the horizontal diffusion should be smaller at this depth. However, the oil is moving adequately and the longitude of the current is recognized

b) HYCOM in nc3 format (generated from the original nc4 current file), command PyCurrentMover.from_netCDF The subsurface case ran a little slower than the previous case, but the oil did not move. The surface case was fast but it also did not recognize the current

c) HYCOM in nc4 format, command GridCurrentMover returned an error, probably because of the nc4 format

d) HYCOM in nc4 format, command PyCurrentMover.from_netCDF For a surface spill, it was slow and did not recognize the current. For a subsurface spill, which was the slowest case that I have commented in my previous question, it took a lot of time and did not recognize the current.

Is there any way of downloading the HYCOM file in a -180 to 180 format or changing it after downloading (I am not used to altering netCDF files)?

The ncdump -h of the HYCOM current in nc4 format is below.

Thanks

netcdf HYCOMcombined_CLASSIC { dimensions: time = 1361 ; depth = 13 ; lat = 313 ; lon = 157 ; variables: short water_u(time, depth, lat, lon) ; water_u:_FillValue = -30000s ; water_u:long_name = "Eastward Water Velocity" ; water_u:standard_name = "eastward_sea_water_velocity" ; water_u:units = "m/s" ; water_u:NAVO_code = 17 ; water_u:coordinates = "time depth lat lon " ; water_u:add_offset = 0.f ; water_u:scale_factor = 0.001f ; water_u:missing_value = -30000s ; double time(time) ; time:_FillValue = NaN ; time:long_name = "Valid Time" ; time:time_origin = "2000-01-01 00:00:00" ; time:axis = "T" ; time:NAVO_code = 13 ; time:_CoordinateAxisType = "Time" ; time:units = "hours since 2000-01-01" ; time:calendar = "gregorian" ; double depth(depth) ; depth:_FillValue = NaN ; depth:long_name = "Depth" ; depth:standard_name = "depth" ; depth:units = "m" ; depth:positive = "down" ; depth:axis = "Z" ; depth:NAVO_code = 5 ; depth:_CoordinateAxisType = "Height" ; depth:_CoordinateZisPositive = "down" ; double lat(lat) ; lat:_FillValue = NaN ; lat:long_name = "Latitude" ; lat:standard_name = "latitude" ; lat:units = "degrees_north" ; lat:point_spacing = "even" ; lat:axis = "Y" ; lat:NAVO_code = 1 ; lat:_CoordinateAxisType = "Lat" ; double lon(lon) ; lon:_FillValue = NaN ; lon:long_name = "Longitude" ; lon:standard_name = "longitude" ; lon:units = "degrees_east" ; lon:modulo = "360 degrees" ; lon:axis = "X" ; lon:NAVO_code = 2 ; lon:_CoordinateAxisType = "Lon" ; short water_v(time, depth, lat, lon) ; water_v:_FillValue = -30000s ; water_v:long_name = "Northward Water Velocity" ; water_v:standard_name = "northward_sea_water_velocity" ; water_v:units = "m/s" ; water_v:NAVO_code = 18 ; water_v:coordinates = "time depth lat lon " ; water_v:add_offset = 0.f ; water_v:scale_factor = 0.001f ; water_v:missing_value = -30000s ;

// global attributes: :classification_level = "UNCLASSIFIED" ; :distribution_statement = "Approved for public release. Distribution unlimited." ; :downgrade_date = "not applicable" ; :classification_authority = "not applicable" ; :institution = "Fleet Numerical Meteorology and Oceanography Center" ; :source = "HYCOM archive file" ; :history = "archv2ncdf3z" ; :field_type = "instantaneous" ; :Conventions = "CF-1.6 NAVO_netcdf_v1.1" ; :History = "Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)\n", "Original Dataset = GLBy0.08/expt_93.0/uv3z; Translation Date = 2022-05-11T18:09:09.925Z" ; :geospatial_lat_min = -25. ; :geospatial_lat_max = -0.0399999991059303 ; :geospatial_lon_min = -49.0400390625 ; :geospatial_lon_max = -24.0799560546875 ; }

[rsbragio]

The problem of the oil not moving was really due to different formats between the map longitude format (-180 to 180) and the HYCOM current longitude format (0 to 360). This was solved by sutracting 360 degrees from the HYCOM files during their concatenation, with the following Python script:

INITIAL SETUP

import netCDF4 as nc
import numpy as np
import xarray

#OPEN NETCDF FILES
ds = xarray.open_mfdataset('HYCOM_*.nc4',combine = 'by_coords')

#CHANGE LONGITUDE FORMAT
lon = ds['lon'][:]
lonvar = ds.variables['lon']
lonvar = lonvar-360
ds['lon'] = lonvar
print(lonvar)

#SAVE COMBINED FILE 
ds.to_netcdf('HYCOMcombinedStrd2_reduced.nc4',format = \"NETCDF4\")

#CLOSE FILES
ds.close()

#END
print('end')

Thanks very much for your help.