AscatNc: Exception when creating several ascat readers after each other

[awst-baum]

I've run into a weird issue with the AscatNc reader which I've managed to reproduce in a unit test:

Test:

def test_ascat_reader(self):
    folder_name = 'ASCAT/ASCAT_H113'
    for ind in range(3):
        print(ind)
        ascat_data_folder = os.path.join(folder_name, 'data')
        ascat_grid_path = os.path.join(folder_name, 'grid/TUW_WARP5_grid_info_2_3.nc')
        ascat_static_folder = os.path.join(folder_name, 'static_layer')
        reader = AscatNc(path=ascat_data_folder, fn_format="{:04d}", grid_filename=ascat_grid_path, static_layer_path=ascat_static_folder)
        assert reader
        data = reader.read_ts(16.366667, 48.2)
        assert data

Output:

0
1
2
F

Trace:

in test_ascat_reader
    reader = AscatNc(path=ascat_data_folder, fn_format="{:04d}", grid_filename=ascat_grid_path, static_layer_path=ascat_static_folder)
../lib/python3.6/site-packages/ascat/timeseries.py:221: in __init__
    self.slayer = StaticLayers(static_layer_path)
../lib/python3.6/site-packages/ascat/timeseries.py:170: in __init__
    os.path.join(path, 'frozen_snow_probability.nc'))
netCDF4/_netCDF4.pyx:2135: in netCDF4._netCDF4.Dataset.__init__
    ???
netCDF4/_netCDF4.pyx:1752: in netCDF4._netCDF4._ensure_nc_success
    ???
E   OSError: [Errno -101] NetCDF: HDF error: b'ASCAT/ASCAT_H113/static_layer/frozen_snow_probability.nc'

Additional output:

Exception ignored in: <bound method StaticLayers.__del__ of <ascat.timeseries.StaticLayers object at 0x7f580127fc50>>
Traceback (most recent call last):
File "../lib/python3.6/site-packages/ascat/timeseries.py", line 176, in __del__
    self.frozen_snow_prob.close()
AttributeError: 'StaticLayers' object has no attribute 'frozen_snow_prob'

Sounds like closing frozen_snow_prob doesn't work as expected? I don't really understand what's going on...

[sebhahn]

Staticlayers are now loaded into memory once the AscatNc object has been initialized, which takes some time though (~30 sec). This is also better for reading performance. The above mentioned error should no longer be a problem.

[awst-baum]

Thanks for addressing this! One comment: I didn't get that static_layer_path is an 'optional' parameter (which you can set to None) because it's not... optional. Maybe you can give it a default value of 'None' to make that clearer. :)

[sebhahn]

done 1132726

btw, I don't fully understand the use case of creating multiple AscatNc object in a loop. An AscatNc object should created once and than the read method should be called in a loop.

[awst-baum]

Stumbled upon it when I tried to reduce running time of our validations. Wolfgang Preimesberger recommended to restrict the variables read to only the ones used in the current validation by passing something like parameters=['sm'] into the constructor. But the variables that are used can vary - so I have to recreate readers tailored to the current validation. I tried to write a unit test for that behaviour and that's where the problem occurred.

[sebhahn]

Thanks for clarification. Just make sure AscatNc is not re-created in a loop over GPIs.

[awst-baum]

OK. For our code, it's created for each celery job running on a separate node/process. Also, I just realised I don't have to recreate the reader to set the parameters if I do reader.parameters = ['sm']