AntSimi / py-eddy-tracker

Eddy identification and tracking
https://py-eddy-tracker.readthedocs.io/en/latest/
GNU General Public License v3.0
120 stars 48 forks source link

encountered errors applying CMEMS (zos) data to py-eddy-tracker identification #207

Closed doge-py closed 7 months ago

doge-py commented 12 months ago

Hello, I'm trying to apply adt data from cmems GLOBAL_MULTIYEAR_PHY_001_030 to py-eddy-tracker identification, but encounter error as following:

error:

WARNING 2023-07-25 09:49:47,698 grid.__init__ :
        We assume pixel position of grid is centered for /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
        DEBUG 2023-07-25 09:49:47,698   grid.   load_general_features :
                Load general feature from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
        DEBUG 2023-07-25 09:49:47,704   grid.   bessel_high_filter :
                Run filtering with wavelength of 500 km and order of 3 ...
        DEBUG 2023-07-25 09:49:47,704   grid.   grid :
                Load surf_el from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
Remain  0:00:00.045110 ETA  2023-07-25 09:49:55.191888 current kernel size : (443, 337) Step : 125/126
        DEBUG 2023-07-25 09:49:55,194   grid.   bessel_high_filter :
                Filtering done
INFO 2023-07-25 09:49:55,195 grid.add_uv :
        Add u/v variable with stencil method
        DEBUG 2023-07-25 09:49:55,195   grid.   compute_stencil :
                Stencil half width apply : 4
        DEBUG 2023-07-25 09:49:55,198   grid.   compute_stencil :
                Stencil half width apply : 4
INFO 2023-07-25 09:49:55,924 grid.eddy_identification :
        We will apply on step a factor to be coherent with grid : 1.000000
/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numpy/lib/function_base.py:4737: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
  arr.partition(
        DEBUG 2023-07-25 09:49:55,927   grid.   eddy_identification :
                Levels from -0.160388 to 0.145720
INFO 2023-07-25 09:49:55,927 eddy_feature.__init__ :
        Start computing iso lines
        DEBUG 2023-07-25 09:49:55,945   eddy_feature.   __init__ :
                X shape : (251,)
        DEBUG 2023-07-25 09:49:55,946   eddy_feature.   __init__ :
                Y shape : (126,)
        DEBUG 2023-07-25 09:49:55,946   eddy_feature.   __init__ :
                Z shape : (251, 126)
INFO 2023-07-25 09:49:55,946 eddy_feature.__init__ :
        Start computing iso lines with 156 levels from -0.162000 to 0.148000 ...
INFO 2023-07-25 09:49:56,153 eddy_feature.__init__ :
        Finish computing iso lines
INFO 2023-07-25 09:49:56,169 eddy_feature.__init__ :
        Repair 79 closed contours and 0 almost closed contours / 1026 contours
        DEBUG 2023-07-25 09:49:56,172   grid.   eddy_identification :
                doing collection 1, contour value -0.1600, 1 paths
        DEBUG 2023-07-25 09:49:56,188   grid.   eddy_identification :
                doing collection 2, contour value -0.1580, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 3, contour value -0.1560, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 4, contour value -0.1540, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 5, contour value -0.1520, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 6, contour value -0.1500, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 7, contour value -0.1480, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 8, contour value -0.1460, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 9, contour value -0.1440, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 10, contour value -0.1420, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 11, contour value -0.1400, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 12, contour value -0.1380, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 13, contour value -0.1360, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 14, contour value -0.1340, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 26, contour value -0.1100, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 27, contour value -0.1080, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 28, contour value -0.1060, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 29, contour value -0.1040, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 30, contour value -0.1020, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 31, contour value -0.1000, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 32, contour value -0.0980, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 33, contour value -0.0960, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 34, contour value -0.0940, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 35, contour value -0.0920, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 36, contour value -0.0900, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 37, contour value -0.0880, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 38, contour value -0.0860, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 39, contour value -0.0840, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 40, contour value -0.0820, 2 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 41, contour value -0.0800, 3 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 42, contour value -0.0780, 2 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 43, contour value -0.0760, 2 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 44, contour value -0.0740, 2 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 45, contour value -0.0720, 3 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 46, contour value -0.0700, 3 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 47, contour value -0.0680, 6 paths
        DEBUG 2023-07-25 09:49:56,208   grid.   eddy_identification :
                doing collection 48, contour value -0.0660, 9 paths
        DEBUG 2023-07-25 09:49:56,208   grid.   eddy_identification :
                doing collection 49, contour value -0.0640, 9 paths
        DEBUG 2023-07-25 09:49:56,209   grid.   eddy_identification :
                doing collection 50, contour value -0.0620, 10 paths
        DEBUG 2023-07-25 09:49:56,210   grid.   eddy_identification :
                doing collection 51, contour value -0.0600, 9 paths
        DEBUG 2023-07-25 09:49:56,210   grid.   eddy_identification :
                doing collection 52, contour value -0.0580, 12 paths
        DEBUG 2023-07-25 09:49:56,211   grid.   eddy_identification :
                doing collection 53, contour value -0.0560, 11 paths
        DEBUG 2023-07-25 09:49:56,212   grid.   eddy_identification :
                doing collection 54, contour value -0.0540, 11 paths
        DEBUG 2023-07-25 09:49:56,212   grid.   eddy_identification :
                doing collection 55, contour value -0.0520, 9 paths
        DEBUG 2023-07-25 09:49:56,213   grid.   eddy_identification :
                doing collection 56, contour value -0.0500, 11 paths
        DEBUG 2023-07-25 09:49:56,214   grid.   eddy_identification :
                doing collection 57, contour value -0.0480, 11 paths
        DEBUG 2023-07-25 09:49:56,215   grid.   eddy_identification :
                doing collection 58, contour value -0.0460, 15 paths
        DEBUG 2023-07-25 09:49:56,216   grid.   eddy_identification :
                doing collection 59, contour value -0.0440, 16 paths
        DEBUG 2023-07-25 09:49:56,217   grid.   eddy_identification :
                doing collection 60, contour value -0.0420, 21 paths
        DEBUG 2023-07-25 09:49:56,218   grid.   eddy_identification :
                doing collection 61, contour value -0.0400, 21 paths
        DEBUG 2023-07-25 09:49:56,220   grid.   eddy_identification :
                doing collection 62, contour value -0.0380, 19 paths
        DEBUG 2023-07-25 09:49:56,221   grid.   eddy_identification :
                doing collection 63, contour value -0.0360, 22 paths
        DEBUG 2023-07-25 09:49:56,223   grid.   eddy_identification :
                doing collection 64, contour value -0.0340, 18 paths
        DEBUG 2023-07-25 09:49:56,225   grid.   eddy_identification :
                doing collection 65, contour value -0.0320, 17 paths
        DEBUG 2023-07-25 09:49:56,226   grid.   eddy_identification :
                doing collection 66, contour value -0.0300, 18 paths
        DEBUG 2023-07-25 09:49:56,228   grid.   eddy_identification :
                doing collection 67, contour value -0.0280, 14 paths
        DEBUG 2023-07-25 09:49:56,230   grid.   eddy_identification :
                doing collection 68, contour value -0.0260, 13 paths
        DEBUG 2023-07-25 09:49:56,240   grid.   eddy_identification :
                doing collection 69, contour value -0.0240, 11 paths
        DEBUG 2023-07-25 09:49:56,241   grid.   eddy_identification :
                doing collection 70, contour value -0.0220, 12 paths
        DEBUG 2023-07-25 09:49:56,242   grid.   eddy_identification :
                doing collection 71, contour value -0.0200, 10 paths
        DEBUG 2023-07-25 09:49:56,243   grid.   eddy_identification :
                doing collection 72, contour value -0.0180, 8 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 73, contour value -0.0160, 5 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 74, contour value -0.0140, 5 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 75, contour value -0.0120, 3 paths
        DEBUG 2023-07-25 09:49:56,245   grid.   eddy_identification :
                doing collection 76, contour value -0.0100, 5 paths
        DEBUG 2023-07-25 09:49:56,245   grid.   eddy_identification :
                doing collection 77, contour value -0.0080, 6 paths

Traceback (most recent call last):
  File "/home/user/eddy_identification.py", line 19, in <module>
    a, c = h.eddy_identification(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 849, in eddy_identification
    ) = self.get_uavg(
        ^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 996, in get_uavg
    max_average_speed = self.speed_coef_mean(original_contour)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 1911, in speed_coef_mean
    return mean_on_regular_contour(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<built-in function getitem>) found for signature:

 >>> getitem(bool, UniTuple(int64 x 2))

There are 22 candidate implementations:
      - Of which 22 did not match due to:
      Overload of function 'getitem': File: <numerous>: Line N/A.
        With argument(s): '(bool, UniTuple(int64 x 2))':
       No match.

During: typing of intrinsic-call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (280)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 280:
def interp2d_bilinear(x_g, y_g, z_g, m_g, x, y):
    <source elided>
            if not masked:
                if m_g[i0, j0] or m_g[i0, j1] or m_g[i1, j0] or m_g[i1, j1]:
                ^

During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)

During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 201:
def interp2d_geo(x_g, y_g, z_g, m_g, x, y, nearest=False):
    <source elided>
    else:
        return interp2d_bilinear(x_g, y_g, z_g, m_g, x, y)
        ^

During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)

During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 130:
def mean_on_regular_contour(
    <source elided>
    x_new, y_new = uniform_resample(x_val, y_val, num_fac, fixed_size)
    values = interp2d_geo(x_g, y_g, z_g, m_g, x_new[1:], y_new[1:])

my code:

#Activate verbose

from py_eddy_tracker import start_logger

start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification

from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset

grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"

h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
    "zos",
    "ugos",
    "vgos",  # Variables used for identification
    date,  # Date of identification
    0.002,  # step between two isolines of detection (m)
    pixel_limit=(5, 2000),  # Min and max pixel count for valid contour
    shape_error=55,  # Error max (%) between ratio of circle fit and contour
)
#Save identification data

from netCDF import Dataset

with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
    a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
    c.to_netcdf(h)

other information:

numpy version : 1.24.4 numba version : 0.57.1 python version : 3.11.3

Best regards,

doge

doge-py commented 11 months ago

after changing acess data method from MOTU to ftp, got the same errors from here Unit error block EddyId #2.

July 27, 2023 after some research, problem solved when I add force_speed_unit="m/s" to eddy_identification, but I still want to know how to apply data obtained from MOTU because it seems that ftp can't subset data variables? (not quite sure about it)

new code:

#Activate verbose

from py_eddy_tracker import start_logger

start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification

from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset

grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"

h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
    "zos",
    "ugos",
    "vgos",  # Variables used for identification
    date,  # Date of identification
    0.002,  # step between two isolines of detection (m)
    pixel_limit=(5, 2000),  # Min and max pixel count for valid contour
    shape_error=55,  # Error max (%) between ratio of circle fit and contour
    force_speed_unit="m/s"
)
#Save identification data

from netCDF import Dataset

with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
    a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
    c.to_netcdf(h)

Best regards,

doge

AntSimi commented 11 months ago

MOTU do some modifications from original file, last time i check this tools create a mix of nan and masked value. Maybe you could try to use 'nan_masking' options https://github.com/AntSimi/py-eddy-tracker/issues/111#issuecomment-947071363

doge-py commented 11 months ago

I followed your suggestions and add "nan_masking=True" to the code, but unfortunately, the issue still persists and got exactly same error message. I'm thankful for your assistance.

AntSimi commented 11 months ago

could you share your file in order to reproduce bug?

doge-py commented 11 months ago

I subsetted it as following: time: 2020,12,31 ~ 2020,12,31 lon: 67 ~ 105 lat: -37 ~ -29 elevation: -0.49 ~ -0.49 variable: uo, vo, zos

AntSimi commented 9 months ago

with latest version you must avoid this exception with& without nan_masking option

doge-py commented 7 months ago

It worked; I really appreciate your help!