AntSimi / py-eddy-tracker

Eddy identification and tracking
https://py-eddy-tracker.readthedocs.io/en/latest/
GNU General Public License v3.0
132 stars 55 forks source link

encountered errors applying CMEMS (zos) data to py-eddy-tracker identification #207

Closed doge-py closed 11 months ago

doge-py commented 1 year ago

Hello, I'm trying to apply adt data from cmems GLOBAL_MULTIYEAR_PHY_001_030 to py-eddy-tracker identification, but encounter error as following:

error:

WARNING 2023-07-25 09:49:47,698 grid.__init__ :
        We assume pixel position of grid is centered for /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
        DEBUG 2023-07-25 09:49:47,698   grid.   load_general_features :
                Load general feature from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
        DEBUG 2023-07-25 09:49:47,704   grid.   bessel_high_filter :
                Run filtering with wavelength of 500 km and order of 3 ...
        DEBUG 2023-07-25 09:49:47,704   grid.   grid :
                Load surf_el from /home/user/mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc
Remain  0:00:00.045110 ETA  2023-07-25 09:49:55.191888 current kernel size : (443, 337) Step : 125/126
        DEBUG 2023-07-25 09:49:55,194   grid.   bessel_high_filter :
                Filtering done
INFO 2023-07-25 09:49:55,195 grid.add_uv :
        Add u/v variable with stencil method
        DEBUG 2023-07-25 09:49:55,195   grid.   compute_stencil :
                Stencil half width apply : 4
        DEBUG 2023-07-25 09:49:55,198   grid.   compute_stencil :
                Stencil half width apply : 4
INFO 2023-07-25 09:49:55,924 grid.eddy_identification :
        We will apply on step a factor to be coherent with grid : 1.000000
/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numpy/lib/function_base.py:4737: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
  arr.partition(
        DEBUG 2023-07-25 09:49:55,927   grid.   eddy_identification :
                Levels from -0.160388 to 0.145720
INFO 2023-07-25 09:49:55,927 eddy_feature.__init__ :
        Start computing iso lines
        DEBUG 2023-07-25 09:49:55,945   eddy_feature.   __init__ :
                X shape : (251,)
        DEBUG 2023-07-25 09:49:55,946   eddy_feature.   __init__ :
                Y shape : (126,)
        DEBUG 2023-07-25 09:49:55,946   eddy_feature.   __init__ :
                Z shape : (251, 126)
INFO 2023-07-25 09:49:55,946 eddy_feature.__init__ :
        Start computing iso lines with 156 levels from -0.162000 to 0.148000 ...
INFO 2023-07-25 09:49:56,153 eddy_feature.__init__ :
        Finish computing iso lines
INFO 2023-07-25 09:49:56,169 eddy_feature.__init__ :
        Repair 79 closed contours and 0 almost closed contours / 1026 contours
        DEBUG 2023-07-25 09:49:56,172   grid.   eddy_identification :
                doing collection 1, contour value -0.1600, 1 paths
        DEBUG 2023-07-25 09:49:56,188   grid.   eddy_identification :
                doing collection 2, contour value -0.1580, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 3, contour value -0.1560, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 4, contour value -0.1540, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 5, contour value -0.1520, 1 paths
        DEBUG 2023-07-25 09:49:56,201   grid.   eddy_identification :
                doing collection 6, contour value -0.1500, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 7, contour value -0.1480, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 8, contour value -0.1460, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 9, contour value -0.1440, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 10, contour value -0.1420, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 11, contour value -0.1400, 1 paths
        DEBUG 2023-07-25 09:49:56,202   grid.   eddy_identification :
                doing collection 12, contour value -0.1380, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 13, contour value -0.1360, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 14, contour value -0.1340, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 26, contour value -0.1100, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 27, contour value -0.1080, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 28, contour value -0.1060, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 29, contour value -0.1040, 1 paths
        DEBUG 2023-07-25 09:49:56,203   grid.   eddy_identification :
                doing collection 30, contour value -0.1020, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 31, contour value -0.1000, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 32, contour value -0.0980, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 33, contour value -0.0960, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 34, contour value -0.0940, 2 paths
        DEBUG 2023-07-25 09:49:56,204   grid.   eddy_identification :
                doing collection 35, contour value -0.0920, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 36, contour value -0.0900, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 37, contour value -0.0880, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 38, contour value -0.0860, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 39, contour value -0.0840, 2 paths
        DEBUG 2023-07-25 09:49:56,205   grid.   eddy_identification :
                doing collection 40, contour value -0.0820, 2 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 41, contour value -0.0800, 3 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 42, contour value -0.0780, 2 paths
        DEBUG 2023-07-25 09:49:56,206   grid.   eddy_identification :
                doing collection 43, contour value -0.0760, 2 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 44, contour value -0.0740, 2 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 45, contour value -0.0720, 3 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 46, contour value -0.0700, 3 paths
        DEBUG 2023-07-25 09:49:56,207   grid.   eddy_identification :
                doing collection 47, contour value -0.0680, 6 paths
        DEBUG 2023-07-25 09:49:56,208   grid.   eddy_identification :
                doing collection 48, contour value -0.0660, 9 paths
        DEBUG 2023-07-25 09:49:56,208   grid.   eddy_identification :
                doing collection 49, contour value -0.0640, 9 paths
        DEBUG 2023-07-25 09:49:56,209   grid.   eddy_identification :
                doing collection 50, contour value -0.0620, 10 paths
        DEBUG 2023-07-25 09:49:56,210   grid.   eddy_identification :
                doing collection 51, contour value -0.0600, 9 paths
        DEBUG 2023-07-25 09:49:56,210   grid.   eddy_identification :
                doing collection 52, contour value -0.0580, 12 paths
        DEBUG 2023-07-25 09:49:56,211   grid.   eddy_identification :
                doing collection 53, contour value -0.0560, 11 paths
        DEBUG 2023-07-25 09:49:56,212   grid.   eddy_identification :
                doing collection 54, contour value -0.0540, 11 paths
        DEBUG 2023-07-25 09:49:56,212   grid.   eddy_identification :
                doing collection 55, contour value -0.0520, 9 paths
        DEBUG 2023-07-25 09:49:56,213   grid.   eddy_identification :
                doing collection 56, contour value -0.0500, 11 paths
        DEBUG 2023-07-25 09:49:56,214   grid.   eddy_identification :
                doing collection 57, contour value -0.0480, 11 paths
        DEBUG 2023-07-25 09:49:56,215   grid.   eddy_identification :
                doing collection 58, contour value -0.0460, 15 paths
        DEBUG 2023-07-25 09:49:56,216   grid.   eddy_identification :
                doing collection 59, contour value -0.0440, 16 paths
        DEBUG 2023-07-25 09:49:56,217   grid.   eddy_identification :
                doing collection 60, contour value -0.0420, 21 paths
        DEBUG 2023-07-25 09:49:56,218   grid.   eddy_identification :
                doing collection 61, contour value -0.0400, 21 paths
        DEBUG 2023-07-25 09:49:56,220   grid.   eddy_identification :
                doing collection 62, contour value -0.0380, 19 paths
        DEBUG 2023-07-25 09:49:56,221   grid.   eddy_identification :
                doing collection 63, contour value -0.0360, 22 paths
        DEBUG 2023-07-25 09:49:56,223   grid.   eddy_identification :
                doing collection 64, contour value -0.0340, 18 paths
        DEBUG 2023-07-25 09:49:56,225   grid.   eddy_identification :
                doing collection 65, contour value -0.0320, 17 paths
        DEBUG 2023-07-25 09:49:56,226   grid.   eddy_identification :
                doing collection 66, contour value -0.0300, 18 paths
        DEBUG 2023-07-25 09:49:56,228   grid.   eddy_identification :
                doing collection 67, contour value -0.0280, 14 paths
        DEBUG 2023-07-25 09:49:56,230   grid.   eddy_identification :
                doing collection 68, contour value -0.0260, 13 paths
        DEBUG 2023-07-25 09:49:56,240   grid.   eddy_identification :
                doing collection 69, contour value -0.0240, 11 paths
        DEBUG 2023-07-25 09:49:56,241   grid.   eddy_identification :
                doing collection 70, contour value -0.0220, 12 paths
        DEBUG 2023-07-25 09:49:56,242   grid.   eddy_identification :
                doing collection 71, contour value -0.0200, 10 paths
        DEBUG 2023-07-25 09:49:56,243   grid.   eddy_identification :
                doing collection 72, contour value -0.0180, 8 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 73, contour value -0.0160, 5 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 74, contour value -0.0140, 5 paths
        DEBUG 2023-07-25 09:49:56,244   grid.   eddy_identification :
                doing collection 75, contour value -0.0120, 3 paths
        DEBUG 2023-07-25 09:49:56,245   grid.   eddy_identification :
                doing collection 76, contour value -0.0100, 5 paths
        DEBUG 2023-07-25 09:49:56,245   grid.   eddy_identification :
                doing collection 77, contour value -0.0080, 6 paths

Traceback (most recent call last):
  File "/home/user/eddy_identification.py", line 19, in <module>
    a, c = h.eddy_identification(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 849, in eddy_identification
    ) = self.get_uavg(
        ^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 996, in get_uavg
    max_average_speed = self.speed_coef_mean(original_contour)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 1911, in speed_coef_mean
    return mean_on_regular_contour(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/home/user/mambaforge/envs/pet/lib/python3.11/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<built-in function getitem>) found for signature:

 >>> getitem(bool, UniTuple(int64 x 2))

There are 22 candidate implementations:
      - Of which 22 did not match due to:
      Overload of function 'getitem': File: <numerous>: Line N/A.
        With argument(s): '(bool, UniTuple(int64 x 2))':
       No match.

During: typing of intrinsic-call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (280)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 280:
def interp2d_bilinear(x_g, y_g, z_g, m_g, x, y):
    <source elided>
            if not masked:
                if m_g[i0, j0] or m_g[i0, j1] or m_g[i1, j0] or m_g[i1, j1]:
                ^

During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)

During: resolving callee type: type(CPUDispatcher(<function interp2d_bilinear at 0x7efed0ffdc60>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py (201)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/generic.py", line 201:
def interp2d_geo(x_g, y_g, z_g, m_g, x, y, nearest=False):
    <source elided>
    else:
        return interp2d_bilinear(x_g, y_g, z_g, m_g, x, y)
        ^

During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)

During: resolving callee type: type(CPUDispatcher(<function interp2d_geo at 0x7efed0ffd620>))
During: typing of call at /home/user/mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py (130)

File "../../../mambaforge/envs/pet/lib/python3.11/site-packages/pyEddyTracker-3.6.1+9.g6597187-py3.11.egg/py_eddy_tracker/dataset/grid.py", line 130:
def mean_on_regular_contour(
    <source elided>
    x_new, y_new = uniform_resample(x_val, y_val, num_fac, fixed_size)
    values = interp2d_geo(x_g, y_g, z_g, m_g, x_new[1:], y_new[1:])

my code:

#Activate verbose

from py_eddy_tracker import start_logger

start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification

from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset

grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"

h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
    "zos",
    "ugos",
    "vgos",  # Variables used for identification
    date,  # Date of identification
    0.002,  # step between two isolines of detection (m)
    pixel_limit=(5, 2000),  # Min and max pixel count for valid contour
    shape_error=55,  # Error max (%) between ratio of circle fit and contour
)
#Save identification data

from netCDF import Dataset

with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
    a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
    c.to_netcdf(h)

other information:

numpy version : 1.24.4 numba version : 0.57.1 python version : 3.11.3

Best regards,

doge

doge-py commented 1 year ago

after changing acess data method from MOTU to ftp, got the same errors from here Unit error block EddyId #2.

July 27, 2023 after some research, problem solved when I add force_speed_unit="m/s" to eddy_identification, but I still want to know how to apply data obtained from MOTU because it seems that ftp can't subset data variables? (not quite sure about it)

new code:

#Activate verbose

from py_eddy_tracker import start_logger

start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG
#Run identification

from datetime import datetime
from py_eddy_tracker.dataset.grid import RegularGridDataset

grid_name = "mercatorglorys12v1_gl12_mean_20110101_R20110105_subset.nc"
lon_name = "longitude"
lat_name = "latitude"

h = RegularGridDataset(grid_name, lon_name, lat_name)
h.bessel_high_filter("zos", 500, order=3)
date = datetime(2011, 1, 1)
h.add_uv("zos","ugos","vgos")
a, c = h.eddy_identification(
    "zos",
    "ugos",
    "vgos",  # Variables used for identification
    date,  # Date of identification
    0.002,  # step between two isolines of detection (m)
    pixel_limit=(5, 2000),  # Min and max pixel count for valid contour
    shape_error=55,  # Error max (%) between ratio of circle fit and contour
    force_speed_unit="m/s"
)
#Save identification data

from netCDF import Dataset

with Dataset(date.strftime("Anticyclonic_%Y%m%d.nc"), "w") as h:
    a.to_netcdf(h)
with Dataset(date.strftime("Cyclonic_%Y%m%d.nc"), "w") as h:
    c.to_netcdf(h)

Best regards,

doge

AntSimi commented 1 year ago

MOTU do some modifications from original file, last time i check this tools create a mix of nan and masked value. Maybe you could try to use 'nan_masking' options https://github.com/AntSimi/py-eddy-tracker/issues/111#issuecomment-947071363

doge-py commented 1 year ago

I followed your suggestions and add "nan_masking=True" to the code, but unfortunately, the issue still persists and got exactly same error message. I'm thankful for your assistance.

AntSimi commented 1 year ago

could you share your file in order to reproduce bug?

doge-py commented 1 year ago

I subsetted it as following: time: 2020,12,31 ~ 2020,12,31 lon: 67 ~ 105 lat: -37 ~ -29 elevation: -0.49 ~ -0.49 variable: uo, vo, zos

AntSimi commented 1 year ago

with latest version you must avoid this exception with& without nan_masking option

doge-py commented 11 months ago

It worked; I really appreciate your help!