python crash using catchment delineation

[rosadav]

Hi! I am usind pyshed to delineate catchment on several points along a river network to have all the subcatchments. It works for the majority of points but in some cases, calculation is stopped without error message, and python crash (exit). I can't find the reason of the crash. Do you have an idea? Thanks!

[mdbartos]

Greetings,

My guess is that it's one of the following:

1) Maximum recursion depth was exceeded 2) Ran out of memory 3) There are cycles in the flow direction raster, meaning that the process will never terminate (and thus cases 1 or 2 will eventually happen).

Make sure that you remove depressions and flats from the dem before creating the flow direction grid (grid.fill_depressions and grid.resolve_flats). You can also check for cycles in the flow direction raster by using grid.detect_cycles.

Let me know if that helps.

Thanks, MDB

[rosadav]

Thanks for your answer! I have check for cycles on the flow direction rester, there are some cycles only on borders. I have also increase the recursion depth, I meet the same crash (only for some outlet points). What can I do find the error and fix it? Can I send you the dem and the points coordinates where python crash, to understand the error? Thanks for your help Rosalie

---

De : Matt Bartos notifications@github.com Envoyé : dimanche 8 mars 2020 18:50 À : mdbartos/pysheds pysheds@noreply.github.com Cc : Vandromme Rosalie r.vandromme@brgm.fr; Author author@noreply.github.com Objet : Re: [mdbartos/pysheds] python crash using catchment delineation (#104)

Greetings,

My guess is that it's one of the following:

1. Maximum recursion depth was exceeded
2. Ran out of memory
3. There are cycles in the flow direction raster, meaning that the process will never terminate (and thus cases 1 or 2 will eventually happen).

Make sure that you remove depressions and flats from the dem before creating the flow direction grid (grid.fill_depressions and grid.resolve_flats). You can also check for cycles in the flow direction raster by using grid.detect_cycles.

Let me know if that helps.

Thanks, MDB

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/mdbartos/pysheds/issues/104?email_source=notifications&email_token=AHW4LE67TD6IP7OFYKLUCW3RGPSHXA5CNFSM4LC2Q6TKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOE4OUQ*issuecomment-596232018__;Iw!!KbSiYrE!1D_x1MX3HOJk3WfmSXX1uMfuHoxzx8Z2XVAP2Q0g36h6zpfHyispC9sBEW3Xlvp7LQ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AHW4LE4H4BL7RHSHHMXPTLLRGPSHXANCNFSM4LC2Q6TA__;!!KbSiYrE!1D_x1MX3HOJk3WfmSXX1uMfuHoxzx8Z2XVAP2Q0g36h6zpfHyispC9sBEW0mlz6xkw$.

[mdbartos]

Sure, I can take a look at it.

[rosadav]

Great, I have sent you the data in personnal mail.

[mdbartos]

Greetings,

I can't seem to replicate the error. I'll post my code.

It seems to work fine, except that a few of the catchments are very small---you may want to use grid.snap_to_mask to make sure that your pour points are located on cells with high accumulation.

Thanks, MDB

[mdbartos]

My code:

import numpy as np
from pysheds.grid import Grid

points = {
    575 : {
        'x' : 582826.1,
        'y' : 6716119.3,
        'i' : 712,
        'j' : 1049
    },
    1775 : {
        'x' : 571304.0,
        'y' : 6715779.5,
        'i' : 251,
        'j' : 1063
    },
    577 : {
        'x' : 579794.0,
        'y' : 6716000.9,
        'i' : 591,
        'j' : 1054
    },
    1410 : {
        'x' : 599127.8,
        'y' : 6716301.1,
        'i' : 1364,
        'j' : 1042
    },
    1657 : {
        'x' : 597718.9,
        'y' : 6717645.3,
        'i' : 1308,
        'j' : 988
    }
}

dirmap = (64, 128, 1, 2, 4, 8, 16, 32)

# Prepare flow directions
grid = Grid.from_raster('/Users/mdbartos/Downloads/DEM_RV.tif', data_name='dem')
grid.fill_depressions('dem', out_name='flooded_dem')
grid.resolve_flats('flooded_dem', out_name='inflated_dem')
grid.clip_to('inflated_dem')
grid.flowdir(data='inflated_dem', out_name='dir', dirmap=dirmap)

# Delineate catchments
catchments = {}
for name, point in points.items():
    catch = grid.catchment(point['x'], point['y'], data='dir', inplace=False, xytype='label')
    catchments[name] = catch

# See number of cells in each delineated catchment
[np.count_nonzero(catchment) for catchment in catchments.values()]

[1, 1180504, 1746966, 1200310, 1]

[rosadav]

Your code also crash for some points on my pc. So the problem is due to my python env, I am working with windows 10.

I have tried your code on a linux virtual machine and it works, there is no crash !

Many thanks !

De : Matt Bartos notifications@github.com Envoyé : mercredi 1 avril 2020 18:40 À : mdbartos/pysheds pysheds@noreply.github.com Cc : Vandromme Rosalie r.vandromme@brgm.fr; Author author@noreply.github.com Objet : Re: [mdbartos/pysheds] python crash using catchment delineation (#104)

My code:

import numpy as np

from pysheds.grid import Grid

points = {

575 : {

    'x' : 582826.1,

    'y' : 6716119.3,

    'i' : 712,

    'j' : 1049

},

1775 : {

    'x' : 571304.0,

    'y' : 6715779.5,

    'i' : 251,

    'j' : 1063

},

577 : {

    'x' : 579794.0,

    'y' : 6716000.9,

    'i' : 591,

    'j' : 1054

},

1410 : {

    'x' : 599127.8,

    'y' : 6716301.1,

    'i' : 1364,

    'j' : 1042

},

1657 : {

    'x' : 597718.9,

    'y' : 6717645.3,

    'i' : 1308,

    'j' : 988

}

}

dirmap = (64, 128, 1, 2, 4, 8, 16, 32)

Prepare flow directions

grid = Grid.from_raster('/Users/mdbartos/Downloads/DEM_RV.tif', data_name='dem')

grid.fill_depressions('dem', out_name='flooded_dem')

grid.resolve_flats('flooded_dem', out_name='inflated_dem')

grid.clip_to('inflated_dem')

grid.flowdir(data='inflated_dem', out_name='dir', dirmap=dirmap)

Delineate catchments

catchments = {}

for name, point in points.items():

catch = grid.catchment(point['x'], point['y'], data='dir', inplace=False, xytype='label')

catchments[name] = catch

See number of cells in each delineated catchment

[np.count_nonzero(catchment) for catchment in catchments.values()]

[1, 1180504, 1746966, 1200310, 1]

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/mdbartos/pysheds/issues/104*issuecomment-607360273__;Iw!!KbSiYrE!zH5TED6hvar21AsR9rEmrZbbOyIla_vGLKd3oj4DcUaJ7tWIQXPu1uY6dG7Zz28c2g$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AHW4LE7N5KRKEJ25UOR44DTRKNU7FANCNFSM4LC2Q6TA__;!!KbSiYrE!zH5TED6hvar21AsR9rEmrZbbOyIla_vGLKd3oj4DcUaJ7tWIQXPu1uY6dG5Z7gQgKw$.

[mdbartos]

Very interesting.

If you have time, would you mind posting your python version, pysheds version, and versions of dependencies (numpy, pandas, scipy, pyproj, scikit-image) on your Windows machine?

If you use pip, you can get these by typing pip freeze into your terminal.

[rosadav]

Hi! Here are the different dependencies versions I have on my Windows machine :
python 3.7.6 pysheds 0.2.6 numpy 1.18.1 pandas 1.0.1 scipy 1.4.1 pyproj 2.5.0 scikit-image 0.16.2

[bmalbusca]

issue closed? @mdbartos

[rosadav]

Yes! Thank you!

[lar84]

Hello, I am running into this same issue. I am on windows 10 as well, but I don't have a Linux machine to use. I have confirmed this issue is not due to lack of memory, recursion limit, or flow direction cycles. Below are the versions/dependencies I'm running. Thanks!

python 3.9.5 pysheds 0.2.7 numpy 1.21.0 pandas 1.2.5 scipy 1.6.3 pyproj 3.1.0 scikit-image 0.18.2

EDIT: This is a recursion issue where Grid.catchment was exceeding the default max recursion limit on Windows OS. I was able to get everything to work on Windows 10 with the above versions/dependencies by running my code in a new thread with a larger stack size and recursion limit.

https://stackoverflow.com/questions/2917210/what-is-the-hard-recursion-limit-for-linux-mac-and-windows

[mdbartos]

Greetings, this should be fixed in pysheds v0.3 due to the use of numba for the recursive code. I'd highly recommend upgrading. Reopen a new issue if the problem persists.