OSGeo / gdal

GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
https://gdal.org
Other
4.8k stars 2.51k forks source link

gdal2tile.py pool not closed in case of failure #5581

Closed zeevdr closed 2 years ago

zeevdr commented 2 years ago

The pool opened in https://github.com/OSGeo/gdal/blob/95c9d336b2db0c5d240f191638c116c37fd8bb02/swig/python/gdal-utils/osgeo_utils/gdal2tiles.py#L3299

Will not be closed if an exceptio occurs before https://github.com/OSGeo/gdal/blob/95c9d336b2db0c5d240f191638c116c37fd8bb02/swig/python/gdal-utils/osgeo_utils/gdal2tiles.py#L3337

should add try/finally.

Somthing like:

    if pool is not None:
        ...
        created_pool = False
    else:
        ...
        pool = Pool(processes=nb_processes)
        created_pool = True
    try:
        ...
    finally:
        if created_pool:
            pool.terminate()
rouault commented 2 years ago

can you submit a pull request for that ?

zeevdr commented 2 years ago

@rouault will do, once I have the time

rouault commented 2 years ago

@jcphill would you be interested in implementing this ? The above proposal that relies on calling pool.terminate() will require some monkey patching of the MPI pool object to provide a terminate() method

jcphill commented 2 years ago

Sure, although I'll probably refactor to use a with statement rather than try/finally/terminate/close/join.

jcphill commented 2 years ago

@zeevdr I'd appreciate your comments on https://github.com/OSGeo/gdal/pull/5675