[Processing] gdal gives no output in standalone script

jfbourdon commented 4 years ago

Describe the bug I'm unable to use any GDAL algorithm in a standalone script (QGIS 3.12.1 on Windows 10). All processing.run(native: ) and processing.run(saga: ) algorithms work without any problem, but when using a processing.run(gdal: ) algorithms, I only get an empty result (no error). However, the same processing(gdal: ) algorithm works when I'm using the python console within QGIS.

Thinking that it may be related to environment variables, I used os.environ inside QGIS' python console to see what variables were loaded, then loaded the exact same ones in my standalone script but it didn't work. I asked for help on SO some time ago got no answer.

How to Reproduce Create a py file, change the inpath and outpath variables and run it with QGIS' python (note that the paths are with an OSGeo4W64 installation at C:/OSGeo4W64):

import os

os.environ['PATH'] = ";".join([
    "C:/OSGeo4W64/apps/qgis/bin",
    "C:/OSGeo4W64/apps/Python37",
    "C:/OSGeo4W64/apps/Python37/Scripts",
    "C:/OSGeo4W64/apps/qt5/bin",
    "C:/OSGeo4W64/apps/Python27/Scripts",
    "C:/OSGeo4W64/bin",
    "C:/WINDOWS/system32",
    "C:/WINDOWS",
    "C:/WINDOWS/system32/WBem",
    "C:/OSGeo4W64/apps/msys/bin",
    "C:/OSGeo4W64/apps/Python37/lib/site-packages/pywin32_system32",
    "C:/OSGeo4W64/apps/Python37/lib/site-packages/numpy/.libs",
    "C:/OSGeo4W64/apps/Python37/lib/site-packages/scipy/.libs"
])
os.environ['PROJ_LIB'] = 'C:/OSGeo4W64/share/proj'
os.environ['PYTHONHOME'] = 'C:/OSGeo4W64/apps/Python37'
os.environ['PYTHONPATH'] = 'C:/OSGeo4W64/apps/Python37;C:/OSGeo4W64/apps/Python37/Scripts'
os.environ['GDAL_DATA'] = 'C:/OSGeo4W64/share/gdal'
os.environ['GDAL_DRIVER_PATH'] = 'C:/OSGeo4W64/bin/gdalplugins'
os.environ['GDAL_FILENAME_IS_UTF8'] = 'YES'
os.environ['QGIS_PREFIX_PATH'] = 'C:/OSGeo4W64/apps/qgis'
os.environ['HOME']= 'C:/Users/bouj1a'
os.environ['GEOTIFF_CSV'] = 'C:/OSGeo4W64/share/epsg_csv'
os.environ['JPEGMEM']= '1000000'
os.environ['O4W_QT_BINARIES']= 'C:/OSGeo4W64/apps/Qt5/bin'
os.environ['O4W_QT_HEADERS']= 'C:/OSGeo4W64/apps/Qt5/include'
os.environ['O4W_QT_LIBRARIES']= 'C:/OSGeo4W64/apps/Qt5/lib'
os.environ['O4W_QT_PLUGINS']= 'C:/OSGeo4W64/apps/Qt5/plugins'
os.environ['O4W_QT_PREFIX']= 'C:/OSGeo4W64/apps/Qt5'
os.environ['O4W_QT_TRANSLATIONS']= 'C:/OSGeo4W64/apps/Qt5/translations'
os.environ['OSGEO4W_ROOT']= 'C:/OSGeo4W64'
os.environ['QT_PLUGIN_PATH']= 'C:/OSGeo4W64/apps/qgis/qtplugins;C:/OSGeo4W64/apps/qt5/plugins'
os.environ['VSI_CACHE']= 'TRUE'
os.environ['VSI_CACHE_SIZE']= '1000000'

from qgis.core import *
QgsApplication.setPrefixPath("C:/OSGeo4W64/bin", True)

import processing
from processing.core.Processing import Processing
from qgis.analysis import QgsNativeAlgorithms
from PyQt5.QtCore import *

qgs = QgsApplication([], False)
qgs.initQgis()
Processing.initialize()
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())

inpath = 'E:/raster.tif'
outpath = 'E:/polygon.shp'

params = {
    'INPUT':inpath,
    'BAND':1,
    'FIELD':'VALUE',
    'EIGHT_CONNECTEDNESS':False,
    'OUTPUT':outpath 
}
processing.run("gdal:polygonize", params)

# Result (but no file created)
#> {'OUTPUT': 'E:/polygon.shp'}

QGIS and OS versions QGIS 3.12.2-București QGIS code revision 8a1fb33 Compiled against Qt 5.11.2 Windows 10 (10.0)

gioman commented 4 years ago

@jfbourdon you should ask in the users or developers mailing list and check if your script is ok.

earthcli commented 4 years ago

@jfbourdon Have you solved this problem and could you share your experience ？ Thanks.

jfbourdon commented 4 years ago

I haven't solved this yet... I rather avoided the problem by only running my scripts with the GUI and/or by directly calling GDAL executables with subprocess.run(). I asked on the dev mailing list (message here) but got no answer unfortunately.

github-actions[bot] commented 3 years ago

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

jfbourdon commented 3 years ago

As a follow-up, I'm still unable to call any processing.run(gdal: ) in a standalone script, however since 3.14, we can call directly processing tools, including processing plugins, in CLI with qgis_process which allow me to completely replace my standalone scripts and avoid any issue of the sort. So the original issue persists, but there is now a workaround which is even better.