bug Python for processing (SOLWEIG)

[simonmarti1992]

Dear all,

I'm facing a bug while trying to execute the example

Description Here is the error code found in the console

"C:\Program Files\QGIS 3.28.0\apps\Python39\python.exe" C:\Users\simon\python-scripts\tipee-umep\main.py 
C:\Users\simon\AppData\Roaming\Python\Python39\site-packages\pandas\core\reshape\merge.py:916: FutureWarning: In a future version, the Index constructor will not infer numeric dtypes when passed object-dtype sequences (matching Series behavior)
  key_col = Index(lvals).where(~mask_left, rvals)

Process finished with exit code -1073741819 (0xC0000005)

To Reproduce

# Export sys path to user home (you can specify an other directory is you like)
import json
import os
import sys

import pandas as pd

sys.path.extend([r'C:\Program Files\QGIS 3.28.0\apps\qgis\python\plugins/',
                 r'C:\Program Files\QGIS 3.28.0\apps\qgis\python/',
                 r'C:\Users\simon\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins/'])

# For Windows users
user_home = os.path.expanduser("~")
third_party_path = os.path.join(user_home, "\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins")

export_directory = os.path.expanduser("~")
paths = sys.path
df = pd.DataFrame({'paths':paths})
df.to_csv(os.path.join(export_directory,'.qgis_sys_paths.csv'), index=False)

# Export environment variables
env = dict(os.environ)
rem = ['SECURITYSESSIONID', 'LaunchInstanceID', 'TMPDIR']
_ = [env.pop(r, None) for r in rem]
with open(os.path.join(export_directory,'.qgis_env.json'), 'w') as f:
    json.dump(env, f, ensure_ascii=False, indent=4)

# Defines the directory where have been exported the path and variables
export_directory = os.path.expanduser("~")

# Set up system paths
qspath = os.path.join(export_directory, '.qgis_sys_paths.csv')
paths = pd.read_csv(qspath).paths.tolist()
sys.path += paths

# Set up environment variables
qepath = os.path.join(export_directory, '.qgis_env.json')

with open(qepath, 'r') as file:
    js = json.loads(file.read())
for k, v in js.items():
    os.environ[k] = v

from qgis import processing

from processing.core.Processing import Processing
from qgis.core import QgsApplication
from processing_umep.processing_umep_provider import ProcessingUMEPProvider

Processing.initialize()

umep_provider = ProcessingUMEPProvider()

QgsApplication.processingRegistry().addProvider(umep_provider)

#===================================================================
#                            
#===================================================================

# Input files definition
input_directory = "C:/Users/simon\python-scripts/tipee-umep/Goteborg_SWEREF99_1200/"
input_mask = "mask_layer.geojson"
input_cdsm = 'CDSM_KRbig.asc'
input_dsm = 'DSM_KRbig.tif'
input_dem = 'DEM_KRbig.tif'
input_landcover = 'landcover.tif'
input_meteo = 'gbg19970606_2015a.txt'

# Defines an output directory where will be stored your outputs (and intermediate results)
output_dir = "C:/Users/simon/python-scripts/tipee-umep/results/"

# Set the EPSG code for the .asc file which has no EPSG
from qgis.core import QgsCoordinateReferenceSystem
cdsm_epsg = QgsCoordinateReferenceSystem('EPSG:3007')
input_cdsm_filename = input_cdsm.split(".")[0]
crop_cdsm = processing.run("gdal:cliprasterbymasklayer",
                           {'INPUT': os.path.join(input_directory, input_cdsm),
                            'MASK': os.path.join(input_directory, input_mask),
                            'SOURCE_CRS':cdsm_epsg,
                            'TARGET_CRS':cdsm_epsg,
                            'NODATA':None,'ALPHA_BAND':False,'CROP_TO_CUTLINE':True,
                            'KEEP_RESOLUTION':True,'SET_RESOLUTION':False,'X_RESOLUTION':None,
                            'Y_RESOLUTION':None,'MULTITHREADING':False,'OPTIONS':'',
                            'DATA_TYPE':0,'EXTRA':'','OUTPUT': os.path.join(output_dir,
                                                                            "Crop_" + \
                                                                            input_cdsm_filename + ".tif")})
input_dsm_filename = input_dsm.split(".")[0]
crop_dsm = processing.run("gdal:cliprasterbymasklayer",
                           {'INPUT': os.path.join(input_directory, input_dsm),
                            'MASK': os.path.join(input_directory, input_mask),
                            'SOURCE_CRS':None,
                            'TARGET_CRS':None,
                            'NODATA':None,'ALPHA_BAND':False,'CROP_TO_CUTLINE':True,
                            'KEEP_RESOLUTION':True,'SET_RESOLUTION':False,'X_RESOLUTION':None,
                            'Y_RESOLUTION':None,'MULTITHREADING':False,'OPTIONS':'',
                            'DATA_TYPE':0,'EXTRA':'','OUTPUT': os.path.join(output_dir,
                                                                            "Crop_" + \
                                                                            input_dsm_filename + ".tif")})
input_dem_filename = input_dem.split(".")[0]
crop_dem = processing.run("gdal:cliprasterbymasklayer",
                           {'INPUT': os.path.join(input_directory, input_dem),
                            'MASK': os.path.join(input_directory, input_mask),
                            'SOURCE_CRS':None,
                            'TARGET_CRS':None,
                            'NODATA':None,'ALPHA_BAND':False,'CROP_TO_CUTLINE':True,
                            'KEEP_RESOLUTION':True,'SET_RESOLUTION':False,'X_RESOLUTION':None,
                            'Y_RESOLUTION':None,'MULTITHREADING':False,'OPTIONS':'',
                            'DATA_TYPE':0,'EXTRA':'','OUTPUT': os.path.join(output_dir,
                                                                            "Crop_" + \
                                                                            input_dem_filename + ".tif")})
input_landcover_filename = input_landcover.split(".")[0]
crop_landcover = processing.run("gdal:cliprasterbymasklayer",
                           {'INPUT': os.path.join(input_directory, input_landcover),
                            'MASK': os.path.join(input_directory, input_mask),
                            'SOURCE_CRS':None,
                            'TARGET_CRS':None,
                            'NODATA':None,'ALPHA_BAND':False,'CROP_TO_CUTLINE':True,
                            'KEEP_RESOLUTION':True,'SET_RESOLUTION':False,'X_RESOLUTION':None,
                            'Y_RESOLUTION':None,'MULTITHREADING':False,'OPTIONS':'',
                            'DATA_TYPE':0,'EXTRA':'','OUTPUT': os.path.join(output_dir,
                                                                            "Crop_" + \
                                                                            input_landcover_filename + ".tif")})

Desktop (please complete the following information):

• OS: Windows 11
• QGIS 3.28.0
• UMEP 1.7.8
• PyCharm 2022.2.3

[biglimp]

Were you able to follow this tutorial: https://umep-docs.readthedocs.io/projects/tutorial/en/latest/Tutorials/PythonProcessing1.html? It seems that you have some issue with pandas. Maybe @j3r3m1 can help. He wrote this execrise.

[simonmarti1992]

@biglimp thank you for your answer.

Yes I have tried the shadowmapping generation from exemple n°1 with the python console of QGIS (works well). Now I'd like to run the example from pycharm.

I'll check the "jupyter notebook" section of this exemple 👍

Best regards

[j3r3m1]

Description Here is the error code found in the console

"C:\Program Files\QGIS 3.28.0\apps\Python39\python.exe" C:\Users\simon\python-scripts\tipee-umep\main.py 
C:\Users\simon\AppData\Roaming\Python\Python39\site-packages\pandas\core\reshape\merge.py:916: FutureWarning: In a future version, the Index constructor will not infer numeric dtypes when passed object-dtype sequences (matching Series behavior)
  key_col = Index(lvals).where(~mask_left, rvals)

Process finished with exit code -1073741819 (0xC0000005)

Thank you for reporting. Can you give more information about the error message you got ? Do you know where exactly the code stopped ? I only see a warning in the message you mentionned.

Note that the first part of the script you mention has to be executed within the Python console. The second part (starting from "# Defines the directory where have been exported the path and variables") has to be executed in a separate script (outside QGIS).

[simonmarti1992]

thank you @biglimp and @j3r3m1

[j3r3m1]

How did you solve your problem ? Was the problem related to the fact that the 1st part of the script should be executed in a QGIS Python console ?

[simonmarti1992]

I solved the problem by executing the script directly in the QGIS python console. I'll try to generate 100% python code after solving all the issues I face with generating proper databases ;)