search

kartoza / GEEST

Gender Enabling Environments Spatial Tool (GEEST)
https://worldbank.github.io/GEEST/
MIT License
1 stars 2 forks source link

Country Test Result #134

Open bennyistanto opened 2 months ago

bennyistanto commented 2 months ago

Fiji, Test 12 Sep 2024

QGIS LTR 3.34.4 and GEEST 0.1.64 (Sent by Michelle via email on 5 Sep 2024) running on Windows 11 Pro

Initial Configuration

QGIS Processing Options: General > Invalid features filtering. Set to > Skip (ignore) features with invalid geometries

Screenshot 2024-09-12 185350

Projection: EPSG:8859

Fiji located in 180th meridian. Using Geographic projection EPSG:4326, it will splitted into different areas, while using UTM also divided into 2 zone. EPSG:8859 might the best option for Fiji as the projection will set the World centred on Asia and Pacific

Screenshot 2024-09-12 185832

Data

Link: https://drive.google.com/file/d/1-3UpliiCotIfNoTtB6_CmQoBjrj6z6bX/view?usp=sharing All data has been converted into EPSG:8859

Test result

FYI, @ClaraIV @mvmaltitz

Contextual

Each Factors in Contextual Dimension completed.

Screenshot 2024-09-12 190112 Screenshot 2024-09-12 190406 Screenshot 2024-09-12 190544

Contextual_score also succeed with some Python error. Although the Contextual_score.tif exist but the value is incorrect, then it means the script failed to generate the correct value and save as a file.

2024-09-12T19:06:09     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 6009, in contextual_aggregation
              with rasterio.open(aggregation, "w", **meta1) as dst:
              File "C:\Users\benny\AppData\Roaming\Python\Python39\site-packages\rasterio\env.py", line 451, in wrapper
              return f(*args, **kwds)
              File "C:\Users\benny\AppData\Roaming\Python\Python39\site-packages\rasterio\__init__.py", line 341, in open
              dataset = writer(
              File "rasterio\\_io.pyx", line 1464, in rasterio._io.DatasetWriterBase.__init__
              File "rasterio\\_io.pyx", line 330, in rasterio._io._delete_dataset_if_exists
              File "rasterio\\_err.pyx", line 195, in rasterio._err.exc_wrap_int
             rasterio._err.CPLE_AppDefinedError: Deleting Contextual_score.tif failed: Permission denied

Accessibility

Women Travel Pattern

Women Travel Patterns completed successfully for each factors, which rely on ORS.

Screenshot 2024-09-12 192552

But aggregating the output into final WTP trigger a warning, although the file successfully created

2024-09-12T19:23:44     WARNING    warning:C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py:2566: RuntimeWarning: invalid value encountered in divide
              aggregation = np.where(valid_count > 0, cumulative_sum / valid_count, -9999)

             traceback: File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 2566, in wtp_aggregate
              aggregation = np.where(valid_count > 0, cumulative_sum / valid_count, -9999)

Public Transport

Public Transport completed with Python error below. The file mentioned below is not used in other windows. My Windows only has 1 QGIS open for running GEEST.

Screenshot 2024-09-12 195322

2024-09-12T19:47:51     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 294, in 
              self.dlg.PBT_Execute_PB.clicked.connect(lambda: self.ServiceArea(0))
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 1746, in ServiceArea
              shutil.rmtree(tempDir)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 759, in rmtree
              return _rmtree_unsafe(path, onerror)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 629, in _rmtree_unsafe
              onerror(os.unlink, fullname, sys.exc_info())
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
              os.unlink(fullname)
             PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp\\SA_subset_100_PBT.dbf'

Education and Training Facilities

Education and Training Facilities completed with Python error below. The file mentioned below is not used in other windows.

Screenshot 2024-09-12 200208

2024-09-12T19:59:47     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 301, in 
              self.dlg.ETF_Execute_PB.clicked.connect(lambda: self.ServiceArea(1))
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 1746, in ServiceArea
              shutil.rmtree(tempDir)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 759, in rmtree
              return _rmtree_unsafe(path, onerror)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 629, in _rmtree_unsafe
              onerror(os.unlink, fullname, sys.exc_info())
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
              os.unlink(fullname)
             PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp\\SA_subset_10_ETF.dbf'

Health Facilities

Health Facilities completed with Python error below. The file mentioned below is not used in other windows

Screenshot 2024-09-12 200505

2024-09-12T20:04:08     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 308, in 
              self.dlg.HEA_Execute_PB.clicked.connect(lambda: self.ServiceArea(3))
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 1746, in ServiceArea
              shutil.rmtree(tempDir)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 759, in rmtree
              return _rmtree_unsafe(path, onerror)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 629, in _rmtree_unsafe
              onerror(os.unlink, fullname, sys.exc_info())
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
              os.unlink(fullname)
             PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp\\SA_subset_10_HEF.dbf'

Financial Facilities

Financial Facilities completed with Python error below. The file mentioned below is not used in other windows

Screenshot 2024-09-12 200748

2024-09-12T20:06:15     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 315, in 
              self.dlg.FIF_Execute_PB.clicked.connect(lambda: self.ServiceArea(4))
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 1746, in ServiceArea
              shutil.rmtree(tempDir)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 759, in rmtree
              return _rmtree_unsafe(path, onerror)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 629, in _rmtree_unsafe
              onerror(os.unlink, fullname, sys.exc_info())
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
              os.unlink(fullname)
             PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp\\SA_subset_10_FIF.dbf'

Accessibility Aggregate

Accessibility aggregate completed and file also successfully created and loaded as a layer in the map. But it also trigger a Python error below

Screenshot 2024-09-12 201027

2024-09-12T20:09:51     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 6168, in accessibilty_aggregation
              with rasterio.open(aggregation, "w", **meta1) as dst:
              File "C:\Users\benny\AppData\Roaming\Python\Python39\site-packages\rasterio\env.py", line 451, in wrapper
              return f(*args, **kwds)
              File "C:\Users\benny\AppData\Roaming\Python\Python39\site-packages\rasterio\__init__.py", line 341, in open
              dataset = writer(
              File "rasterio\\_io.pyx", line 1464, in rasterio._io.DatasetWriterBase.__init__
              File "rasterio\\_io.pyx", line 330, in rasterio._io._delete_dataset_if_exists
              File "rasterio\\_err.pyx", line 195, in rasterio._err.exc_wrap_int
             rasterio._err.CPLE_AppDefinedError: Deleting Accessibility_score.tif failed: Permission denied

Place Characterization

Active Transport

Active Transport is not responding without any error information, not long after pressing the Execute button

Screenshot 2024-09-12 202600

Safety

Safety trigger a warning on OGR when open the NTL data, although the data is available and loaded into map, dragged from Explorer window.

Screenshot 2024-09-12 211350

2024-09-12T21:05:41     WARNING    Cannot open D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_saf_ntl_2023_8859_clipped.tif ().()

After Execute the button, the label next to the button appear with text: 

Processing failed: Unable to execute algorithm
Could not load source layer for INPUT: temp\clippedClassified.tif not found

See below the python warning

2024-09-12T21:09:16     WARNING    warning:C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\site-packages\osgeo\ogr.py:596: FutureWarning: Neither ogr.UseExceptions() nor ogr.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
              warnings.warn(

             traceback: File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 3304, in SAFRasterizerDelegator
              self.SAFnightTimeLights_v2()
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 3740, in SAFnightTimeLights_v2
              processing.run(
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\tools\general.py", line 109, in run
              return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\core\Processing.py", line 176, in runAlgorithm
              ret, results = execute(alg, parameters, context, feedback, catch_exceptions=False)
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\gui\AlgorithmExecutor.py", line 70, in execute
              results, ok = alg.run(parameters, context, feedback, {}, False)
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\algs\gdal\GdalAlgorithm.py", line 132, in processAlgorithm
              commands = self.getConsoleCommands(parameters, context, feedback, executing=True)
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\algs\gdal\ClipRasterByMask.py", line 170, in getConsoleCommands
              maskLayer, maskLayerName = self.getOgrCompatibleSource(self.MASK, parameters, context, feedback, executing)
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\algs\gdal\GdalAlgorithm.py", line 89, in getOgrCompatibleSource
              ogr_layer_name = GdalUtils.ogrLayerName(ogr_data_path)
              File "C:\PROGRA~1/QGIS33~1.4/apps/qgis-ltr/./python/plugins\processing\algs\gdal\GdalUtils.py", line 439, in ogrLayerName
              ds = ogr.Open(basePath)
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\site-packages\osgeo\ogr.py", line 7481, in Open
              _WarnIfUserHasNotSpecifiedIfUsingExceptions()
              File "C:\PROGRA~1\QGIS33~1.4\apps\Python39\lib\site-packages\osgeo\ogr.py", line 596, in _WarnIfUserHasNotSpecifiedIfUsingExceptions
              warnings.warn(

And below the Processing log

2024-09-12T21:09:16     INFO    Results: {'OUTPUT': 'output_e54fbcd2_4078_43f6_b086_52d0e9371988'}
2024-09-12T21:09:16     INFO    gdalwarp -overwrite -of GTiff -cutline C:/TEMP/processing_JvEyfI/72efe44926e247ec969f00d98a4ce638/MASK.gpkg -cl MASK -crop_to_cutline -dstnodata -3.4028234663852886e+38 temp\tempClassified.tif temp\clippedClassified.tif
2024-09-12T21:09:16     INFO    GDAL command:
2024-09-12T21:09:16     INFO    gdalwarp -overwrite -of GTiff -cutline C:/TEMP/processing_JvEyfI/72efe44926e247ec969f00d98a4ce638/MASK.gpkg -cl MASK -crop_to_cutline -dstnodata -3.4028234663852886e+38 temp\tempClassified.tif temp\clippedClassified.tif
2024-09-12T21:09:16     INFO    GDAL command output:
2024-09-12T21:09:17     CRITICAL    Process returned error code 1
2024-09-12T21:09:17     INFO    Results: {'OUTPUT': 'temp\\clippedClassified.tif'}
2024-09-12T21:09:20     INFO    Results: {'OUTPUT': 'temp/countryUTMLayerBuf.shp'}
2024-09-12T21:09:22     CRITICAL    Unable to execute algorithm
             Could not load source layer for INPUT: temp\clippedClassified.tif not found
2024-09-12T21:09:22     CRITICAL    Warning 1: Ring Self-intersection at or near point 2617160.3599800956 -2438488.8009277876
2024-09-12T21:09:22     CRITICAL    ERROR 1: Cutline polygon is invalid.

And OGR log

2024-09-12T21:05:41     WARNING    Cannot open D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_saf_ntl_2023_8859_clipped.tif ().()
2024-09-12T21:09:16     WARNING    Cannot open EPSG:8859 ().()

Digital Inclusion

Digital Inclusion working perfectly without any warning

Screenshot 2024-09-12 212303

Environmental Hazard

Progress Label show that Processing Complete!, but unfortunately no output files are generated. See the Processing log below.

2024-09-12T22:07:47     INFO    Results: {'OUTPUT': 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/countryUTMLayerBuf.shp'}
2024-09-12T22:07:47     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_firedensity_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:07:47     INFO    GDAL command:
2024-09-12T22:07:47     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_firedensity_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:07:47     INFO    GDAL command output:
2024-09-12T22:07:49     INFO    Process completed successfully
2024-09-12T22:07:49     INFO    Results: {'OUTPUT': 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif'}
2024-09-12T22:07:59     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Fire_Density.tif
2024-09-12T22:07:59     INFO    GDAL command:
2024-09-12T22:07:59     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Fire_Density.tif
2024-09-12T22:07:59     INFO    GDAL command output:
2024-09-12T22:08:00     CRITICAL    Process returned error code 1
2024-09-12T22:08:00     INFO    Results: {'OUTPUT': 'Hazard_Fire_Density.tif'}
2024-09-12T22:08:00     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_land_susceptibility_nasa_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:08:00     INFO    GDAL command:
2024-09-12T22:08:00     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_land_susceptibility_nasa_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:08:00     INFO    GDAL command output:
2024-09-12T22:08:01     INFO    Process completed successfully
2024-09-12T22:08:01     INFO    Results: {'OUTPUT': 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif'}
2024-09-12T22:08:13     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Landslide_Susceptibility.tif
2024-09-12T22:08:13     INFO    GDAL command:
2024-09-12T22:08:13     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Landslide_Susceptibility.tif
2024-09-12T22:08:13     INFO    GDAL command output:
2024-09-12T22:08:14     CRITICAL    Process returned error code 1
2024-09-12T22:08:14     INFO    Results: {'OUTPUT': 'Hazard_Landslide_Susceptibility.tif'}
2024-09-12T22:08:14     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_cyclone_100yr_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:08:14     INFO    GDAL command:
2024-09-12T22:08:14     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_cyclone_100yr_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:08:14     INFO    GDAL command output:
2024-09-12T22:08:15     INFO    Process completed successfully
2024-09-12T22:08:15     INFO    Results: {'OUTPUT': 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif'}
2024-09-12T22:08:34     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Cyclones.tif
2024-09-12T22:08:34     INFO    GDAL command:
2024-09-12T22:08:34     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Cyclones.tif
2024-09-12T22:08:34     INFO    GDAL command output:
2024-09-12T22:08:34     CRITICAL    Process returned error code 1
2024-09-12T22:08:34     INFO    Results: {'OUTPUT': 'Hazard_Cyclones.tif'}
2024-09-12T22:08:34     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_drought_spei_1dot5_100yr_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:08:34     INFO    GDAL command:
2024-09-12T22:08:34     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -dstnodata -9999.0 -tr 100.0 100.0 -r near -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -te_srs EPSG:8859 -of GTiff D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_drought_spei_1dot5_100yr_8859.tif D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif
2024-09-12T22:08:34     INFO    GDAL command output:
2024-09-12T22:08:35     INFO    Process completed successfully
2024-09-12T22:08:35     INFO    Results: {'OUTPUT': 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvResample.tif'}
2024-09-12T22:09:26     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Drought.tif
2024-09-12T22:09:26     INFO    GDAL command:
2024-09-12T22:09:26     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/temp/tempEnvClipResample.tif Hazard_Drought.tif
2024-09-12T22:09:26     INFO    GDAL command output:
2024-09-12T22:09:26     CRITICAL    Process returned error code 1
2024-09-12T22:09:26     INFO    Results: {'OUTPUT': 'Hazard_Drought.tif'}
2024-09-12T22:09:27     INFO    Creating output file that is 4365P x 10757L. 
             Processing D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_firedensity_8859.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
2024-09-12T22:09:27     CRITICAL    Warning 1: Ring Self-intersection at or near point 2617160.3599800956 -2438488.8009277876
2024-09-12T22:09:27     CRITICAL    ERROR 1: Cutline polygon is invalid.
2024-09-12T22:09:27     CRITICAL    Warning 1: for band 1, destination nodata value has been clamped to -128, the original value being out of range.
2024-09-12T22:09:27     INFO    Creating output file that is 4365P x 10757L. 
             Processing D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_land_susceptibility_nasa_8859.tif [1/1] : 0Using internal nodata values (e.g. 127) for image D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_land_susceptibility_nasa_8859.tif. 
             ...10...20...30...40...50...60...70...80...90...100 - done.
2024-09-12T22:09:27     CRITICAL    Warning 1: Ring Self-intersection at or near point 2617160.3599800956 -2438488.8009277876
2024-09-12T22:09:27     CRITICAL    ERROR 1: Cutline polygon is invalid.
2024-09-12T22:09:27     INFO    Creating output file that is 4365P x 10757L. 
             Processing D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_cyclone_100yr_8859.tif [1/1] : 0Using internal nodata values (e.g. 1.70141e+38) for image D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_cyclone_100yr_8859.tif. 
             ...10...20...30...40...50...60...70...80...90...100 - done.
2024-09-12T22:09:27     CRITICAL    Warning 1: Ring Self-intersection at or near point 2617160.3599800956 -2438488.8009277876
2024-09-12T22:09:27     CRITICAL    ERROR 1: Cutline polygon is invalid.
2024-09-12T22:09:27     INFO    Creating output file that is 4365P x 10757L. 
             Processing D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_drought_spei_1dot5_100yr_8859.tif [1/1] : 0Using internal nodata values (e.g. -3.4e+38) for image D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_place_env_drought_spei_1dot5_100yr_8859.tif. 
             ...10...20...30...40...50...60...70...80...90...100 - done.
2024-09-12T22:09:27     CRITICAL    Warning 1: Ring Self-intersection at or near point 2617160.3599800956 -2438488.8009277876
2024-09-12T22:09:27     CRITICAL    ERROR 1: Cutline polygon is invalid.

Using self and manually generated Hazard layer and renamed following the Plugin requirement, the Aggregate Natural Disaster Risk also tested and it works perfectly with some warning in projection

Screenshot 2024-09-12 224324

No transform is available between WGS_1984_Equal_Earth_Asia_Pacific and EPSG:8859 - WGS 84 / Equal Earth Asia-Pacific.
No coordinate operations are available between these two reference systems

Education

Education working perfectly

Screenshot 2024-09-12 224347

Fragility, Conflict, and Violence

FCV is not responding without any error information, not long after pressing the Execute button

Screenshot 2024-09-12 224638

Water and Sanitation

Water process failed. The label next to Execute button appear with text

Error: Clipping failed. Putput file not created: D\temp\geest\geest_qgis_template\output\fji_test_20240912\Place Characterization\WAS.tif

Screenshot 2024-09-12 231756

And trigger a Python error below

2024-09-12T23:11:14     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 4778, in WAS_rasterizer_v2
              raise FileNotFoundError(f"Clipping failed. Output file not created: {rasOutput}")
             FileNotFoundError: Clipping failed. Output file not created: D:\temp\geest\geest_qgis_templates\output\fji_test_20240912\Place Characterization\WAS.tif

And the Processing log below

2024-09-12T23:11:12     INFO    Results: {'OUTPUT': 'output_89901603_4adc_4a87_ba38_8ac620cac862'}
2024-09-12T23:11:12     INFO    Results: {'OUTPUT': 'output_aad3ff82_d243_476c_814e_39e342def1f7'}
2024-09-12T23:11:12     INFO    Results: {'OUTPUT': 'output_d4f51d78_0506_482f_9440_44b73f7990a3'}
2024-09-12T23:11:13     INFO    gdal_rasterize -l INPUT -a buffer_value -tr 100.0 100.0 -init 0.0 -a_nodata -9999.0 -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -ot Float32 -of GTiff C:/TEMP/processing_ePBkbN/07f655912a654e3887e1e993e52eac61/INPUT.gpkg D:\temp\geest\geest_qgis_templates\output\fji_test_20240912\temp\temp_WAS.tif
2024-09-12T23:11:13     INFO    GDAL command:
2024-09-12T23:11:13     INFO    gdal_rasterize -l INPUT -a buffer_value -tr 100.0 100.0 -init 0.0 -a_nodata -9999.0 -te 2521404.4815840046 -2672812.0677260184 2957862.1149176797 -1597099.3760041117 -ot Float32 -of GTiff C:/TEMP/processing_ePBkbN/07f655912a654e3887e1e993e52eac61/INPUT.gpkg D:\temp\geest\geest_qgis_templates\output\fji_test_20240912\temp\temp_WAS.tif
2024-09-12T23:11:13     INFO    GDAL command output:
2024-09-12T23:11:13     INFO    Process completed successfully
2024-09-12T23:11:13     INFO    Results: {'OUTPUT': 'D:\\temp\\geest\\geest_qgis_templates\\output\\fji_test_20240912\\temp\\temp_WAS.tif'}
2024-09-12T23:11:13     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:\temp\geest\geest_qgis_templates\output\fji_test_20240912\temp\temp_WAS.tif "D:\\temp\\geest\\geest_qgis_templates\\output\\fji_test_20240912\\Place Characterization\\WAS.tif"
2024-09-12T23:11:13     INFO    GDAL command:
2024-09-12T23:11:13     INFO    gdalwarp -overwrite -t_srs EPSG:8859 -of GTiff -cutline D:\temp\geest\geest_qgis_templates\layers\iso3\fji_input\fji_polbnda_adm0_country_8859.shp -cl fji_polbnda_adm0_country_8859 -crop_to_cutline -dstnodata -9999.0 D:\temp\geest\geest_qgis_templates\output\fji_test_20240912\temp\temp_WAS.tif "D:\\temp\\geest\\geest_qgis_templates\\output\\fji_test_20240912\\Place Characterization\\WAS.tif"
2024-09-12T23:11:13     INFO    GDAL command output:
2024-09-12T23:11:14     CRITICAL    Process returned error code 1
2024-09-12T23:11:14     INFO    Results: {'OUTPUT': 'D:\\temp\\geest\\geest_qgis_templates\\output\\fji_test_20240912\\Place Characterization\\WAS.tif'}
2024-09-12T23:11:14     INFO    0...10...20...30...40...50...60...70...80...90...100 - done.
2024-09-12T23:11:14     CRITICAL    Warning 1: Ring Self-intersection at or near point 2617160.3599800956 -2438488.8009277876
2024-09-12T23:11:14     CRITICAL    ERROR 1: Cutline polygon is invalid.

Aggregate on Place Characterization

Using self and manually generated other Factors that failed during process via Plugin, the Place score aggregation also experiencing an error with trigger Python error below

2024-09-12T23:18:14     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 6320, in place_characterization_aggregation
              (np.where(WLK_ras != -9999, WLK_ras, 0) * WLK_weight / 100) +
             ValueError: operands could not be broadcast together with shapes (10756,4352) (10756,4364)

Screenshot 2024-09-12 232044

And Python warning below

2024-09-12T23:18:14     WARNING    warning:C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py:6320: RuntimeWarning: overflow encountered in multiply
              (np.where(WLK_ras != -9999, WLK_ras, 0) * WLK_weight / 100) +

             traceback: File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 6320, in place_characterization_aggregation
              (np.where(WLK_ras != -9999, WLK_ras, 0) * WLK_weight / 100) +

Final Aggregate

Using self and manually generated Place_score.tif, the final aggregation has been tested and failed, with below Python error

2024-09-12T23:23:48     WARNING    Traceback (most recent call last):
              File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 6461, in dimensions_aggregation
              (np.where(CD_ras != -9999, CD_ras, 0) * CD_weight / 100) +
             ValueError: operands could not be broadcast together with shapes (10757,4364) (10756,4352)

Screenshot 2024-09-12 233919

And Python warning

2024-09-12T23:23:48     WARNING    warning:C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py:6463: RuntimeWarning: overflow encountered in multiply
              (np.where(PD_ras != -9999, PD_ras, 0) * PD_weight / 100)

             traceback: File "C:\Users/benny/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\gender_indicator_tool\gender_indicator_tool.py", line 6463, in dimensions_aggregation
              (np.where(PD_ras != -9999, PD_ras, 0) * PD_weight / 100)

Insight

Step 1 Classify into discrete layer both completed

Screenshot 2024-09-13 000943

Step 2 Combined score also completed

Screenshot 2024-09-13 001043

Step 3 Aggregation also completed with warning on invalid geometries. See below

Screenshot 2024-09-13 001358

2024-09-13T00:11:27     CRITICAL    Feature (5) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:27     CRITICAL    Feature (20) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:28     CRITICAL    Feature (37) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:28     CRITICAL    Feature (39) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:28     CRITICAL    Feature (51) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:29     CRITICAL    Feature (60) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:29     CRITICAL    Feature (61) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:29     CRITICAL    Feature (69) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:29     CRITICAL    Feature (70) from “aggregation_polygon_utm” has invalid geometry and has been skipped. Please fix the geometry or change the “Invalid features filtering” option for this input or globally in Processing settings.
2024-09-13T00:11:30     INFO    Results: {'OUTPUT': 'D:/temp/geest/geest_qgis_templates/output/fji_test_20240912/Insights/4) Aggregation/WEE_pop_adm_score.shp'}

[!NOTE]

  • As not all boundary are topologically clean, maybe some of them has invalid geometry, consider using GDALWARP_IGNORE_BAD_CUTLINE=YES during clip with GDAL, to minimize error.
  • During aggregation, result from the plugin are different compare to manually aggregate using QGIS or other software. I think this is about how to handle NaN value. At the processing cell location, if any of the input rasters has NoData, then that NoData value will be ignored. The statistics will be computed by only considering the cells with valid data. This is the default.
bennyistanto commented 2 months ago

Tunisia, Test 13 Sep 2024

Using projection EPSG:32632, some of the test going well, and some has an error. The problem, warning and error are similar with what I have in Fiji test.

Data

Link: https://drive.google.com/file/d/1euhgy5oQVPi_B-sUpQUMUjJrp30dYxAj/view?usp=drive_link All data has been converted into EPSG:32632

timlinux commented 2 months ago

Thanks for this really detailed breakdown @bennyistanto. Just a question regarding CRS: is there any reason we can't just use epsg:3857 spherical Mercator? I think the accuracy loss will be minimal and we can avoid complicated workflows.

mvmaltitz commented 1 month ago

@timlinux Please see the child tickets below:

Contextual

Accessibility

Place Characterization

bennyistanto commented 1 month ago

Thanks for this really detailed breakdown @bennyistanto. Just a question regarding CRS: is there any reason we can't just use epsg:3857 spherical Mercator? I think the accuracy loss will be minimal and we can avoid complicated workflows.

Hi @timlinux I did test using epsg:3857 and it's also working fine for most country. I don't know the idea behind why we use different projection for each country as written on this https://worldbank.github.io/GEEST/docs/user_guide.html#list-of-crss-for-sids My best guess would be on accuracy and distortion. The team try to avoid significant distortion on higher latitude.

But the epsg:3857 really not working on country like Fiji which located in 180th meridian and the area will be splitted. @dragosgontariu spent lot of time to work on Fiji without succeed (with various error), then he splitted the country into 2 file (west and east), and still experienced an error in some step.

Other reason will be the availability and geometry quality of the boundary it self. We rely on HDX for each country boundary. And for the case of Fiji, the boundary from HDX using World_Merctor_150, is a modified Mercator projection centered on the 150E, and no standard EPSG code for this.

So, I decided to use epsg:8859 as alternative and it will recognized by QGIS.

Below picture is the Fiji boundary using 3857, translated from World_Merctor_150.

Screenshot 2024-09-23 175836

timlinux commented 1 month ago

So there is a pretty crazy bug here, I am quite surprised how anything ever worked:

image

Above you can see what the signature for creating a vector layer is

Below we can see the original author (1) created a vector file (2) using the geojson dump of that file (I mean the actual json data, not the file handle) as its URI (3)

image

Also the processing tools as you rightly surmised @bennyistanto do not omit invald geometries. I have added logic that can let us do that but I need to work through each processing call and make sure that is set up...

timlinux commented 1 month ago

Also the idea of writing utm data to geojson is not sound since geojson requires the data be in 4326:

image