Issues in creating weights for WofE (ArcGIS 10.6 and 10.2)

[dgupt20]

I am having a similar issue in creating weights as mentioned in issue #8. I have tried both in ArcGIS 10.6 and 10.2. All my raster layers are in the same projection and in m. I am getting two different types of errors. I am trying to understand the problem but ran out of ideas after trying many different changes in the mask layer and cell sizes. Any suggestions would be greatly appreciated. Below are the error codes:

Error 1:

Number of Training Sites: 184 Study Area Raster: F:\734 data\WoE1\boundary1 Study Area Area (sq km): 1508.84 Unit Cell Area (sq km): 0.04 Map Units: meter GP ERRORS: Executing: CreateTable "F:\734 data\WoE1" con.dbf 'F:\734 data\WoE1\WtsStatistics0.dbf' # Start Time: Fri May 29 21:51:58 2020 ERROR 999999: Error executing function. Failed to execute (CreateTable). Failed at Fri May 29 21:51:58 2020 (Elapsed Time: 0.05 seconds)

PYTHON ERRORS: Traceback Info: File "C:\SDM\Scripts\NewWtsofEvIter.py", line 182, in gp.CreateTable_management(os.path.dirname(wtstable), os.path.basename(wtstable), Statistics)

Error Info: <class 'arcgisscripting.ExecuteError'>: ERROR 999999: Error executing function. Failed to execute (CreateTable).

Failed script CalculateWeights...

Traceback (most recent call last): File "C:\SDM\Scripts\NewWtsofEvIter.py", line 182, in gp.CreateTable_management(os.path.dirname(wtstable), os.path.basename(wtstable), Statistics) ExecuteError: ERROR 999999: Error executing function. Failed to execute (CreateTable).

Failed to execute (CalculateWeights). Failed at Fri May 29 21:51:58 2020 (Elapsed Time: 3.98 seconds)

Error 2:

Executing: CalculateWeights Elev_relcass.tif VALUE 2016Survey_Red Categorical "F:\734 data\WoE1\elev_CT.dbf" 1.96 0.04 -99 Start Time: Fri May 29 22:07:32 2020 Running script CalculateWeights... Workspace: F:\734 data\WoE1 Scratch workspace: F:\734 data\WoE\Temp PYTHON ERRORS: Traceback Info: File "C:\SDM\Scripts\SDMValues.py", line 25, in appendSDMValues maskrows = gp.SearchCursor(gp.describe(gp.mask).catalogpath)

Error Info: <type 'exceptions.RuntimeError'>: ERROR 999999: Error executing function.

GP ERRORS: Executing: GetRasterProperties Elev_relcass.tif VALUETYPE # Start Time: Fri May 29 22:07:33 2020 Value type = Unsigned Integer Succeeded at Fri May 29 22:07:33 2020 (Elapsed Time: 0.05 seconds)

PYTHON ERRORS: Traceback Info: File "C:\SDM\Scripts\NewWtsofEvIter.py", line 171, in SDMValues.appendSDMValues(gp, Unitarea, TrainingSites)

Error Info: <type 'exceptions.RuntimeError'>: ERROR 999999: Error executing function.

Failed script CalculateWeights...

Traceback (most recent call last): File "C:\SDM\Scripts\NewWtsofEvIter.py", line 171, in SDMValues.appendSDMValues(gp, Unitarea, TrainingSites) File "C:\SDM\Scripts\SDMValues.py", line 25, in appendSDMValues maskrows = gp.SearchCursor(gp.describe(gp.mask).catalogpath) RuntimeError: ERROR 999999: Error executing function.

Failed to execute (CalculateWeights). Failed at Fri May 29 22:07:33 2020 (Elapsed Time: 0.34 seconds)

[artolaiho]

QUOTE: "Executing: CreateTable "F:\734 data\WoE1" ..."

Can you remove all spaces from your folder and file names and test again. If You still get errors, copy whole Messages block to comment box of this issue.

[dgupt20]

Sorry for the delay. I tried the whole process after removing spaces but getting a different error message this time. The error looks the same both in ArcGIS 10.2 and 10.6. Any help would be greatly appreciated. Here is the error log:

Executing: CalculateWeights Clip_elevation Value SB_2016survey Categorical H:\Stratification\WoE\elev_CT.dbf 1.96 0.0484 -99 Start Time: Wed Jun 3 13:34:29 2020 Running script CalculateWeights... Workspace: H:\Stratification\WoE Scratch workspace: H:\Stratification\WoE\Temp Prior Probability: 0.005900 Training Set: H:\Stratification\SB_2016survey.shp Number of Training Sites: 184 Study Area Raster: H:\Stratification\WoE\734Bound.tif Study Area Area (sq km): 1509.354 Unit Cell Area (sq km): 0.0484 Map Units: meter GP ERRORS: Executing: AddField H:\Stratification\WoE\elev_CT.dbf W_STD DOUBLE 10 6 # "Generalized Weight Std" NULLABLE NON_REQUIRED # Start Time: Wed Jun 3 13:34:39 2020 Adding W_STD to H:\Stratification\WoE\elev_CT.dbf... Succeeded at Wed Jun 3 13:34:39 2020 (Elapsed Time: 0.04 seconds)

PYTHON ERRORS: Traceback Info: File "H:\Stratification\SDM\Scripts\NewWtsofEvIter.py", line 234, in rows = gp.SearchCursor(EvidenceLayer)

Error Info: <type 'exceptions.IOError'>: "C:\Users\sdasgupt\AppData\Local\Temp\82\arc1D1E\xa52c7334_b302_4406_aaf5_22657d10f56ay0.afr" does not exist

Failed script CalculateWeights...

Traceback (most recent call last): File "H:\Stratification\SDM\Scripts\NewWtsofEvIter.py", line 234, in rows = gp.SearchCursor(EvidenceLayer) IOError: "C:\Users\sdasgupt\AppData\Local\Temp\82\arc1D1E\xa52c7334_b302_4406_aaf5_22657d10f56ay0.afr" does not exist

Failed to execute (CalculateWeights). Failed at Wed Jun 3 13:34:40 2020 (Elapsed Time: 11.24 seconds)

[dgupt20]

Did not notice before but the error message in 10.2 is a bit different than the one in 10.6. Here is the error log:

Executing: CalculateWeights Clip_elevation.tif Value 2016Survey_Red Categorical F:\734data\WoE1\elev_CT.dbf 1.96 0.04 -99 Start Time: Wed Jun 03 16:06:20 2020 Running script CalculateWeights... Workspace: F:\734data\WoE1 Scratch workspace: F:\734data\WoE\Temp Prior Probability: 0.004031 Training Set: F:\734data\2016Survey_Red.shp Number of Training Sites: 184 Study Area Raster: F:\734data\WoE1\boundary1 Study Area Area (sq km): 1825.6964 Unit Cell Area (sq km): 0.04 Map Units: meter GP ERRORS: Executing: AddField F:\734data\WoE1\elev_CT.dbf W_STD DOUBLE 10 6 # "Generalized Weight Std" NULLABLE NON_REQUIRED # Start Time: Wed Jun 03 16:06:27 2020 Adding W_STD to F:\734data\WoE1\elev_CT.dbf... Succeeded at Wed Jun 03 16:06:27 2020 (Elapsed Time: 0.08 seconds)

PYTHON ERRORS: Traceback Info: File "C:\SDM\Scripts\NewWtsofEvIter.py", line 234, in rows = gp.SearchCursor(EvidenceLayer)

Error Info: <type 'exceptions.RuntimeError'>: ERROR 999999: Error executing function.

Failed script CalculateWeights...

Traceback (most recent call last): File "C:\SDM\Scripts\NewWtsofEvIter.py", line 234, in rows = gp.SearchCursor(EvidenceLayer) RuntimeError: ERROR 999999: Error executing function.

Failed to execute (CalculateWeights). Failed at Wed Jun 03 16:06:28 2020 (Elapsed Time: 7.33 seconds)

[artolaiho]

I will continue testing the Grand Wofe but I've already checked calculate weights and calculate response tools. You can find them and others from arto-dev branch. There is also Grand Wofe and Logical Regression but maybe I will modify them further when I have looked all issues.

I've found that if Evidence Layer is RasterBand, Calculate Weights gives error 999999 on the line "rows = gp.SearchCursor(EvidenceLayer)". Evidence Layer can be RasterLayer or RasterDataset. Can You test again using tools from arto-dev. There is also Readme describing changes.

[dgupt20]

Thank you. Tried with the tools in arto-dev. Getting the following error:

Executing: CalculateWeightsTool Clip_aspect # 2016Survey_Red Categorical H:\Stratification\WoE\aspectN_CT 1.96 0.04 -99 # Start Time: Thu Jun 4 20:01:02 2020 Running script CalculateWeightsTool... Reloaded arcsdm modules Evidence Layer is Clip_aspect and its data type is RasterLayer

========== arcsdm values ========== ArcSdm 5.01.07 for ArcGis (Pro and Desktop)

Arcgis environment: Desktop (10.6.1) Workspace: H:\Stratification\WoE (FileSystem) Scratch workspace: H:\Stratification\WoE\Temp (FileSystem) Map Units: meter Mask: "734Bound.tif" and it is RasterLayer Mask size: 1509.354 Cell Size: 220

Training Sites: 1

Unit Cell Area: 0.04km^2, Cells in area: 37733.85 Prior Probability: 0.000027 Training Set: H:\Stratification\2016Survey_Red.shp Study Area Raster: H:\Stratification\WoE\734Bound.tif Study Area Area: 1509.354km^2 ========== Calculate weights ========== Creating table: H:\Stratification\WoE\aspectN_CT.dbf (Categorical) "C:\Users\sdasgupt\AppData\Local\Temp\102\arcF1CE\x896d9338_9493_429c_b480_d8d682f8d353y0.afr" does not exist GPMEs: 0 CW - PYTHON ERRORS: Traceback Info: File "H:\Stratification\SDM\ArcSDM-arto-dev\Toolbox\arcsdm\calculateweights.py", line 312, in Calculate rows = gp.SearchCursor(EvidenceLayer)

Error Info: <function format_exc at 0x2D0B7130>

Unhandled exception caught Traceback (most recent call last): File "H:\Stratification\SDM\ArcSDM-arto-dev\Toolbox\arcsdm\common.py", line 88, in execute_tool func(self, parameters, messages) File "H:\Stratification\SDM\ArcSDM-arto-dev\Toolbox\arcsdm\calculateweights.py", line 312, in Calculate rows = gp.SearchCursor(EvidenceLayer) IOError: "C:\Users\sdasgupt\AppData\Local\Temp\102\arcF1CE\x896d9338_9493_429c_b480_d8d682f8d353y0.afr" does not exist

Completed script CalculateWeightsTool... Failed to execute (CalculateWeightsTool). Failed at Thu Jun 4 20:01:26 2020 (Elapsed Time: 23.70 seconds)

[artolaiho]

Thank you for the feedback. I have found that the error on line "rows = gp.SearchCursor(EvidenceLayer)" due to the fact that the raster type is RasterBand. I can't understand that "afr file does not exist error" Why the tool uses TEMP folder? I have seen it before but I can't remember the situation. You have used tif files as input. I have to look for one somewhere and test again using them. This will take a few days but I will not give up with this. I uploaded latest version of calculateweights.py to arto-dev a moment ago but I don't think that version fixes your problem. I'll get back.

[dgupt20]

Thanks a bunch for your time. I can share the input raster evidence and training vector files if that would make it easier to regenerate the error. I have about eight raster evidence files but all of them give a similar error. Please let me know where to send/share the download link. Thanks again for your help.

[artolaiho]

Can you write download link here? You can remove those files from download folder after I have copied them. I don't want to publish my email address - neither private nor work address.

[dgupt20]

Here is the link: There is a text file describing the file names, projection, and problem objective. Let me know if I missed anything. Thanks for doing this!

[artolaiho]

Thanks. I have downloaded and extracted your test data.

[artolaiho]

Hi. I found the cause of the error in calculateweights.py line "rows = gp.SearchCursor (EvidenceLayer)". There is not an attribute table in Your Evidence Raster TIF files (Clip_aspect.tif, Clip_elevation.tif and ClipSlope.tif). Use ArcMap's "Build Raster Attribute" tool to add an attribute table to your TIF rasters. You can find that tool using ArcMap's Search function by typing "Build Raster Attribute" as the search term. I didn't get that "afr file does not exist" error on my own machine. I don't know why you got that error. I will continue with Grand Wofe. There is something strange in logisticregression.py.

[dgupt20]

Great! Thanks a lot. I will change the rasters and try this again to see if it works on my station; Will get back soon with an update.

[artolaiho]

Hello. If you still get "C:\Users\sdasgupt\AppData\Local\Temp\102\arcF1CE\x896d9338_9493_429c_b480_d8d682f8d353y0.afr does not exist" error, try to change your TEMP environment variable to refer to a shorter folder path. You must change the TEMP variable in your user profile before starting ArcMap.

[artolaiho]

Download new versions of ArcSDM.pyt (from Toolbox); common.py, calculateweights.py, calculateresponse.py, logisticregression.py and grand_wofe_lr.py (from arcsdm) and also new xml files (from Toolbox). All those files are in arto-dev branch. Grand Wofe is now working with your test data when you are using ArcMap 10.6.1. There is still some problems with ArcGIS Pro 2.5 and File system workspace. common.py creates folder C:\SDMtemp and sets environment variable TEMP to it. calculateweights.py tests that evidence raster has attribute table. other changes is described at readme.

[dgupt20]

Thank you. Will try with the new updated files. With smaller number of evidence rasters, 'calculate weight' and 'calculate response' options seem to be working but ArcGIS is crashing with a full suite of evidence raster. Not sure if it is a memory problem. Is there a user manual for Grand Wofe? The original ArcSDM did not have this feature.

[artolaiho]

Sorry, I don't have any information about any ArcSDM user manual and I don't know who could write one.

[tachyon-work]

I run training courses on mineral potential mapping using ArcSDM, but the material isn’t available for free. I run this as a commercial short course by request. See here: http://www.kenex.co.nz/Services/training.asp

With regards to the aforementioned issues, my tinkering with the code basically showed that ArcMap 10.7 and above require the use of geodatabases in the Current and Scratch workspaces in order for Calculate Response to work. Due to the fact that Grand WofE basically just calls Calculate Weights and then Calculate Response, this is why your environment settings will have to be set to geodatabases for the tool to work. I currently have all the WofE tools running successfully in 10.7 without issue, but I have had to make code tweaks for it to work.

For what it’s worth: I have removed the Grand WofE script from the version of ArcSDM that I am working with, because it makes assumptions that are frequently illogical in terms of where to place thresholds. It produces statistically valid results that are geologically nonsense. I try to teach the use of Calculate Weights and Calculate Response separately to force the user to actually review the weights tables to make sure that they make geological sense before integrating the maps using the Calculate Response tool 🙂

[dgupt20]

Thanks for the training link. I will check it out.

For a smaller number of evidence rasters, the tool is running and calculating both weights and responses. However, ArcGIS is still crashing when I add more than 6/7 layers. I will run it on a different machine. The problem could be related to running it over the network.

[tachyon-work]

Just throwing some ideas out there...

1. After you have run calculate weights, try reclassifying your original multi-class rasters to binary, then recalcuate the weights for the binary maps and use these binary maps and weights tables in calculate response? I know calculate response is designed to read in the gen_class field to determine the threshold, but reclassifying to binary and using these in calculate response is my standard approach, and I have no problems integrating 20+ maps (in either 10.3 or 10.7).

2. If you are using Grand WofE for this rather than the standalone calculate response tool, are you choosing Logistic Regression as the integration method? Or WofE? There are limits on the LR tool that are hard coded into the sdmlr.exe executable that the LR function calls. You can only have a maximum of 6000 unique conditions for LR as it is coded in ArcSDM (which is why it needs to be recoded from scratch, it's just nobody really has the time to do it). So basically figure out how many unique conditions you have (how many classes are in each of your input maps? then multiply the number of classes for each map together), and if it's >6000 then LR will fail.