search

gtkfi / ArcSDM

Spatial Data Modeler 6 for ArcGIS Pro
33 stars 18 forks source link

Logistic Regression Errors #103

Closed sysuxiaofan closed 4 years ago

sysuxiaofan commented 6 years ago

Caught ExecuteError in logistic regression. Details: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine). Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

trotor commented 6 years ago

Which version and what inputs?

On Tue, Jul 3, 2018, 11:25 cugxiaofan notifications@github.com wrote:

Caught ExecuteError in logistic regression. Details: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine). Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gtkfi/ArcSDM/issues/103, or mute the thread https://github.com/notifications/unsubscribe-auth/AKV5eexDkianp23p83qBQS2wHMESsE3Uks5uCypbgaJpZM4VAgKM .

sysuxiaofan commented 6 years ago

Executing: LogisticRegressionTool bwof_gch1;bwof_ghy1;bwof_int1;bwof_sta1;bwof_str1 f;f;f;f;f F:\data_processed\ltemp\bwof_gch1_Weights.dbf;F:\data_processed\ltemp\bwof_ghy1_Weights.dbf;F:\data_processed\ltemp\bwof_int1_Weights.dbf;F:\data_processed\ltemp\bwof_sta1_Weights.dbf;F:\data_processed\ltemp\bwof_str1_Weights.dbf cumo -99 4 F:\data_processed\temp\blr_ply.dbf F:\data_processed\temp\blr_coef.dbf F:\data_processed\temp\blr_post.tif F:\data_processed\temp\blr_poststd.tif F:\data_processed\temp\blr_conf.tif Start Time: Tue Jul 03 15:14:23 2018 Running script LogisticRegressionTool... Checking environment... Input rasters: [u'bwof_gch1', u'bwof_ghy1', u'bwof_int1', u'bwof_sta1', u'bwof_str1'] Evidence_types: [u'f', u'f', u'f', u'f', u'f'] Wts_Tables: [u'F:\data_processed\ltemp\bwof_gch1_Weights.dbf', u'F:\data_processed\ltemp\bwof_ghy1_Weights.dbf', u'F:\data_processed\ltemp\bwof_int1_Weights.dbf', u'F:\data_processed\ltemp\bwof_sta1_Weights.dbf', u'F:\data_processed\ltemp\bwof_str1_Weights.dbf'] F:\data_processed\ltemp\bwof_gch1_Weights.dbf F:\data_processed\ltemp\bwof_ghy1_Weights.dbf F:\data_processed\ltemp\bwof_int1_Weights.dbf F:\data_processed\ltemp\bwof_sta1_Weights.dbf F:\data_processed\ltemp\bwof_str1_Weights.dbf TrainPts: cumo MissingDataValue: -99 thmUC: F:\data_processed\ltemp\tmp_UCras0

========== arcsdm values ========== ArcSdm 5.01.05 for ArcGis (Pro and Desktop)

Arcgis environment: Desktop (10.2) Workspace: F:\data_processed\ltemp (FileSystem) Scratch workspace: F:\data_processed\ltemp (FileSystem) Map Units: meter Mask: "bwof_gch1" and it is RasterLayer Mask size: 52296.0 Cell Size: 2000

Training Sites: 12

Unit Cell Area: 4.0km^2, Cells in area: 13074.0 Prior Probability: 0.000918 Training Set: F:\data_processed\2016-10-13\weights\cumo.shp Study Area Raster: F:\data_processed\2016-10-13\weights\bwof_gch1 Study Area Area: 52296.0km^2 Creating Generalized Class rasters. Output_Raster: F:\data_processed\ltemp\bwof_gch1_G44 Temp_Raster: F:\data_processed\ltemp\tmp_rst220 Output_Raster: F:\data_processed\ltemp\bwof_ghy1_G44 Temp_Raster: F:\data_processed\ltemp\tmp_rst221 Output_Raster: F:\data_processed\ltemp\bwof_int1_G44 Temp_Raster: F:\data_processed\ltemp\tmp_rst222 Output_Raster: F:\data_processed\ltemp\bwof_sta1_G44 Temp_Raster: F:\data_processed\ltemp\tmp_rst223 Output_Raster: F:\data_processed\ltemp\bwof_str1_G44 Temp_Raster: F:\data_processed\ltemp\tmp_rst224 Combining...F:\data_processed\ltemp\bwof_gch1_G44;F:\data_processed\ltemp\bwof_ghy1_G44;F:\data_processed\ltemp\bwof_int1_G44;F:\data_processed\ltemp\bwof_sta1_G44;F:\data_processed\ltemp\bwof_str1_G44

Caught ExecuteError in logistic regression. Details: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute gp.Combine_sa(Input_Combine_rasters, str(thmUC))
Error Info: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Traceback (most recent call last): File "", line 1562, in execute File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute gp.Combine_sa(Input_Combine_rasters, str(thmUC))
ExecuteError: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Failed to execute (LogisticRegressionTool). Failed at Tue Jul 03 15:16:03 2018 (Elapsed Time: 1 minutes 40 seconds)

sysuxiaofan commented 6 years ago

ArcGIS 10.2, five WofE layers.

trotor commented 6 years ago

Ok, please create empty text file called DEBUG into toolbox\arcsdm\ directory and rerun the project for more detailed output. It could be some error in parameters or perhaps incompatibility with 10.2.

sysuxiaofan commented 6 years ago

I modified the following codes:

thmUC = gp.createscratchname("tmp_UCras", '', 'raster', gp.scratchworkspace) # Xiao Fan modified

    thmUC ="r\""+ gp.createscratchname("tmp_UCras", '', 'raster',  gp.scratchworkspace) + "\""    # Xiao Fan modified

Input_Combine_rasters =";".join(Wts_Rasters) # Xiao Fan modified

    Input_Combine_rasters ="\""+"r'" + "'; r'".join(Wts_Rasters) + "'"+"\"" # Xiao Fan modified

The original bugs seem have been repaired. But, it shows the following erros Caught ExecuteError in logistic regression. Details: Failed to execute. Parameters are not valid. ERROR 000878: Output raster: The length of the grid base name in F:\data_processed\ltemp\r"F:\data_processed\ltemp\tmp_UCras0" is longer than 13. Failed to execute (Combine).

Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 164, in Execute gp.Combine_sa(Input_Combine_rasters, thmUC) # Xiao Fan modified

Error Info: Failed to execute. Parameters are not valid. ERROR 000878: Output raster: The length of the grid base name in F:\data_processed\ltemp\r"F:\data_processed\ltemp\tmp_UCras0" is longer than 13. Failed to execute (Combine).

Traceback (most recent call last): File "", line 1562, in execute File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 164, in Execute gp.Combine_sa(Input_Combine_rasters, thmUC) # Xiao Fan modified ExecuteError: Failed to execute. Parameters are not valid. ERROR 000878: Output raster: The length of the grid base name in F:\data_processed\ltemp\r"F:\data_processed\ltemp\tmp_UCras0" is longer than 13. Failed to execute (Combine).

Failed to execute (LogisticRegressionTool).

sysuxiaofan commented 6 years ago

And, I have tried to short the file name (tmp_UCras0, e.g. UCr), but the erros still exists.

sysuxiaofan commented 6 years ago

I have referenced the ArcGIS help, but I don't think my new codes have any problem.

Combine_sample.py Description: Combines multiple rasters such that a unique output value is assigned to each unique combination of input values. Requirements: None Author: ESRI Date: Sept 6, 2005 Import system modules import arcgisscripting Create the Geoprocessor object gp = arcgisscripting.create() try:

Set local variables

outRaster = "C:/data/final_1"

Check out Spatial Analyst extension license

gp.CheckOutExtension("Spatial")

Process: Combine...

gp.Combine_sa("'C:/data/ras_1';'C:/data/ras_2'", outRaster) except:

If an error occurred while running a tool, then print the messages.

print gp.GetMessages()

trotor commented 6 years ago

What is the output of this (print it after it is set): Input_Combine_rasters ="""+"r'" + "'; r'".join(Wts_Rasters) + "'"+""" # Xiao Fan modified

sysuxiaofan commented 6 years ago

Input_Combine_rasters ="r'F:\data_processed\ltemp\bwof_gch1_G1'; r'F:\data_processed\ltemp\bwof_ghy1_G1'; r'F:\data_processed\ltemp\bwof_int1_G1'; r'F:\data_processed\ltemp\bwof_sta1_G1'; r'F:\data_processed\ltemp\bwof_str1_G1'"

gp.Combine_sa("'C:/data/ras_1';'C:/data/ras_2'", outRaster)

sysuxiaofan commented 6 years ago

using r to make \t or \r not work.

sysuxiaofan commented 6 years ago

and thmUC=r"F:\data_processed\ltemp\tmp_UCras0"

trotor commented 6 years ago

You tried also without r? I think it could be the problem... You can also try replacing \ with / as iirc they are replaceable.

On Wed, Jul 4, 2018 at 3:28 PM cugxiaofan notifications@github.com wrote:

Input_Combine_rasters ="r'F:\data_processed\ltemp\bwof_gch1_G1'; r'F:\data_processed\ltemp\bwof_ghy1_G1'; r'F:\data_processed\ltemp\bwof_int1_G1'; r'F:\data_processed\ltemp\bwof_sta1_G1'; r'F:\data_processed\ltemp\bwof_str1_G1'"

gp.Combine_sa("'C:/data/ras_1';'C:/data/ras_2'", outRaster)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gtkfi/ArcSDM/issues/103#issuecomment-402464403, or mute the thread https://github.com/notifications/unsubscribe-auth/AKV5eUoDbLicRiOZHoxtCRpNx9v_Qn2Gks5uDLTVgaJpZM4VAgKM .

sysuxiaofan commented 6 years ago

Caught ExecuteError in logistic regression. Details: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 165, in Execute gp.Combine_sa("'F:/data_processed/ltemp/bwof_gch1_G0';'F:/data_processed/ltemp/bwof_ghy1_G0';'F:/data_processed/ltemp/bwof_int1_G0';'F:/data_processed/ltemp/bwof_sta1_G0';'F:/data_processed/ltemp/bwof_str1_G0'", "F:/data_processed/ltemp/tmp_UCras0") # Xiao Fan modified

Error Info: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Traceback (most recent call last): File "", line 1562, in execute File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 165, in Execute gp.Combine_sa("'F:/data_processed/ltemp/bwof_gch1_G0';'F:/data_processed/ltemp/bwof_ghy1_G0';'F:/data_processed/ltemp/bwof_int1_G0';'F:/data_processed/ltemp/bwof_sta1_G0';'F:/data_processed/ltemp/bwof_str1_G0'", "F:/data_processed/ltemp/tmp_UCras0") # Xiao Fan modified ExecuteError: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Failed to execute (LogisticRegressionTool). Failed at Wed Jul 04 22:22:06 2018 (Elapsed Time: 13.32 seconds)

trotor commented 6 years ago

Okay, perhaps there is something wrong with 10.2.

Could you try something in this manner: outCombine = Combine(["filter", "zone", "source.img", "dec.tif"]) outCombine.save("C:/sapyexamples/output/outcombine2")

So in this case it would be just outCombine = arcpy.Combine( Wts_Rasters); outCombine.save( outRaster )

On Wed, Jul 4, 2018 at 5:23 PM cugxiaofan notifications@github.com wrote:

Caught ExecuteError in logistic regression. Details: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 165, in Execute gp.Combine_sa("'F:/data_processed/ltemp/bwof_gch1_G0';'F:/data_processed/ltemp/bwof_ghy1_G0';'F:/data_processed/ltemp/bwof_int1_G0';'F:/data_processed/ltemp/bwof_sta1_G0';'F:/data_processed/ltemp/bwof_str1_G0'", "F:/data_processed/ltemp/tmp_UCras0") # Xiao Fan modified

Error Info: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Traceback (most recent call last): File "", line 1562, in execute File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 165, in Execute gp.Combine_sa("'F:/data_processed/ltemp/bwof_gch1_G0';'F:/data_processed/ltemp/bwof_ghy1_G0';'F:/data_processed/ltemp/bwof_int1_G0';'F:/data_processed/ltemp/bwof_sta1_G0';'F:/data_processed/ltemp/bwof_str1_G0'", "F:/data_processed/ltemp/tmp_UCras0") # Xiao Fan modified ExecuteError: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Failed to execute (LogisticRegressionTool). Failed at Wed Jul 04 22:22:06 2018 (Elapsed Time: 13.32 seconds)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gtkfi/ArcSDM/issues/103#issuecomment-402493742, or mute the thread https://github.com/notifications/unsubscribe-auth/AKV5eeLwIUjT9AqrAaKtv3s-a-pVk4gMks5uDM_WgaJpZM4VAgKM .

Eliasmgprado commented 6 years ago

I had a hard time running the logistiregression.py script.

The main problems were:

some of these problems have already been reported in this post. after a lot of work, I was able to solve the problems and finally run the script successfully.

Here are the script modifications i need to do:

Running on ArcMap 10.5.1

trotor commented 6 years ago

Thank you for your thorough analysis and debugging on this issue! I will try this out ASAP and hopefully incorporate this fix into next version.

On Wed, Jul 18, 2018 at 5:42 PM Elias Martins Guerra Prado < notifications@github.com> wrote:

I had a hard time running the logistiregression.py script.

The main problems were:

  • Arcmap crashes when running AddJoin_management function
  • Files are not correctly identified
  • Some unecessary lines giving error
  • con_sa () function giving error

some of these problems have already been reported in this post. after a lot of work, I was able to solve the problems and finally run the script successfully.

Here are the script modifications i need to do:

  • added on line 25 the import of spatial analyst functions from arcpy.sa import *
  • modified the lines 94 and 95: Input_Rasters[i] = s.strip("'"); if arcsdm.common.testandwarn_filegeodatabase_source(s.strip("'")):
  • modified lines 141 to 148 : Temp_Raster = gp.CreateScratchName('tmp_rst', '', 'rst', gp.scratchworkspace) gp.copyraster_management(Input_Raster, Temp_Raster) gp.JoinField_management(Temp_Raster, 'Value', Wts_Table, 'CLASS') gp.AddMessage('Temp_Raster: %s'%(str(Temp_Raster))) gp.Lookup_sa(Temp_Raster, "GEN_CLASS", Output_Raster)
  • commented the lines 461, 462, 464, 466 strLine = fLR.readline() #vTabUCrows = workarounds_93.rowgen(gp.SearchCursor(vTabUC)) #vTabUCrow = vTabUCrows.Next() #ttl = 0 #while vTabUCrow: #for vTabUCrow in vTabUCrows: ttl += 1
  • comment line 582:

    gp.makerasterlayer_management(cmb, cmbrl)

  • modified lines 587 to 589: cmb_cpy = gp.createscratchname("cmb_cpy", '', 'raster', arcpy.env.scratchFolder) gp.copyraster_management(cmb, cmb_cpy) gp.JoinField_management(cmb_cpy, 'Value', tbltv, 'ID')
  • modified lines 607 to 609: outcon1 = Con(cmb_cpy, Lookup(cmb_cpy,"LRPOSTPROB"),"0","LRPOSTPROB > 0") outcon1.save(outRaster1) outcon2 = Con(cmb_cpy,Lookup(cmb_cpy,"LR_STD_DEV"),"0","LR_STD_DEV > 0") outcon2.save(outRaster2) outcon3 = Con(cmb_cpy,Lookup(cmb_cpy,"LRTVALUE"),"0","LRTVALUE > 0") outcon3.save(outRaster3)

Running on ArcMap 10.5.1

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gtkfi/ArcSDM/issues/103#issuecomment-405955595, or mute the thread https://github.com/notifications/unsubscribe-auth/AKV5eRrLHSGl1bXEnwGscyatI_HsG1G3ks5uH0k3gaJpZM4VAgKM .

trotor commented 6 years ago

Could you send me the fixed .py file here as attachement? Others could benefit from your valuable work!

Eliasmgprado commented 6 years ago

Here is the file, code needs cleaning of some unecessary commented lines. logisticregression.zip

trotor commented 6 years ago

Update on this issue: I think I haven't yet merged the code - we had summer holidays and some computer related problems. Will need to check and do when smoke clears.

trotor commented 6 years ago

Hi, merged these changes - if you have time, please test if it went correctly.

Mahfoudb commented 5 years ago

Caught ExecuteError in logistic regression. Details: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine). Traceback File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info: ERROR 010328: Syntax error at or near symbol ,. ERROR 010267: Syntax error in parsing grid expression. Failed to execute (Combine).

Hello, Have you found a solution for this problem because I get the same one.

Mahfoudb commented 5 years ago

Can you help me please to solve this problem

trotor commented 5 years ago

Can you help me please to solve this problem

What inputs are you using?

Mahfoudb commented 5 years ago

Executing: LogisticRegression pnt;rh;ocp;lth;frc;exp;elv f;f;f;f;f;f;f pnt_ct;rh_ct;ocp_ct;lth_ct;frc_ct;exp_ct;elv_ct TP -99 1 C:\Data\LR_logpol.dbf C:\Data\LR_Coeff.dbf C:\Data\LR_pprb C:\Data\LR_std C:\Data\LR_conf Start Time: Fri Nov 15 02:33:26 2019 Running script LogisticRegression... Workspace: C:\Data Scratch workspace: C:\temp Prior Probability: 0.103672 Training Set: C:\Data\TP.shp Number of Training Sites: 107 Study Area Raster: C:\Data\mask Study Area Area (sq km): 1032.1034 Unit Cell Area (sq km): 1.0 Map Units: meter GP ERRORS: ERROR 010070: Raster read error. ERROR 010067: Error in executing grid expression. Failed to execute (Combine).

PYTHON ERRORS: Traceback Info: File "C:\SDM\Scripts\LogisticRegressionAll.py", line 97, in gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info: <class 'arcgisscripting.ExecuteError'>: ERROR 010070: Raster read error. ERROR 010067: Error in executing grid expression. Failed to execute (Combine).

Failed script LogisticRegression...

Traceback (most recent call last): File "C:\SDM\Scripts\LogisticRegressionAll.py", line 753, in raise 'Stopped on error.' TypeError: exceptions must be old-style classes or derived from BaseException, not str

Failed to execute (LogisticRegression). Failed at Fri Nov 15 02:34:09 2019 (Elapsed Time: 43.23 seconds)

Mahfoudb commented 5 years ago

This was the error message. I Have used Arsdm many times before with no problem. and when I bring the old data and run logistic regression with those old data it works.. I wonder if there is any restriction on the study area surface?

Mahfoudb commented 5 years ago

Executing: LogisticRegression pnt;rh;ocp;lth;frc;exp;elv c;c;c;c;c;c;c pnt_ct;rh_ct;ocp_ct;lth_ct;frc_ct;exp_ct;elv_ct TP -99 1 C:\Data\LR_logpol.dbf C:\Data\LR_Coeff.dbf C:\Data\LR_pprb C:\Data\LR_std C:\Data\LR_conf Start Time: Fri Nov 15 02:43:47 2019 Running script LogisticRegression... Workspace: C:\Data Scratch workspace: C:\Tmp Prior Probability: 0.103672 Training Set: C:\Data\TP.shp Number of Training Sites: 107 Study Area Raster: C:\Data\mask Study Area Area (sq km): 1032.1034 Unit Cell Area (sq km): 1.0 Map Units: meter Running C:\Tmp\sdmlr.bat: Logistic regression output file C:\Tmp\logpol.tba does not exist. Error in case.dat or param.dat. GP ERRORS:

PYTHON ERRORS: Traceback Info: File "C:\SDM\Scripts\LogisticRegressionAll.py", line 571, in raise Exception

Error Info: <type 'exceptions.Exception'>:

Failed script LogisticRegression...

Traceback (most recent call last): File "C:\SDM\Scripts\LogisticRegressionAll.py", line 753, in raise 'Stopped on error.' TypeError: exceptions must be old-style classes or derived from BaseException, not str

Failed to execute (LogisticRegression). Failed at Fri Nov 15 02:45:40 2019 (Elapsed Time: 1 minutes 53 seconds)

Mahfoudb commented 5 years ago

Sometimes it gives this 2nd error

trotor commented 5 years ago

There is no restriction that I know of, but the tools seemed to be picky sometimes. Can you do operations fine on them manually? Unfortunately I have no clue on the cause. Anyone else?

artolaiho commented 4 years ago

Hello, try to set workspace and scratch workspace to same folder. You can also test my fixes on arto-dev branch. See readme.

artolaiho commented 4 years ago

Load the newest version of ArcSDM (V5.02) and test again. If the problem persists, reopen this issue.