Identify and fix any inefficiencies in raster object and workspace use

[meganjennings]

I'm trying to run LP with expert corridor importance values for a very large project file. It fails because it appears Spatial Analyst calculations during the calculation of the blended priority are saving to my OS drive by default and filling it up before the run can finish. I've looked around in the lp_settings and lp_main scripts and can't seem to find a way to change this. I've also searched ESRI forums, to no avail.

Currently, it goes to: C:/Users/my_drive/AppData/Local/ESRI/Desktop10.2/SpatialAnalyst

Would setting KEEPINTERMEDIATE to True affect this by saving intermediate files elsewhere?

Any suggestions would be much appreciated.

[meganjennings]

Update: when I try to set a different scratch workspace using the Environments option, I get the following error after about 2 hrs run time:

Calculating Blended Priority (BP): -Clipping NLCC rasters to CWD threshold

ERROR 010429: Error in GRID IO: GetOpenTileChannel: Could not open tile. name: l:\megan_~2\wrentit\chfa_b~1\historic\chfa_2~4\datapass\nlcc\nlc2\120121, adepth: 32, type: 2, iomode: 3, version: 3, compression: 1, eval case: 1 . ERROR 010429: Error in GRID IO: GetOpenTileChannel: Could not open tile. name: l:\megan~2\wrentit\chfa_b~1\historic\chfa_2~4\datapass\nlcc\nlc2\120_121, adepth: 32, type: 2, iomode: 3, version: 3, compression: 1, eval case: 1 . ERROR 010429: Error in GRID IO: S_VXfer: Write error. ERROR 010296: Error in writing raster d:\mjtemp\extract_120_1. ERROR 010429: Error in GRID IO: S_VXfer: Write error. ERROR 010325: Run-time error - ***. 2- Another person or application is accessing this directory ERROR 010067: Error in executing grid expression. Failed to execute (ExtractByAttributes).

Traceback (most recent call last): File "C:\Users\Megan\Documents\linkage-mapper-2.1.0-beta.5\toolbox\scripts\lp_main.py", line 845, in main run_analysis() File "C:\Users\Megan\Documents\linkage-mapper-2.1.0-beta.5\toolbox\scripts\lp_main.py", line 786, in run_analysis calc_blended_priority(core_lyr, lcp_lines) File "C:\Users\Megan\Documents\linkage-mapper-2.1.0-beta.5\toolbox\scripts\lp_main.py", line 195, in calc_blended_priority nlcc_rast = clip_nlcc_to_threashold(lcp_list) File "C:\Users\Megan\Documents\linkage-mapper-2.1.0-beta.5\toolbox\scripts\lp_main.py", line 144, in clip_nlcc_to_threashold "VALUE <= {}".format(lm_env.CWDTHRESH)) File "c:\program files (x86)\arcgis\desktop10.2\arcpy\arcpy\sa\Functions.py", line 1151, in ExtractByAttributes where_clause) File "c:\program files (x86)\arcgis\desktop10.2\arcpy\arcpy\sa\Utils.py", line 47, in swapper result = wrapper(*args, **kwargs) File "c:\program files (x86)\arcgis\desktop10.2\arcpy\arcpy\sa\Functions.py", line 1147, in Wrapper out_raster)

Completed script LinkagePriority2... Failed to execute (LinkagePriority2).

[johngallo]

good idea setting the scratchworkspace somewher ele than the default (C: Drive).

THe first clue I see is this:

ERROR 010296: Error in writing raster d:\mjtemp\extract_120_1. ERROR 010429: Error in GRID IO: S_VXfer: Write error. ERROR 010325: Run-time error - ***. 2- Another person or application is accessing this directory

Maybe ArcMap is or was accessing that? Maybe reboot your computer to be safe on this troubleshoot approach? Just got anther call so need to bounce before addressing the KeepIntermedaiate = True More after I hear from you and get back from town.

[dkav]

Changing KeepIntermediate = True will have no impact on where ArcGIS Desktop places its temporary files.

Another option to direct temporary output (according to this answer on Stack Exchange) is to change the path of the Windows tmp directory.

[johngallo]

Keep Intermediate = True is a distraction, as you say. The issue is that she wants to have an option (or default) for ArcGIS placing temporary files on another drive besides C: . This may be part of ne of the new issues. If so, lets note which one and then re-close this.

[dkav]

The issue is that she wants to have an option (or default) for ArcGIS placing temporary files on another drive besides C: . This may be part of ne of the new issues. If so, lets note which one and then re-close this.

This is not part of the new issues.

On a broad level, I think that we should let ArcGIS manage as much of the temporary output as we can. Performance wise you want as much I/O to happen locally as possible. It also saves Linkage Mapper having to manage the extra files. If a user has disk capacity issues and has an alternative local drive, I would suggest they change their Windows environmental settings to match their needs. That said, there are three areas that I think could be explored to improve matters:

• We can save or delete runtime raster objects. The code would need to be audited to identify which objects are currently not saved or deleted in a timely manner. Irrespective of whether this resolves this issue, the audit could highlight inefficiencies. See ESRI's help page The interaction of the Raster object for more info.
• Add and set the ARCTMPDIR environment variable. You want insure that this isn't permanently set given that changes apply to the application. See Temporary files created by geoprocessing tools for more info.
• The use of current and scratch workspace environments could be audited. I am not confident that this will have much impact. See Using the current and scratch workspace environments for more info.

[johngallo]

I think that given all our other priorities, that we should go with your "broad level" comment, and mark this as a low priority, but keep it open. Also, I suggest we change the name of the issue to be more specific, but will leave that to you to decide if and what that should be. Thanks.

On Tue, Mar 30, 2021 at 11:58 PM Darren Kavanagh @.***> wrote:

The issue is that she wants to have an option (or default) for ArcGIS placing temporary files on another drive besides C: . This may be part of ne of the new issues. If so, lets note which one and then re-close this.

This is not part of the new issues.

On a broad level, it think that we should let ArcGIS manage as much of the temporary output as we can. Performance wise you want as much I/O to happen locally as possible. It also saves Linkage Mapper having to manage extra files. If the user has disk capacity issues and has an alternative local drive, I would suggest they change their Windows environmental settings to match their needs. That said, there are three areas that I think could be explored to improve matters:

• We can save or delete runtime raster objects. The code would need to be audited to identify which objects are currently not saved or deleted in a timely manner. Irrespective of whether this resolves this issue, the audit could highlight inefficiencies. See ESRI's help page The interaction of the Raster object https://desktop.arcgis.com/en/arcmap/latest/extensions/spatial-analyst/map-algebra/the-interaction-of-the-raster-object.htm for more info.
• Add and set the ARCTMPDIR environment variable. You want insure that this isn't permanently set given that changes apply to the application. See Temporary files created by geoprocessing tools https://desktop.arcgis.com/en/arcmap/latest/tools/supplement/temporary-files-created-by-geoprocessing-tools.htm for more info.
• The use of current and scratch workspace environments could be audited. I am not confident that this will have much impact. See Using the current and scratch workspace environments https://desktop.arcgis.com/en/arcmap/latest/analyze/executing-tools/using-the-current-and-scratch-workspace-environments.htm for more info.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/linkagescape/linkage-mapper/issues/81#issuecomment-810824160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7CJ23B7GDAAMK5GO3CGCDTGLB2ZANCNFSM4HNZQB2A .