Esri / OptimizeRasters

OptimizeRasters is a set of tools for converting raster data to optimized Tiled TIF or MRF files, moving data to cloud storage, and creating Raster Proxies.
Apache License 2.0
127 stars 50 forks source link

Hangs on "Converting" when run from script edit - don't call gdal #171

Closed willhakai closed 3 months ago

willhakai commented 3 months ago

When I try to call OptimizeRasters from a script it hangs on "Converting". When I try from ArcGIS Pro Toolbox, it runs on the same folder correctly. I am on ArcGIS Pro 3.3.1 Any tricks on how to debug this?

#tool for creating RPAS products
#Requires ESRI OptimizeRasters
import sys
import os
from osgeo import gdal
import shutil
import arcpy

#LIST OF FILES TO OPTIMIZE
fList = [
   r'C:\Users\will.mcinnes\Desktop\Scratch\OptimizeTesting\Products\Kiixin_AGISOFT_MOS_U1230.tif'
]

OptimizeRastersRoot = 'C:/Image_Mgmt_Workflows/OptimizeRasters' # default intallation location for OptimizeRasters.
sys.path.append(OptimizeRastersRoot)    # Set the root folder where the OptimizeRasters module is found unless the client app is loaded from the same root.

try:
    import OptimizeRasters  # import OptimizeRasters/OR module.
except ImportError as e:
    print ('Err. {}'.format(str(e)))
    exit(1)

def messages(msg, status):
    print ('**{} - {}'.format(msg, status))
    return True

def callOptimize(inPath, outPath,f):
    args = {
        'config': 'C:/Image_Mgmt_Workflows/OptimizeRasters/Templates/Imagery_to_TIF_LZW.xml',
        'output': outPath.replace("\\","/")+"/",
        'input': inPath.replace("\\","/")+"/",
        'subs': 'false',    # Do we included subfolders?
        'resume':'True',
        'cloudupload': 'false',
        'op': 'report',
        'pyramids': 'true'
    }

    rpt = OptimizeRasters.Report(OptimizeRasters.Base())
    writeToPath = OptimizeRastersRoot
    print("writeToPath", writeToPath)

    ORJobFile = os.path.join(writeToPath,"logs", '{}{}'.format(rpt.getUniqueFileName(), rpt.CJOB_EXT))
    rpt.init(ORJobFile)
    for key in args.keys():
        rpt.addHeader(key, args[key])   # add necessary headers
    rpt.addFile(f)  # eg. c:/input/mydata/readme.txt
    print('added ', f)
    # please note, when adding files into the job file, it's important that all entries should have the same parent folder. In this case, it's (c:/input/mydata/)
    rpt.write()  # create the OR job/.orjob file.
    print('rpt:', rpt)
    args['input'] = ORJobFile      # input now points to the newly created OptimizeRasters Job file.
    app = OptimizeRasters.Application(args)  # The args{} can contain any valid cmd-line argument name without the prefix '-'
    app.registerMessageCallback(messages)   # Optional. If messages need to be brought back onto the caller's side.
    if (not app.init()):
        return False
    print('running')
    try:
        app.run()  # Do processing..
        rpt = app.getReport ()   # Get report/log status
    except ImportError as e:
        print ('Err. {}'.format(str(e)))
        exit(1)

    isSuccess = False
    if (rpt and
            not rpt.hasFailures()):  # If log has no failures, consider the processing as successful.
        isSuccess = True
    print ('Results> {}'.format(str(isSuccess)))
    return isSuccess

def ensure_path_exists(path):

    if not os.path.exists(path):
        os.mkdir(path)
        print(f"Created directory: {path}")
    else:
        print(f"Directory already exists: {path}")

for f in fList:
    prodDir = os.path.dirname(f)
    print("(proddir(inPath): ", prodDir)
    outPath = os.path.join(prodDir,"optimized").replace("\\","/")
    ensure_path_exists(outPath)
    print ("OutPath: ", outPath)
    print ("f: ", f)

    # RUN OPTIMIZATION!
    callOptimize(prodDir, outPath,f)

Output:

PS C:\Users\will.mcinnes\Documents\hakai-gis-tools> & "c:/Program Files/ArcGIS/Pro/bin/Python/envs/arcgispro-py3/python.exe" c:/Users/will.mcinnes/Documents/hakai-gis-tools/rpas-tools/CreateRPASProduct.py
(proddir(inPath):  C:\Users\will.mcinnes\Desktop\Scratch\OptimizeTesting\Products
Directory already exists: C:/Users/will.mcinnes/Desktop/Scratch/OptimizeTesting/Products/optimized
OutPath:  C:/Users/will.mcinnes/Desktop/Scratch/OptimizeTesting/Products/optimized
f:  C:\Users\will.mcinnes\Desktop\Scratch\OptimizeTesting\Products\Kiixin_AGISOFT_MOS_U1230.tif
writeToPath C:/Image_Mgmt_Workflows/OptimizeRasters
added  C:\Users\will.mcinnes\Desktop\Scratch\OptimizeTesting\Products\Kiixin_AGISOFT_MOS_U1230.tif
rpt: <OptimizeRasters.Report object at 0x0000018A9B81DD90>
**OptimizeRasters.py v2.0.8/20231006 - 0
**Convert raster formats to a valid output format through GDAL_Translate.

Please Note:
OptimizeRasters.py is entirely case-sensitive, extensions/paths in the config file are case-sensitive and the program will fail if the correct path/case is not entered at the cmd-line or in the config file.
 - 0
**ORJob> Reading/Preparing data. Please wait.. - 0
**ORJob> Done. - 0
Log-path set to (C:\Image_Mgmt_Workflows\OptimizeRasters\logs)
log-msg:version=v2.0.8/20231006
log-msg:-config=C:/Image_Mgmt_Workflows/OptimizeRasters/Templates/Imagery_to_TIF_LZW.xml -output=C:/Users/will.mcinnes/Desktop/Scratch/OptimizeTesting/Products/optimized/ -input=C:/Image_Mgmt_Workflows/OptimizeRasters\logs\OR_20240807T094202275517.orjob -subs=false -resume=True -cloudupload=false -op=report -pyramids=true
log-msg:Mode=tif
log-msg:RasterFormatFilter=['tif', 'tiff', 'TIF', 'TIFF', 'JPG', 'jpg', 'img', 'IMG', 'sid', 'SID']
log-msg:ExcludeFilter=['tmp', 'rrd', 'idx', 'lrc', 'mrf_cache', 'pjp', 'ppng', 'pft', 'pzp', 'ovr', 'aux.xml', 'aux', 'tfw', 'TFW', 'pjg']
log-msg:IncludeSubdirectories=true
log-msg:Compression=LZW
log-msg:Interleave=Pixel
log-msg:BuildPyramids=true
log-msg:PyramidSampling=average
log-msg:PyramidCompression=LZW
log-msg:BlockSize=512
log-msg:KeepExtension=false
log-msg:Threads=4
log-msg:CloudUpload=false
log-msg:Out_S3_DeleteAfterUpload=true
**Checking for updates.. - 0
**'NewVersion' - 0
running
**[Resume] Using job (C:/Image_Mgmt_Workflows/OptimizeRasters\logs\OR_20240807T094202275517.orjob) - 0
**Copying non rasters/aux files (C:/Users/will.mcinnes/Desktop/Scratch/OptimizeTesting/Products/=>C:/Users/will.mcinnes/Desktop/Scratch/OptimizeTesting/Products/optimized/).. - None
**Done. - None
**Converting.. - 0
**Converting (C:/Users/will.mcinnes/Desktop/Scratch/OptimizeTesting/Products/Kiixin_AGISOFT_MOS_U1230.tif).. - 0
willhakai commented 3 months ago

The problem was that I was importing GDAL from osgeo. when I remove that line it works correctly.