Improve featurecopy to deal with HTTP 504 errors

[palmerj]

Within the feature loops in the featureCopyIncremental and featureCopy DataStore methods it would be a good idea to trap HTTP 504 errors and potentially re-entry (be careful to roll-back the transaction first). Here's some code that generically traps OGR exceptions:

import ogr
import gdal

try:
    progress = gdal.TermProgress_nocb
except:
    progress = gdal.TermProgress

def print_exception(msg = None):
    errtype = {
        gdal.CE_None:'None',
        gdal.CE_Debug:'Debug',
        gdal.CE_Warning:'Warning',
        gdal.CE_Failure:'Failure',
        gdal.CE_Fatal:'Fatal'
    }
    err_msg = gdal.GetLastErrorMsg().replace('\n',' ')
    err_num = gdal.GetLastErrorNo()
    err_type = errtype.get(gdal.GetLastErrorType(), 'None')
    gdal.ErrorReset()

    print 'Error Number: %s' % (err_num)
    print 'Error Type: %s' % (err_type)
    print 'Error Message: %s' % (err_msg)
    return

# feature loop with exception handling
gdal.ErrorReset()
ogr.UseExceptions()

try:
    layer.ResetReading()
    feat = layer.GetNextFeature()
    new_layer.StartTransaction()
    while feat is not None:
        count += 1
        new_layer.CreateFeature(feat)
        feat = layer.GetNextFeature()
        progress( count / float(layer_count)  )
    feat = None
    new_layer.CommitTransaction()
except Exception, err:
    print "Exception: %s" % (str(err))
    print_exception()

ogr.DontUseExceptions()

[josephramsay]

Did something like that in the last commit but the problem now is OGRError: General Error. Another instance of enabling exception now catching errors that weren't otherwise a problem. And the 504's recur, at least with v:x772

[josephramsay]

Implemented, though gdal.GetLastErrorMsg() seems to be less reliable than reading RuntimeErrors for getting crash information. Also added some retry code initialising a new source (LDS) DS. Looks as if the root cause is 504 or at least some kind of server originating problem and isn't going to be easily addressed client side.