Open julietcohen opened 1 year ago
Made branch feature-raster-retry
(from develop
branch) to test modification to RasterTiler.py
to integrate 3 tries total for staged files that fail to rasterize. The retries simply apply the same rasterization approach, and includes log statements when retries are applied. The syntax & execution for this is show in the following example:
# define function to exemplify retry approach
def test_func(x, y):
for attempt in range(2):
try:
z = x / y
print(f'Success: {x} / {y} = {z}')
return z
except Exception as e:
print(f'Error dividing {x} by {y} so trying again.')
else:
break
else:
print(f'Error dividing {x} by {y}, ran out of retries.')
return None
Apply function with expected input types that won't error:
test_func(x = 9, y = 3)
output:
Success: 9 / 3 = 3.0
3.0
Apply function with unexpected input types that will error:
test_func(x = 'elephant', y = 'cat')
Output:
Error dividing elephant by cat so trying again.
Error dividing elephant by cat so trying again.
Error dividing elephant by cat, ran out of retries.
This retry feature is applied to the rasterize_vector()
function.
We apply this with the virtual env rasterRetry
with local develop
branch of viz-staging, local feature-raster-retry
branch of viz-workflow
, sqlite3
, and parsl
.
Data sample is 3000 randomly sampled polygons from lake size change dataset (1000 from each UTM zone). In order to ensure the retry code is run, we purposefully corrupt one of the GeoPackage files by using sqlite3
to delete one of the mandatory tables after the file is staged. See this in the script here.
Executed a run with the same environment and config as above, but inserted variable e
into log statement that is printed for the initial tries. e
cannot be printed in the final attempt's error message printed by __end_tracking()
because of the break statement just before.
I'm curious if there is a benefit to printing the explicit e
in the __end_tracking()
rather than a general log statement. If there is a benefit to that, this change to rasterize_vector()
should be adjusted further to enable printing e
after the final try. If printing e
in the initial tries' log statements is equally as helpful to printing it in the final __end_tracking()
message, then I believe this code is good to go!
For this run, which staged 3000 polygons, then we corrupted 1 before rasterization started: files in staged dir | files in geotiff dir | files in web tiles dir |
---|---|---|
2964 | 6496 | 12992 |
No errors were reported in log.log
besides the two initial tries for the purposefully corrupted file, and the error for the final try:
For the run with original develop branch RasterTiler.py script without retries, with the same starting 3000 polygons, and none are corrupted:
files in staged dir |
files in geotiff dir | files in web tiles dir |
---|---|---|
2964 | 6498 | 12996 |
No errors reported in that log.log
at all. This means that corrupting only that one staged file resulted in 2 fewer rasters and 4 fewer web tiles.
Just tested this with the same everything except reduced the amount of retries to 1. I figure this is better because with large datasets such as IWP, even with only ~0.5% of the files initially failing, reducing the amount of retries from 2 to 1 will save time and Delta credits. We don't want to waste resources retrying the same file an unnecessary amount of times.
I would ideally like to test this workflow on a larger amount of data now that the syntax and number of tries has shown to be successful. I could use all the lake change files Ingmar uploaded to the PDG google drive. I am not sure if Robyn already processed them, but I believe she has not. It would be interesting to run all those files on the current develop
branch (without the retry) and see how many files fail to rasterize, then process the same files with this branch and see how many more were rasterized. That would allow for a deeper check for this update without using any credits on Delta.
Rasterization step logs errors for relatively few staged files compared to the total amount of staged files. Occurs for both the maximum z-level as well as parent z-levels according to
log.log
. For the lake size change data sample, the GeoPackage files that errored during a certain parsl run are not obviously corrupt, and when rasterization is applied again (outside of the workflow but still with parallelization) the files are successfully rasterized. See this documented here. This was also seen in the ray workflow with IWP dataset, for which Robyn noted:It would be helpful to do more runs with various datasets to determine if the failing files are ever consistent, and if the errors are random (and can therefore be solved by just trying to rasterize these few staged files again with the same approach), or if the files are actually corrupt, or if it has to do with the files not transferring to scratch dir like Robyn suggested.