Module removing complete zip files except one when present in download directory

[zkwurst]

This section in the code (lines 205-230)

tile_API_count = int(return_JSON['total'])
if tile_API_count > 0:
    dwnld_size = []
    dwnld_URL = []
    dataset_name = []
    tile_titles = []
    exist_zip_list = []
    exist_zip_count = len(exist_zip_list)
    for tile in return_JSON['items']:
        TNM_title = tile['title']
        TNM_tile_URL = str(tile['downloadURL'])
        TNM_tile_size = int(tile['sizeInBytes'])
        TNM_zip_name = TNM_tile_URL.split('/')[-1]
        pre_local_zip = os.path.join(work_dir, TNM_zip_name)
        zip_exists = os.path.exists(pre_local_zip)
        existing_LZ_size = os.path.getsize(pre_local_zip)
        if zip_exists:
            if existing_LZ_size != TNM_tile_size:
                cleanup_list.append(pre_local_zip)
                down_list()
            else:
                exist_zip_list.append(pre_local_zip)
        else:
            down_list()
elif tile_API_count == 0:
    gscript.fatal("Zero tiles available for given input parameters.")

tile_download_count = tile_API_count - exist_zip_count
if exist_zip_list:
    exist_msg = "{0} ZIP archive(s) exist locally and will be used by module.".format(len(exist_zip_list))
    gscript.verbose(exist_msg)
if cleanup_list:
    cleanup_msg = "{0} existing incomplete ZIP archive(s) detected and removed. Run module again.".format(len(cleanup_list))
    gscript.fatal(cleanup_msg)'

...is causing already downloaded and complete zip archives to get deleted except for the first file in the loop.

For example, with 4 complete zip archives in the download directory, running the module removes 3 of the 4, outputting this:

(Tue Jul 25 12:18:41 2017)
/home/zechariah/git/GSoC2017-GRASS-GIS/r.in.usgsned/r.in.usgsned.py --overwrite --verbose product=ned resolution=1/3 arc-second output_directory=/home/zechariah/Downloads/07-24_test output=out_rast_07-25 The default resampling method for product ned is bilinear Assuming current location as input Input parameters: '+proj=longlat +no_defs +a=6378137 +rf=298.257222101 +towgs84=0,0,0,0,0,0,0' Output parameters: '+proj=longlat +ellps=GRS80 +datum=NAD83 +nodefs' Assuming current location as input Input parameters: '+proj=longlat +no_defs +a=6378137 +rf=298.257222101 +towgs84=0,0,0,0,0,0,0' Output parameters: '+proj=longlat +ellps=GRS80 +datum=NAD83 +nodefs' TNM API Query URL: https://viewer.nationalmap.gov/tnmaccess/api/products?datasets=National+Elevation+Dataset+%28NED%29+1%2F3+arc-second&bbox=-79.12944451,35.29814813,-78.13398155,36.27185184&prodFormats=IMG 1 ZIP archive(s) exist locally and will be used by module. ERROR: 3 existing incomplete ZIP archive(s) detected and removed. Run module again. (Tue Jul 25 11:18:43 2017) Command finished (1 sec)

This part being what I can't figure out:

1 ZIP archive(s) exist locally and will be used by module. ERROR: 3 existing incomplete ZIP archive(s) detected and removed. Run module again.

Any thoughts?

[zkwurst]

It's something about this: if zip_exists: if existing_LZ_size != TNM_tile_size: cleanup_list.append(pre_local_zip) down_list()

Seems like the script is appending files to the cleanup_list either because the file size doesn't match or because the variable assignments are getting messed up in the loop?

Another thing that isn't getting updated properly is the tile_download_count variable. It always lists the same number as are available from the TNM API call...

This part of the script is also important because it's where any partial files left over from someone stopping the script mid-download would get removed.

I feel like the solution has to be something obvious and simple, but I can't figure it out.

[petrasovaa]

It seems that one of the zip files has different size than the API reports - just one byte difference. It's weird, but I would ignore it, you can solve it with:

size_diff_tolerance = 5
if abs(existing_LZ_size - TNM_tile_size) > tolerance: