mdales
commented
1 month ago This is a bug/regression introduced by me in commit 76aeac029b8d569f94e3498225a49558fdd954d9, where I switched us to generating files in a temporary directory and only moving them to the final location if the operation was successful.
Unfortunately the old code was written before GDAL had a Close() method on datasets, and we didn't explicitly delete the object back then as it was the final operation. This meant when I added the move to the code, it was before GDAL had released all the data from memory, so sometimes we'd hit an issue whereby we'd move the file before GDAL had flushed it. If both /tmp and the final destination were on the same disk then I suspect we'd survive this, but they aren't for the common case on our servers.
Yirgacheffe has since been updated to work with the with style of resource management in Python, so to fix this problem I updated all of generate slopes to work with that style. I'll make more tests and then push a PR with the fix.
I'm afraid @abbyevewilliams and @epingchris you might need to review your files since the fix for slopes went in. Old data would have been fine, and sometimes new data would have, but not always.
When I checked the k.parquet and pairs for this project, I saw that the slope was all 0s, even though the generate slopes and rescale slopes stages seem to have worked OK (output below). I've re-run this a couple of times but to no avail.
Shapefile is available at
/maps/aew85/projects/mbelling.geojson. The project is in Indonesia.aew85@sherwood:~/tmf-implementation$ proj='mbelling' aew85@sherwood:~/tmf-implementation$ t0=2022 aew85@sherwood:~/tmf-implementation$ eval_year=2022 aew85@sherwood:~/tmf-implementation$ input_dir='/maps/aew85/projects' aew85@sherwood:~/tmf-implementation$ output_dir='/maps/aew85/tmf_pipe_out' aew85@sherwood:~/tmf-implementation$ tmfpython3 -m methods.inputs.generate_slope --input "${output_dir}/srtm/tif" --output "${output_dir}/slopes" 2024-08-20 10:07:16,473 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_41_12.tif already exists, skipping. 2024-08-20 10:07:16,474 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_24_13.tif already exists, skipping. 2024-08-20 10:07:16,474 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_24_14.tif already exists, skipping. 2024-08-20 10:07:16,474 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_35_11.tif already exists, skipping. 2024-08-20 10:07:16,475 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_69_14.tif already exists, skipping. 2024-08-20 10:07:16,475 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_42_11.tif already exists, skipping. 2024-08-20 10:07:16,475 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_26_14.tif already exists, skipping. 2024-08-20 10:07:16,476 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_26_13.tif already exists, skipping. 2024-08-20 10:07:16,476 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_43_12.tif already exists, skipping. 2024-08-20 10:07:16,476 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_22_15.tif already exists, skipping. 2024-08-20 10:07:16,476 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_40_11.tif already exists, skipping. 2024-08-20 10:07:16,477 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_60_14.tif already exists, skipping. 2024-08-20 10:07:16,477 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_69_15.tif already exists, skipping. 2024-08-20 10:07:16,477 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_24_15.tif already exists, skipping. 2024-08-20 10:07:16,477 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_35_10.tif already exists, skipping. 2024-08-20 10:07:16,477 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_41_13.tif already exists, skipping. 2024-08-20 10:07:16,478 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_22_13.tif already exists, skipping. 2024-08-20 10:07:16,478 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_22_14.tif already exists, skipping. 2024-08-20 10:07:16,478 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_62_14.tif already exists, skipping. 2024-08-20 10:07:16,478 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_43_13.tif already exists, skipping. 2024-08-20 10:07:16,478 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_26_15.tif already exists, skipping. 2024-08-20 10:07:16,479 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_34_10.tif already exists, skipping. 2024-08-20 10:07:16,479 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_25_15.tif already exists, skipping. 2024-08-20 10:07:16,479 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_40_13.tif already exists, skipping. 2024-08-20 10:07:16,479 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_68_15.tif already exists, skipping. 2024-08-20 10:07:16,479 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_21_14.tif already exists, skipping. 2024-08-20 10:07:16,480 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_61_14.tif already exists, skipping. 2024-08-20 10:07:16,480 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_21_13.tif already exists, skipping. 2024-08-20 10:07:16,480 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_42_13.tif already exists, skipping. 2024-08-20 10:07:16,480 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_23_13.tif already exists, skipping. 2024-08-20 10:07:16,481 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_23_14.tif already exists, skipping. 2024-08-20 10:07:16,481 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_43_11.tif already exists, skipping. 2024-08-20 10:07:16,481 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_21_15.tif already exists, skipping. 2024-08-20 10:07:16,481 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_68_14.tif already exists, skipping. 2024-08-20 10:07:16,481 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_25_13.tif already exists, skipping. 2024-08-20 10:07:16,482 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_40_12.tif already exists, skipping. 2024-08-20 10:07:16,482 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_34_11.tif already exists, skipping. 2024-08-20 10:07:16,482 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_25_14.tif already exists, skipping. 2024-08-20 10:07:16,482 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_35_12.tif already exists, skipping. 2024-08-20 10:07:16,483 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_41_11.tif already exists, skipping. 2024-08-20 10:07:16,483 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_23_15.tif already exists, skipping. 2024-08-20 10:07:16,483 [INFO] /maps/aew85/tmf_pipe_out/slopes/slope-srtm_42_12.tif already exists, skipping. aew85@sherwood:~/tmf-implementation$ tmfpython3 -m methods.inputs.rescale_tiles_to_jrc --jrc /maps/forecol/data/JRC/v1_2022/AnnualChange/tifs \ --tiles "${output_dir}/srtm/tif" \ --output "${output_dir}/rescaled-elevation" aew85@sherwood:~/tmf-implementation$ tmfpython3 -m methods.inputs.rescale_tiles_to_jrc \ --jrc /maps/forecol/data/JRC/v1_2022/AnnualChange/tifs \ --tiles "${output_dir}/slopes" \ --output "${output_dir}/rescaled-slopes" echo "--JRC rescaled.--" --JRC rescaled.--Here is a sample of one of the pairs: