GlenRice-NOAA
commented
11 months ago Thanks for raising this. It is certainly not what we expected. We will see what we can discern.
Open aaime opened 11 months ago
GlenRice-NOAA
commented
11 months ago Thanks for raising this. It is certainly not what we expected. We will see what we can discern.
pgeleg
commented
11 months ago The file actually exists in the GeoPackage, the S3 link works, after manually downloading it and placing it in the right the VRT calculation managed to continue... wondering why the fetch did not notice? The rat file was already there, I only had to manually download the tiff file.
It seems like it did detect the tiles to download since the RAT exists. Maybe something failed in particular downloading the GeoTIFFs?
fetch_tiles does check if the expected files exist in their expected paths, and if not, it will attempt to download them again. If you encounter this again, I'd be curious if subsequent runs of fetch_tiles continues to fail downloading in case it was a one-off network or filesystem issue. Repeat runs of fetch_tiles should, hopefully, get you to a desired state until there is no new tiles or missing tiles.
I will try to recreate the issue or dig into it. Have failed to recreate it so far. Appreciate your time using and reporting of the issue.
In addition, we do intend to implement some more robustness into this process. It originally started as a quick scrape script and we've gradually added more functionality to it. File corruption may lead to read errors like you mentioned. Although there are also mentions in the GDAL mailing list of encountering this read error as a bug from the geotiff libraries, one which resolves itself. I know you said the file itself didn't exist though, so may be not applicable here. In the future, we may get the hash of the file and store it inside the tilescheme geopackage upon upload. So correctness against corruption can be confirmed after download. In addition there are issues where we may be updating and removing older tiles in the bucket while users are actively downloading them. So there are a few ideas we may be implementing to make this a bit more secure of a process, hopefully in the near future. Although I don't think thats what happened here from your reporting so will look into this.
Edit: I will confirm there was a BlueTopo delivery of 87 updated products on October 4th around 10 AM EST, which appears to be what your second run of fetch_tiles is retrieving.
pgeleg
commented
11 months ago Latest commit has a lot of new features for fetch_tiles #21 and hopefully helps with providing more clarity to users with the status of downloads
Please let us know if you encounter any issues @aaime
Thanks
aaime
commented
10 months ago The issue with tile fetching seems to be solved, and the fetch looks nicer and seems faster. The VRT generation failed right away though (I'm using the latest release from pip). Do you see anything wrong in the way I called the scripts?
(bluetopo_env) aaime@colossus ~/devel/gisData/noaa/bluetopo/bluetopo_1027 $ fetch_tiles -d /home/aaime/devel/gisData/noaa/bluetopo/bluetopo_1027/bluetopo_tiles -g ~/devel/gisData/noaa/bluetopo/bluetopo_1027/area.gpkg
[2023-10-27 19:06:25 CEST] BlueTopo: Beginning work in project folder: /home/aaime/devel/gisData/noaa/bluetopo/bluetopo_1027/bluetopo_tiles
[2023-10-27 19:06:28 CEST] BlueTopo: Downloaded BlueTopo_Tile_Scheme_20231027.gpkg
Tracking 3583 available BlueTopo tile(s) discovered in a total of 8253 intersected tile(s) with given polygon.
Resolving fetch list...
0 tile(s) with new data
0 tile(s) already downloaded are missing locally
___________________________________ SUMMARY ___________________________________
Existing:
Number of tiles already existing locally without updates: 3583
[2023-10-27 19:06:37 CEST] BlueTopo: Operation complete after 0:00:11.537686
(bluetopo_env) aaime@colossus ~/devel/gisData/noaa/bluetopo/bluetopo_1027 $ build_vrt -d /home/aaime/devel/gisData/noaa/bluetopo/bluetopo_1027/bluetopo_tiles
[2023-10-27 19:07:14 CEST] BlueTopo: Beginning work in project folder: /home/aaime/devel/gisData/noaa/bluetopo/bluetopo_1027/bluetopo_tiles
Building 197 subregion vrt(s). This may take minutes or hours depending on the amount of tiles.
Building BC26526L band 4m...
Traceback (most recent call last):
File "/home/aaime/miniconda3/envs/bluetopo_env/bin/build_vrt", line 8, in <module>
sys.exit(build_vrt_command())
^^^^^^^^^^^^^^^^^^^
File "/home/aaime/miniconda3/envs/bluetopo_env/lib/python3.11/site-packages/nbs/bluetopo/cli/cli.py", line 66, in build_vrt_command
build_vrt(
File "/home/aaime/miniconda3/envs/bluetopo_env/lib/python3.11/site-packages/nbs/bluetopo/core/build_vrt.py", line 801, in main
fields = build_sub_vrts(
^^^^^^^^^^^^^^^
File "/home/aaime/miniconda3/envs/bluetopo_env/lib/python3.11/site-packages/nbs/bluetopo/core/build_vrt.py", line 211, in build_sub_vrts
create_vrt(tiffs, res_vrt, [4, 8], relative_to_vrt)
File "/home/aaime/miniconda3/envs/bluetopo_env/lib/python3.11/site-packages/nbs/bluetopo/core/build_vrt.py", line 284, in create_vrt
vrt.BuildOverviews("NEAREST", levels)
File "/home/aaime/miniconda3/envs/bluetopo_env/lib/python3.11/site-packages/osgeo/gdal.py", line 3031, in BuildOverviews
return _gdal.Dataset_BuildOverviews(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: ../../BlueTopo/UTM17/BlueTopo_BH4TD56H_20221121.tiff: No such file or directoryThe file is there, but not in the location reported above. Compared to the location where I called the scripts, the file would be at ./bluetopo_tiles/BlueTopo/UTM17/BlueTopo_BH4TD56H_20221121.tiff.
Mind, the build_vrt file has been run from the same directory where the fetch_tiles was run, although, with absolute paths, I'm guessing is should not matter.
pgeleg
commented
10 months ago In this error, the path ../../BlueTopo/UTM17/BlueTopo_BH4TD56H_20221121.tiff would be relative to the subregion VRT that it's currently working on. It is using the relative reference inside of the VRT file to try to find the source tiff when its building overviews.
Which should be the correct relative path based on our directory structure. There is some referencing to issues with relative/absolute paths with VRTs in the GDAL change log for 3.5. I didn't look into it but I did try both 3.4.3 and 3.7.2 and did not run into any issues myself.
Tried replicating in a few different environments and was unable to so far. Will dig deeper into it whenever there is time. Thanks for the report.
pgeleg
commented
10 months ago @aaime
If it is still giving the same error, can you try build_vrt with the flag -r False
This will use absolute paths for source files. You can see the difference looking inside the VRT file.
Although it should work with relative sources anyhow.
aaime
commented
10 months ago Yes, I'm using 3.7.2 locally:
(bluetopo_env) aaime@colossus ~/devel/gisData/noaa $ gdalbuildvrt --version
GDAL 3.7.2, released 2023/09/05I noticed a single vrt file was created, and the .ovr is just a few KB, so I'm assuming if failed there.
However, if I get into the directory of the VRT, and run gdaladdo, it successfully builds the ovr file.
Which made me think, what if I run from the top level directory where I started the "build_vrt" command instead?
gdaladdo bluetopo_tiles/BlueTopo_VRT/BC26526L/BC26526L_4m.vrt 2 4 8 16
0ERROR 4: ../../BlueTopo/UTM17/BlueTopo_BH4TD56H_20221121.tiff: No such file or directory
...10...20...30...40...50...60...70...80...90...100 - done.
Overview building failed.Same error.. and it makes sense, because if I look into the VRT contents, I see:
<SourceFilename relativeToVRT="0">../../BlueTopo/UTM17/BlueTopo_BH4T955W_20231025.tiff</SourceFilename>"relativeToVRT" is 0. With a text editor I flipped all flags to 1, and then calling from another directory works.
I've also started the process with "-r False" and indeed it's working.
One last minor thing: it would be nice if the CLI programs could report a version number. I'm pretty sure I've installed the latest via pip, they have flags and checks that weren't there before, but would be good to be able to confirm.
pgeleg
commented
10 months ago "relativeToVRT" is 0. With a text editor I flipped all flags to 1, and then calling from another directory works.
We don't explicitly set relativeToVRT ourselves. We change the working directory and source paths, and GDAL does the rest based on a set of conditions listed here when creating the VRT https://lists.osgeo.org/pipermail/gdal-dev/2009-October/022358.html
Although what you've posted does not match any of the behaviors they've listed. The link is from 2009 so definitely could have changed. Not sure why this issue seems to be happening in your environment specifically though
In any case, relativeToVRT not being set correctly inside the VRT file seems to be the source of the problem. Will look into it and if I can't find anything, will just modify the file directly
I've also started the process with "-r False" and indeed it's working.
Thanks for checking
One last minor thing: it would be nice if the CLI programs could report a version number. I'm pretty sure I've installed the latest via pip, they have flags and checks that weren't there before, but would be good to be able to confirm.
When you originally created the issue, the pip install was based on the GitHub link. Since then, we have published to PyPI. This should simplify your upgrade process.
Inside the conda environment, I would recommend pip uninstall bluetopo to get rid of your GitHub repo installation from before if you still have it.
Then use the following to get the PyPI installation
pip install bluetopo
To upgrade to the latest version henceforth, you can just use
pip install bluetopo -U
To find the current version of your package, inside the conda environment, you can either use pip show bluetopo or conda list and then compare to the releases section on the GitHub page
But I'll report the package version in the output as well when I fix this issue
aaime
commented
10 months ago Thanks for the feedback. In fact I've never installed via the github: can't really work in Python, I have a checkout just to have a look at it, but never proceeded with an installation.
Good suggestion to use pip to check the version:
(bluetopo_env) aaime@colossus ~ $ pip show bluetopo
Name: BlueTopo
Version: 0.5.2Double checked with -U, seems to be the latest.
I've followed the instructions in the readme to fetch the CLI, and then use to to grab tiles and build VRTS.
Fetching appears to work fine:
Building VRTs eventually fails.. I'm guessing a difference between database and actual files:
The incriminated files indeed do not exist.
My hyphotesis was that the files has maybe been added overnight... Running again the fetch states:
The file actually exists in the GeoPackage, the S3 link works, after manually downloading it and placing it in the right the VRT calculation managed to continue... wondering why the fetch did not notice? The rat file was already there, I only had to manually download the tiff file.