search

ACCESS-Cloud-Based-InSAR / dem-stitcher

Download and merge DEM tiles
Apache License 2.0
39 stars 15 forks source link

Failed to connect to download.agisoft.com port 80 after 131945 ms: #88

Closed abradley60 closed 1 week ago

abradley60 commented 1 week ago

The bug

Issues downloading DEMs. My workflow had worked previously for me but I am now running into an issues with the stitch_dem function timing out.

To Reproduce

python=3.10 dem_stitcher==2.5.5 rasterio==1.3.10

dem_data, dem_meta = stitch_dem((106.021858, -68.843872, 119.033562, -63.540306), dem_name='glo_30', dst_ellipsoidal_height=True, dst_area_or_point='Point' )

Additional context

`2024-06-27 05:39:12 INFO Scene bounds : (106.021858, -68.843872, 119.033562, -63.540306) 2024-06-27 05:39:12 INFO Downloding DEM for bounds : (106.021858, -68.843872, 119.033562, -63.540306) 2024-06-27 05:39:12 INFO type of DEM being downloaded : glo_30 2024-06-27 05:39:14 INFO Found credentials in environment variables. 2024-06-27 05:39:14 INFO Found credentials in environment variables. 2024-06-27 05:39:14 INFO Found credentials in environment variables. 2024-06-27 05:39:14 INFO Found credentials in environment variables. 2024-06-27 05:39:14 INFO Found credentials in environment variables. Reading glo_30 Datasets: 2%|██▏ | 1/46 [00:13<10:26, 13.92s/it]2024-06-27 05:39:28 INFO Found credentials in environment variables. 2024-06-27 05:39:28 INFO Found credentials in environment variables. 2024-06-27 05:39:28 INFO Found credentials in environment variables. 2024-06-27 05:39:28 INFO Found credentials in environment variables. 2024-06-27 05:39:28 INFO Found credentials in environment variables. Reading glo_30 Datasets: 13%|█████████████▎ | 6/46 [00:24<02:23, 3.59s/it]2024-06-27 05:39:42 INFO Found credentials in environment variables. 2024-06-27 05:39:42 INFO Found credentials in environment variables. Reading glo_30 Datasets: 15%|███████████████▌ | 7/46 [00:27<02:14, 3.45s/it]2024-06-27 05:39:42 INFO Found credentials in environment variables. 2024-06-27 05:39:42 INFO Found credentials in environment variables. 2024-06-27 05:39:42 INFO Found credentials in environment variables. 2024-06-27 05:39:55 INFO Found credentials in environment variables. Reading glo_30 Datasets: 24%|████████████████████████▏ | 11/46 [00:41<02:00, 3.44s/it]2024-06-27 05:39:55 INFO Found credentials in environment variables. 2024-06-27 05:39:55 INFO Found credentials in environment variables. 2024-06-27 05:39:55 INFO Found credentials in environment variables. 2024-06-27 05:39:55 INFO Found credentials in environment variables. Reading glo_30 Datasets: 35%|███████████████████████████████████▏ | 16/46 [00:54<01:32, 3.07s/it]2024-06-27 05:40:09 INFO Found credentials in environment variables. 2024-06-27 05:40:09 INFO Found credentials in environment variables. 2024-06-27 05:40:09 INFO Found credentials in environment variables. 2024-06-27 05:40:09 INFO Found credentials in environment variables. 2024-06-27 05:40:09 INFO Found credentials in environment variables. Reading glo_30 Datasets: 46%|██████████████████████████████████████████████ | 21/46 [01:05<01:06, 2.66s/it]2024-06-27 05:40:22 INFO Found credentials in environment variables. 2024-06-27 05:40:22 INFO Found credentials in environment variables. 2024-06-27 05:40:22 INFO Found credentials in environment variables. Reading glo_30 Datasets: 52%|████████████████████████████████████████████████████▋ | 24/46 [01:07<00:48, 2.22s/it]2024-06-27 05:40:22 INFO Found credentials in environment variables. 2024-06-27 05:40:22 INFO Found credentials in environment variables. 2024-06-27 05:40:36 INFO Found credentials in environment variables. 2024-06-27 05:40:36 INFO Found credentials in environment variables. 2024-06-27 05:40:36 INFO Found credentials in environment variables. Reading glo_30 Datasets: 57%|█████████████████████████████████████████████████████████ | 26/46 [01:21<01:02, 3.14s/it]2024-06-27 05:40:36 INFO Found credentials in environment variables. 2024-06-27 05:40:36 INFO Found credentials in environment variables. 2024-06-27 05:40:44 INFO Found credentials in environment variables. 2024-06-27 05:40:49 INFO Found credentials in environment variables. Reading glo_30 Datasets: 67%|████████████████████████████████████████████████████████████████████ | 31/46 [01:37<00:47, 3.15s/it]2024-06-27 05:40:54 INFO Found credentials in environment variables. Reading glo_30 Datasets: 70%|██████████████████████████████████████████████████████████████████████▎ | 32/46 [01:39<00:42, 3.06s/it]2024-06-27 05:40:54 INFO Found credentials in environment variables. 2024-06-27 05:40:54 INFO Found credentials in environment variables. 2024-06-27 05:40:54 INFO Found credentials in environment variables. 2024-06-27 05:40:54 INFO Found credentials in environment variables. 2024-06-27 05:41:07 INFO Found credentials in environment variables. Reading glo_30 Datasets: 83%|███████████████████████████████████████████████████████████████████████████████████▍ | 38/46 [01:52<00:21, 2.65s/it]2024-06-27 05:41:07 INFO Found credentials in environment variables. 2024-06-27 05:41:07 INFO Found credentials in environment variables. 2024-06-27 05:41:07 INFO Found credentials in environment variables. Reading glo_30 Datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 46/46 [02:03<00:00, 2.68s/it] 2024-06-27 05:48:09 INFO GDAL signalled an error: err_no=11, msg="CURL error: Failed to connect to download.agisoft.com port 80 after 131945 ms: Couldn't connect to server" Traceback (most recent call last): File "rasterio/_base.pyx", line 310, in rasterio._base.DatasetBase.init File "rasterio/_base.pyx", line 221, in rasterio._base.open_dataset File "rasterio/_err.pyx", line 221, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_HttpResponseError: CURL error: Failed to connect to download.agisoft.com port 80 after 131945 ms: Couldn't connect to server

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "rtc_otf.py", line 208, in dem_data, dem_meta = stitch_dem(scene_bounds_buf, File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/stitcher.py", line 367, in stitch_dem dem_arr, dem_profile = merge_and_transform_dem_tiles(datasets, File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/stitcher.py", line 204, in merge_and_transform_dem_tiles dem_arr = remove_geoid(dem_arr, File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/geoid.py", line 83, in remove_geoid geoid_arr, geoid_profile = read_geoid(geoid_name, File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/geoid.py", line 41, in read_geoid geoid_arr, geoid_profile = read_raster_from_window(geoid_path, File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/dem_stitcher/rio_window.py", line 113, in read_raster_from_window with rasterio.open(raster_path) as ds: File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/rasterio/env.py", line 451, in wrapper return f(*args, kwds) File "/opt/conda/envs/nrb_env/lib/python3.8/site-packages/rasterio/init.py", line 304, in open dataset = DatasetReader(path, driver=driver, sharing=sharing, kwargs) File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.init rasterio.errors.RasterioIOError: CURL error: Failed to connect to download.agisoft.com port 80 after 131945 ms: Couldn't connect to server

jlmaurer commented 1 week ago

@cmarshak I'm' seeing the same error in RAiDER, just started in the last couple of days.

cmarshak commented 1 week ago

It looks like the geoid link has been taken down from AGIsoft. We will have to put it somewhere else. Have to do this Monday as I am away from computer. It's an easy fix.

abradley60 commented 1 week ago

Great, thanks team!

asjohnston-asf commented 1 week ago

Looking at https://www.agisoft.com/downloads/geoids/, the download URLs suggest http://download.agisoft.com/ has migrated to https://s3-eu-west-1.amazonaws.com/download.agisoft.com/ .

I suspect the two files to point geoid.py at are:

asjohnston-asf commented 1 week ago

While not linked on https://www.agisoft.com/downloads/geoids/ , files with the existing path/filenames are available at:

I'm not familiar enough with the content to confirm whether they're the exact same files, however.

jhkennedy commented 1 week ago

@abradley60 @jlmaurer This should now be fixed in dem-sticher v2.5.6, which is available on PyPI and coming soon to conda-forge.

abradley60 commented 1 week ago

@jhkennedy thanks, can confirm it's working as expected on my end.

cmarshak commented 1 week ago

Thank you, Joe and Andrew!