search

AlexeyPechnikov / pygmtsar

PyGMTSAR (Python InSAR): Powerful and Accessible Satellite Interferometry
http://insar.dev/
BSD 3-Clause "New" or "Revised" License
389 stars 88 forks source link

Problem downloading ASF data #112

Open Smitrgeo15 opened 4 months ago

Smitrgeo15 commented 4 months ago

Set these variables to None and you will be prompted to enter your username and password below.

asf = ASF(asf_username, asf_password)

Optimized scene downloading from ASF - only the required subswaths and polarizations.

print(asf.download_scenes(DATADIR, SCENES, SUBSWATH))

There are two ways to download orbits; you can use any one or both together.

try:

RESORB orbit downloading from ASF has recently failed.

print(asf.download_orbits(DATADIR))

except Exception as e: print (e)

Download missed orbits in case ASF orbit downloading fails.

esa = ESA(esa_username, esa_password)
print (esa.download_orbits(DATADIR))
using this commands in PyGMTSAR i am unable to download the scenes

ERROR: ASF Downloading Sentinel-1 SLC:  33%  8/24 [00:12<03:10, 11.88s/it]

_RemoteTraceback Traceback (most recent call last) _RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/remotezip.py", line 226, in fetch res, range_header = self._request(kwargs) File "/usr/local/lib/python3.10/dist-packages/remotezip.py", line 192, in _request res.raise_for_status() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: NOT FOUND for url: https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20170324T011358_20170324T011425_015827_01A12B_940F%20%20%20%20.zip

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker r = call_item() File "/usr/local/lib/python3.10/dist-packages/joblib/externals/loky/process_executor.py", line 291, in call return self.fn(*self.args, self.kwargs) File "/usr/local/lib/python3.10/dist-packages/joblib/parallel.py", line 589, in call return [func(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/joblib/parallel.py", line 589, in return [func(args, kwargs) File "/usr/local/lib/python3.10/dist-packages/pygmtsar/ASF.py", line 105, in download_scene with asf_search.remotezip(url, session) as remotezip: File "/usr/local/lib/python3.10/dist-packages/asf_search/download/download.py", line 89, in remotezip return RemoteZip(url, session=session) File "/usr/local/lib/python3.10/dist-packages/remotezip.py", line 245, in init super(RemoteZip, self).init(rio) File "/usr/lib/python3.10/zipfile.py", line 1269, in init self._RealGetContents() File "/usr/lib/python3.10/zipfile.py", line 1332, in _RealGetContents endrec = _EndRecData(fp) File "/usr/lib/python3.10/zipfile.py", line 266, in _EndRecData fpin.seek(0, 2) File "/usr/local/lib/python3.10/dist-packages/remotezip.py", line 145, in seek self.buffer = self._fetch_fun((-size, None), stream=False) File "/usr/local/lib/python3.10/dist-packages/remotezip.py", line 230, in fetch raise RemoteIOError(str(e)) remotezip.RemoteIOError: 404 Client Error: NOT FOUND for url: https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20170324T011358_20170324T011425_015827_01A12B_940F%20%20%20%20.zip """

The above exception was the direct cause of the following exception:

RemoteIOError Traceback (most recent call last) in <cell line: 4>() 2 asf = ASF(asf_username, asf_password) 3 # Optimized scene downloading from ASF - only the required subswaths and polarizations. ----> 4 print(asf.download_scenes(DATADIR, SCENES, SUBSWATH)) 5 # There are two ways to download orbits; you can use any one or both together. 6 try:

6 frames /usr/local/lib/python3.10/dist-packages/joblib/parallel.py in _return_or_raise(self) 752 try: 753 if self.status == TASK_ERROR: --> 754 raise self._result 755 return self._result 756 finally:

RemoteIOError: 404 Client Error: NOT FOUND for url: https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20170324T011358_20170324T011425_015827_01A12B_940F%20%20%20%20.zip how to solve this problem

AlexeyPechnikov commented 4 months ago

The error message is clear:

NOT FOUND for url: https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20170324T011358_20170324T011425_015827_01A12B_940F%20%20%20%20.zip

The scene cannot be found because its identifier ends with a set of whitespaces ('...940F%20%20%20%20.zip'). You need to fix the scene names.

Smitrgeo15 commented 4 months ago

Screenshot 2024-03-16 112814 How to correct the scene ID link ? I have just copied the scene ID from ASF but i do not how to clear the whitespaces "NOT FOUND for url: https://datapool.asf.alaska.edu/SLC/SA/S1A_IW_SLC__1SDV_20170324T011358_20170324T011425_015827_01A12B_940F%20%20%20%20.zip"

AlexeyPechnikov commented 4 months ago

Check for any invisible whitespace characters (tiling) and remove them.

y-okur commented 1 month ago

Hi Alexey,

I have a problem with downloading ASF data.

I am trying to run one of your Google Colab Notebook: https://colab.research.google.com/drive/1TARVTB7z8goZyEVDRWyTAKJpyuqZxzW2?usp=sharing

At cell [16], I am receiving this error:

ASF Downloading Sentinel-1 SLC::  67%  8/12 [00:18<01:14, 18.61s/it]

_RemoteTraceback Traceback (most recent call last) _RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker r = call_item() File "/usr/local/lib/python3.10/dist-packages/joblib/externals/loky/process_executor.py", line 291, in call return self.fn(*self.args, self.kwargs) File "/usr/local/lib/python3.10/dist-packages/joblib/parallel.py", line 598, in call return [func(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/joblib/parallel.py", line 598, in return [func(args, kwargs) File "/usr/local/lib/python3.10/dist-packages/pygmtsar/ASF.py", line 148, in download_scene assert os.path.getsize(fullname + '.tmp') == filesize, \ AssertionError: ERROR: Downloaded incomplete scene content """

The above exception was the direct cause of the following exception:

AssertionError Traceback (most recent call last) in <cell line: 4>() 2 asf = ASF(asf_username, asf_password) 3 # Optimized scene downloading from ASF - only the required subswaths and polarizations. ----> 4 print(asf.download_scenes(DATADIR, SCENES, SUBSWATH))

6 frames /usr/local/lib/python3.10/dist-packages/joblib/parallel.py in _return_or_raise(self) 761 try: 762 if self.status == TASK_ERROR: --> 763 raise self._result 764 return self._result 765 finally:

AssertionError: ERROR: Downloaded incomplete scene content

I am also receiving the same error when I try download different scenes with SUBSWATH=123 option.

Do you have any idea what might be the problem?

Thanks in advance

AlexeyPechnikov commented 1 month ago

The error is pretty clear:

ERROR: Downloaded incomplete scene content

You suffer from a network error or issue at the ASF datastore. Run the cell again or relaunch the entire notebook to continue the downloading.

y-okur commented 1 month ago

The error is pretty clear:

ERROR: Downloaded incomplete scene content

You suffer from a network error or issue at the ASF datastore. Run the cell again or relaunch the entire notebook to continue the downloading.

Hi Alexey,

Thank you for the quick response.

I tried downloading one swath at a time using:

print(asf.download_scenes(DATADIR, SCENES, 1))
print(asf.download_scenes(DATADIR, SCENES, 2))
print(asf.download_scenes(DATADIR, SCENES, 3))

and the problem is resolved now.

AlexeyPechnikov commented 1 month ago

Nice. FYI, this is not related to the number of swaths but only a temporary downloading error. PyGMTSAR example notebooks can produce such errors once or even for hours when an unpredictable network connection issue arises or the ASF datastore is in a maintenance state.