search

yannforget / landsatxplore

Search and download Landsat scenes from EarthExplorer.
MIT License
223 stars 98 forks source link

Download is not available #92

Closed HanDuwol closed 1 year ago

HanDuwol commented 1 year ago

I've had a similar experience before, but it seems to change every time, so I write the issue again. Are the others working normally? I think the value of DATA_PRODUCTS is different. If anyone has solved it, please answer me!

[INFO] 2023-01-05 11:23:14,164 > [DOWNLOADING] LC08_L1TP_115036_20221207_20221213_02_T1 (LC81150362022341LGN00) [CRITICAL] 2023-01-05 11:23:21,417 > Download is not available

WortalMombat commented 1 year ago

I checked and this issue affects LS8 and LS9 data after September 2022.

I presume, it is a similar issue to what was discussed and resolved #45 . The solution was to update the data product ids in the earthexplorer.py.

@fkroeber and @hiewliwen, do you remember where you got the information on the updated product id codes?

WortalMombat commented 1 year ago

I tried to update the code as proposed in #45 and it worked! Again, you have to change some lines in the earthexplorer.py:

If you have already implemented the solution from #45, at the beginning of the file add:

DATA_PRODUCTS_III = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "63231219fdd8c4e5",
    "landsat_etm_c2_l1": "5e83d0d08fec8a66",
    "landsat_ot_c2_l1": "5e81f14f92acf9ef",
    "landsat_tm_c2_l2": "632312ba6c0988ef",
    "landsat_etm_c2_l2": "632311068b0935a8",
    "landsat_ot_c2_l2": "632210d4770592cf",
    "sentinel_2a": "5e83a42c6eba8084",
}

Expand the try-except at the end of the file for another check. Nest it, so it can handle multiple errors of the same kind:

try:
            url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id)
            filename = self._download(url, output_dir, timeout=timeout, skip=skip)
        except:
            try:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_II[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)
            except:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_III[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)

You can get the updated codes from the test environment of the M2M API, for that you need to be logged in with your USGS credentials.

Select the Service Endpoint "dataset-download-options" and paste the desired "datasetName", e.g. landsat_ot_c2_l2. Submit, and the ID will show under productId in the response. Select the latest ID of the "Product Bundle".

MrKri03 commented 1 year ago

Thank you hiewliwen for the changes. I suggest that somebody adds the updates from hiewliwen to landsatxplore. I installed it two weeks ago with pip and the changes are not made in that one, so I had to dig into the updates to download Landsat 7, 8 and 9.

HanDuwol commented 1 year ago

Thank you hiewliwen for the changes. I suggest that somebody adds the updates from hiewliwen to landsatxplore. I installed it two weeks ago with pip and the changes are not made in that one, so I had to dig into the updates to download Landsat 7, 8 and 9.

I solved the problem.

In order to share the solution, it was summarized as follows.

ref : https://m2m.cr.usgs.gov/api/docs/json/#section-basicUsage

HanDuwol commented 1 year ago

I tried to update the code as proposed in #45 and it worked! Again, you have to change some lines in the earthexplorer.py:

If you have already implemented the solution from #45, at the beginning of the file add:

DATA_PRODUCTS_III = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "63231219fdd8c4e5",
    "landsat_etm_c2_l1": "5e83d0d08fec8a66",
    "landsat_ot_c2_l1": "5e81f14f92acf9ef",
    "landsat_tm_c2_l2": "632312ba6c0988ef",
    "landsat_etm_c2_l2": "632311068b0935a8",
    "landsat_ot_c2_l2": "632210d4770592cf",
    "sentinel_2a": "5e83a42c6eba8084",
}

Expand the try-except at the end of the file for another check. Nest it, so it can handle multiple errors of the same kind:

try:
            url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id)
            filename = self._download(url, output_dir, timeout=timeout, skip=skip)
        except:
            try:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_II[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)
            except:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_III[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)

You can get the updated codes from the test environment of the M2M API, for that you need to be logged in with your USGS credentials.

Select the Service Endpoint "dataset-download-options" and paste the desired "datasetName", e.g. landsat_ot_c2_l2. Submit, and the ID will show under productId in the response. Select the latest ID of the "Product Bundle".

We find it difficult to respond to anomalous USGS updates. It is necessary to utilize the API provided officially.

ChangpeiHe commented 1 year ago

i still cannnot solve this problem (btw: i operate it in Linux system)

ChangpeiHe commented 1 year ago

I tried to update the code as proposed in #45 and it worked! Again, you have to change some lines in the earthexplorer.py: If you have already implemented the solution from #45, at the beginning of the file add:

DATA_PRODUCTS_III = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "63231219fdd8c4e5",
    "landsat_etm_c2_l1": "5e83d0d08fec8a66",
    "landsat_ot_c2_l1": "5e81f14f92acf9ef",
    "landsat_tm_c2_l2": "632312ba6c0988ef",
    "landsat_etm_c2_l2": "632311068b0935a8",
    "landsat_ot_c2_l2": "632210d4770592cf",
    "sentinel_2a": "5e83a42c6eba8084",
}

Expand the try-except at the end of the file for another check. Nest it, so it can handle multiple errors of the same kind:

try:
            url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id)
            filename = self._download(url, output_dir, timeout=timeout, skip=skip)
        except:
            try:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_II[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)
            except:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_III[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)

You can get the updated codes from the test environment of the M2M API, for that you need to be logged in with your USGS credentials. Select the Service Endpoint "dataset-download-options" and paste the desired "datasetName", e.g. landsat_ot_c2_l2. Submit, and the ID will show under productId in the response. Select the latest ID of the "Product Bundle".

We find it difficult to respond to anomalous USGS updates. It is necessary to utilize the API provided officially.

are you operating it by anaconda3 environment in linux system?

HanDuwol commented 1 year ago

I tried to update the code as proposed in #45 and it worked! Again, you have to change some lines in the earthexplorer.py: If you have already implemented the solution from #45, at the beginning of the file add:

DATA_PRODUCTS_III = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "63231219fdd8c4e5",
    "landsat_etm_c2_l1": "5e83d0d08fec8a66",
    "landsat_ot_c2_l1": "5e81f14f92acf9ef",
    "landsat_tm_c2_l2": "632312ba6c0988ef",
    "landsat_etm_c2_l2": "632311068b0935a8",
    "landsat_ot_c2_l2": "632210d4770592cf",
    "sentinel_2a": "5e83a42c6eba8084",
}

Expand the try-except at the end of the file for another check. Nest it, so it can handle multiple errors of the same kind:

try:
            url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id)
            filename = self._download(url, output_dir, timeout=timeout, skip=skip)
        except:
            try:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_II[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)
            except:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_III[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)

You can get the updated codes from the test environment of the M2M API, for that you need to be logged in with your USGS credentials. Select the Service Endpoint "dataset-download-options" and paste the desired "datasetName", e.g. landsat_ot_c2_l2. Submit, and the ID will show under productId in the response. Select the latest ID of the "Product Bundle".

We find it difficult to respond to anomalous USGS updates. It is necessary to utilize the API provided officially.

are you operating it by anaconda3 environment in linux system?

I tested both Windows and Linux environments and it is still downloading without any problems.

Python's version is running at 3.9.7.

If you use the API provided by USGS to get the download url and send the download url to earthhexplorer, it will be downloaded normally.

Oopschange commented 1 year ago

I tried to update the code as proposed in #45 and it worked! Again, you have to change some lines in the earthexplorer.py:

If you have already implemented the solution from #45, at the beginning of the file add:

DATA_PRODUCTS_III = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "63231219fdd8c4e5",
    "landsat_etm_c2_l1": "5e83d0d08fec8a66",
    "landsat_ot_c2_l1": "5e81f14f92acf9ef",
    "landsat_tm_c2_l2": "632312ba6c0988ef",
    "landsat_etm_c2_l2": "632311068b0935a8",
    "landsat_ot_c2_l2": "632210d4770592cf",
    "sentinel_2a": "5e83a42c6eba8084",
}

Expand the try-except at the end of the file for another check. Nest it, so it can handle multiple errors of the same kind:

try:
            url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id)
            filename = self._download(url, output_dir, timeout=timeout, skip=skip)
        except:
            try:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_II[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)
            except:
                url = EE_DOWNLOAD_URL.format(data_product_id=DATA_PRODUCTS_III[dataset], entity_id=entity_id)
                filename = self._download(url, output_dir, timeout=timeout, skip=skip)

You can get the updated codes from the test environment of the M2M API, for that you need to be logged in with your USGS credentials.

Select the Service Endpoint "dataset-download-options" and paste the desired "datasetName", e.g. landsat_ot_c2_l2. Submit, and the ID will show under productId in the response. Select the latest ID of the "Product Bundle".

I tried and found that the id of landsat_ot_c2_l1 in DATA_PRODUCTS_III was still unavailable. Try to find the latest landsat_ot_c2_l1 id according to your method, but the "dataset-download-options" option is not found

Oopschange commented 1 year ago

Thank you hiewliwen for the changes. I suggest that somebody adds the updates from hiewliwen to landsatxplore. I installed it two weeks ago with pip and the changes are not made in that one, so I had to dig into the updates to download Landsat 7, 8 and 9.

I solved the problem.

In order to share the solution, it was summarized as follows.

ref : https://m2m.cr.usgs.gov/api/docs/json/#section-basicUsage

  • Step
  1. USGS Account API Permission Request Access (https://ers.cr.usgs.gov/profile/access)
  2. Utilize the official USGS API (https://m2m.cr.usgs.gov/api/docs/example/download_data-py) dataset-search -> scene-search -> download-options
sendRequest("dataset-search", payload, apiKey)
sendRequest("scene-search", scenes_payload, apiKey)
sendRequest("download-options", payload, apiKey)
  1. Some modifications to landsatxplore content exception scene entityId, displayId
try:

    downloadid = scene['entityId']
    #earthExplorer.download( downloadid, temppath )
    earthExplorer.download( identifier=downloadid, output_dir=temppath, datasetId=product['productId'], timeout=configs['time_out'] )
except Exception as e:
    downloadid = scene['displayId']
    earthExplorer.download( identifier=downloadid, output_dir=temppath, datasetId=product['productId'], timeout=configs['time_out'] )
finally:
   break

I tried and found that the id of landsat_ot_c2_l1 in DATA_PRODUCTS_III was still unavailable. How do I get the latest landsat_ot_c2_l1 id?