search

m9brady / eodms-api-client

EODMS API Client for Python
https://m9brady.github.io/eodms-api-client/
MIT License
18 stars 5 forks source link

Download stops at size of 1KB #8

Closed jmarkloew closed 1 year ago

jmarkloew commented 1 year ago

I am running eodms from a conda shell in Win 10 with python 3.10 and after executing eodms -c RCM --order-id myid the download initiates as intended and all the ordered files are created, but their size never exceeds 1kb. Also there is no visible progress in the terminal. It is stuck like this:

2022-10-19 11:21:05 | eodmsapi.cli | INFO | Fast-downloading for 1 order 2022-10-19 11:21:05 | eodmsapi.main | INFO | Checking status of 1 order 2022-10-19 11:21:17 | eodmsapi.main | INFO | 50/50 items ready for download 2022-10-19 11:21:17 | eodmsapi.main | INFO | 0/50 items exist locally 2022-10-19 11:21:17 | eodmsapi.main | INFO | Downloading 50 remote items RCM1_OK1971291_PK2020755_1_SC50MC_20220316_170808_HH_HV_GRD.zip: 0%| | 121/619M [00:00<?, ?B/s] RCM3_OK1971291_PK2021980_1_SC50MD_20220317_051443_HH_HV_GRD.zip: 0%| | 121/156M [00:00<22:50, 119kB/s] RCM2_OK1971292_PK2017968_1_SC50MA_20220315_053123_HH_HV_GRD.zip: 0%| | 121/712M [00:00<?, ?B/s] RCM1_OK1971292_PK2020756_1_SC50MC_20220316_170808_HH_HV_GRD.zip: 0%| | 121/677M [00:00<?, ?B/s] RCM3_OK1971292_PK2028828_1_SC50MA_20220319_053137_HH_HV_GRD.zip: 0%| | 121/785M [00:00<?, ?B/s] RCM3_OK1971292_PK2021981_2_SC50MD_20220317_051443_HH_HV_GRD.zip: 0%| | 121/280M [00:00<?, ?B/s] RCM2_OK1799892_PK2018094_1_QP9_20220315_053153_HH_VV_HV_VH_SLC.zip: 0%| | 121/453M [00:00<?, ?B/s] RCM2_OK1971291_PK2035215_1_SC50MD_20220325_051432_HH_HV_GRD.zip: 0%| | 121/156M [00:00<?, ?B/s] RCM2_OK1955819_PK2052959_1_QP10_20220410_164420_HH_VV_HV_VH_SLC.zip: 0%| | 121/511M [00:00<?, ?B/s] RCM2_OK1971292_PK2027085_1_SC50MA_20220318_165213_HH_HV_GRD.zip: 0%| | 121/1.91G [00:00<?, ?B/s] RCM3_OK1971292_PK2021981_1_SC50MD_20220317_051443_HH_HV_GRD.zip: 0%| | 121/1.96G [00:00<?, ?B/s] RCM2_OK1971291_PK2027082_1_SC50MA_20220318_165213_HH_HV_GRD.zip: 0%| | 121/449M [00:00<?, ?B/s] RCM2_OK1971292_PK2035216_2_SC50MD_20220325_051432_HH_HV_GRD.zip: 0%| | 121/280M [00:00<?, ?B/s] RCM2_OK1971291_PK2029731_1_SC50MC_20220320_170819_HH_HV_GRD.zip: 0%| | 121/619M [00:00<?, ?B/s] RCM3_OK1799892_PK2031123_1_QP11_20220321_164431_HH_VV_HV_VH_SLC.zip: 0%| | 121/557M [00:00<?, ?B/s] RCM1_OK1799892_PK2031900_1_QP22_20220322_052341_HH_VV_HV_VH_SLC.zip: 0%|

There is no error message or an indication that the connection might be affected. Is this a server side issue or do I have to fix something locally?

Thanks and best regards

m9brady commented 1 year ago

Hi, thanks for the report.

It looks like the download halts and freezes at ~121 bytes per file? This might be indicative of a network issue/disruption between you and EODMS. Can you try manually downloading your order from the EODMS web browser interface on the same Win10 system? If that also fails then it could be more evidence of a network issue.

I will see if I can reproduce on my end.

m9brady commented 1 year ago

Hi @jmarkloew I was not able to reproduce on an order placed this morning.

2022-10-21 09:23:59 | eodmsapi.cli    | INFO     | Fast-downloading for 1 order
2022-10-21 09:23:59 | eodmsapi.main   | INFO     | Checking status of 1 order
2022-10-21 09:23:59 | eodmsapi.main   | INFO     | 1/1 items ready for download
2022-10-21 09:23:59 | eodmsapi.main   | INFO     | 0/1 items exist locally
2022-10-21 09:23:59 | eodmsapi.main   | INFO     | Downloading 1 remote item
RCM1_OK2048781_PK2307676_1_SC50MA_20221021_130156_VV_VH_GRD.zip: 100%|████████████| 1.28G/1.28G [00:15<00:00, 88.1MB/s]
2022-10-21 09:24:16 | eodmsapi.main   | INFO     | 1/1 items exist locally after latest download

However, I was able to reproduce on an old order. Something I noticed was that for old orders (e.g. more than 7 days old) the EODMS system shows the download links as "available" but the link is returning a html page of ~1KB instead of the zipped image. I think this is related to your issue so what I suggest is:

  1. Confirm that the affected order cannot be downloaded from the EODMS web interface
  2. Use the EODMS web interface to resubmit the affected order (this will generate a new OrderId)
  3. Download the images using the new OrderId eodms -c RCM --order-id <new_order_id> when the new order is finished processing

I do not have a good answer for you regarding why this is happening. Perhaps the amount of time that EODMS allows an order to stay "available for download" has been reduced but the order status remains "active" for longer. I will see about adding some logic in this client package to detect and better-handle this situation.

m9brady commented 1 year ago

@jmarkloew I released a new version (see v1.2.6) that better-informs the user what happened:

2022-10-21 11:31:15 | eodmsapi.cli    | INFO     | Fast-downloading for 1 order
2022-10-21 11:31:15 | eodmsapi.main   | INFO     | Checking status of 1 order
2022-10-21 11:31:15 | eodmsapi.main   | INFO     | 1/1 items ready for download
2022-10-21 11:31:15 | eodmsapi.main   | INFO     | 0/1 items exist locally
2022-10-21 11:31:15 | eodmsapi.main   | INFO     | Downloading 1 remote item
2022-10-21 11:31:16 | eodmsapi.main   | ERROR    | Remote filesize does not match manifest for item RCM2_OK2034581_PK2275100_2_SCLNB_20220928_010656_HH_HV_GRD.zip. You may have to resubmit your order.
2022-10-21 11:31:16 | eodmsapi.main   | INFO     | 0/1 items exist locally after latest download

Let me know if that helps/solves your issue. Unfortunately the best current solution appears to resubmit and use the new OrderId to access the data.

jmarkloew commented 1 year ago

Hi @m9brady, thanks for the fast response. I tried your suggested steps, but it does not work on my end. When I checked status of the order on https://www.eodms-sgdot.nrcan-rncan.gc.ca/index-en.html (after resubmitting) and it said available for download, I tried to download it manually. However, there was only a pop-up window that read Access denied (I tried Firefox and Edge if thats in any way relevant). Hence it seems there is a permission issue on my end.

m9brady commented 1 year ago

That is very strange. You successfully logged onto the EODMS web interface (https://www.eodms-sgdot.nrcan-rncan.gc.ca/index-en.html) but it still says Access Denied when trying to download?

You may want to try sending an email to EODMS support: eodms-sgdot@nrcan-rncan.gc.ca.

jmarkloew commented 1 year ago

Yes and I have a vetted user account for RCm data. Seems to be an issue for the EODMS support.

But thanks for all the advice