Move down credentials check to be able to use netrc properly

[anton-petrov]

Hotfix for previous PR.

upd: ready for merge.

[veroandreo]

Thanks for your quick responses @anton-petrov ! It still yields the same error.... no idea

I re-installed your fork as indicated: pip install git+https://github.com/anton-petrov/pyModis.git and tested the same command again, but I get exactly the same as before:

modis_download.py -s MOTA -p MCD12Q1.061 -t h13v11 -f 2003-01-01 -e 2001-01-01 .
Traceback (most recent call last):
  File "/home/veroandreo/.local/bin/modis_download.py", line 167, in <module>
    main()
  File "/home/veroandreo/.local/bin/modis_download.py", line 147, in main
    modisOgg = downmodis.downModis(url=options.url, 
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 301, in __init__
    raise IOError("You must provide either a token or a user and password")
OSError: You must provide either a token or a user and password

Does the command work for you with the .netrc as you suggested? Seems to me there's something to fix in modis_download.py too, no? It marks line 147

[anton-petrov]

@veroandreo Can you show me your netrc file? Without passwords, ofcourse

[veroandreo]

@veroandreo Can you show me your netrc file? Without passwords, ofcourse

machine urs.earthdata.nasa.gov 
login token 
password My-token-copied-from-earthdata

These are the permissions of the file as in https://github.com/lucadelu/pyModis/issues/137#issuecomment-861836141 -rw-------. 1 veroandreo veroandreo 628 Aug 7 13:40 .netrc

I tried the .netrc like shown above, and also all in the same line. But there's no difference, same error. Should I add anything else ?

[anton-petrov]

@veroandreo Sorry, there was another bug. Right now I tested on my machine using .netrc file:

machine urs.earthdata.nasa.gov login token password eyJ0eXAiOiJK*************

Command modis_download.py -s MOTA -p MCD12Q1.061 -t h13v11 -f 2003-01-01 -e 2001-01-01 . completed successfully. All works as expected!

[anton-petrov]

Also, you need to set correct file permissions for your .netrc file:

chmod 600 ~/.netrc

[anton-petrov]

@lucadelu Now PR ready to merge :)

[veroandreo]

I re-installed from your fork, but now I get a much longer error (.netrc permissions were already fine before):

modis_download.py -s MOTA -p MCD12Q1.061 -t h13v11 -f 2003-01-01 -e 2001-01-01 .
Traceback (most recent call last):
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 714, in _downloadFileHTTP
    http = urllib.request.urlopen(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib64/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 724, in _downloadFileHTTP
    orig_size = http.headers['Content-Length']
                ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
           ~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-length'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/veroandreo/.local/bin/modis_download.py", line 167, in <module>
    main()
  File "/home/veroandreo/.local/bin/modis_download.py", line 161, in main
    modisOgg.downloadsAllDay(clean=options.empty, allDays=options.alldays)
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 871, in downloadsAllDay
    self._downloadAllDaysHTTP(days)
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 887, in _downloadAllDaysHTTP
    self.dayDownload(day, listFilesDown)
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 849, in dayDownload
    self.downloadFile(i, file_hdf, day)
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 698, in downloadFile
    self._downloadFileHTTP(filDown, filHdf, day)
  File "/home/veroandreo/.local/lib/python3.11/site-packages/pymodis/downmodis.py", line 728, in _downloadFileHTTP
    "error {co}, reason {re}".format(co=e.code,
                                        ^^^^^^
AttributeError: 'KeyError' object has no attribute 'code'

and downloads are 0 bytes

The log says unauthorized... modisMCD12Q1.061.log

Token should be valid:

[veroandreo]

I generated a new token, now it works :+1: Thanks for your help and patience!!!