search

jgomezdans / get_modis

Downloading MODIS data from the USGS repository
GNU General Public License v3.0
62 stars 41 forks source link

Python 3.x related changes #4

Open atedstone opened 8 years ago

atedstone commented 8 years ago

Hi Jose

Having used the script for the last few days I've come across a couple more oddities related to Python 3. There are two commits (plus some merging as I didn't have your remote linked to the repo so didn't catch your latest changes) - one deals with handling timeout errors that I've been having, the other deals with the different method of handling content-length in Python 3.

I had a quick go on Py 2.7, seems ok.

cheers Andrew

jgomezdans commented 7 years ago

Hi Andrew,

Many thanks for this. Will try and test tomorrow to ensure it still works on 2.7 (yes, yes, yes, I need to move on...) ;-)

Cheers!

atedstone commented 7 years ago

Ah didn't realise it added the latest commits from my fork to the pull request. I forgot to pull your upstream version before making more edits, then got into all sorts of problems, hence the revert to your version in 46c89bc. So at the moment the script is only working with Py27. I'm going to re-implement the Py35 stuff now so wait for that before doing anything yourself...

atedstone commented 7 years ago

As per commit - see http://docs.python-requests.org/en/master/user/quickstart/#response-content for more information - with this small edit the script works with Py35 again.

atedstone commented 7 years ago

still having problems with timeouts despite commit 16ccf82. If I manage to fix I'll post a commit here

atedstone commented 7 years ago

I've pushed all my recent changes onto my master branch, have a look at whether you want to merge. All of these changes have been to try to get to the bottom of why the script has been so slow at downloading for me. As of this morning's commits I appear to have solved the speed problems: (1) the existing authentication process was revealed to be very flaky once I started checking the response headers; (2) flush and fpsync don't appear to be needed when using the requests module stream=True option. I'm now downloading a tile a minute, much better than a tile per 30 mins or so as was worst case previously.

atedstone commented 4 years ago

I've diverged quite a long way from your implementation now ... so I offer my changes should you want them, but no problem if not! I'm using them all my end so they are serving their purpose.