Does lftp cache redirects?

[AndrewSav]

When I do

pget https://vagrantcloud.com/habsgoalie/boxes/windows2012r2-wmf5/versions/1.0.0/providers/virtualbox.box

it starts downloading, but on a slow connection after awhile it errors out:

pget: /habsgoalie/boxes/windows2012r2-wmf5/versions/1.0.0/providers/virtualbox.box: Access failed: 403 Forbidden (/archivist/boxes/8e46d238-c08c-40d0-9fa5-daeed5c86301?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIA4WZ7ZDX3WM4HDQ/20170915/us-east-1/s3/aws4_request&X-Amz-Date=20170915T020355Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=49f8883fd44b171b5c1fc056f74f265d74c9e40004af6d8625d890b7e5d31b75)

My guess is that vagrantcloud.com issues redirect to aws every time one does the GET request, and every time the token is different. This is time limited. lftp might cache the re-direct and instead of re-issuing every time, uses the cached value that is now expired.

Thoughts?

[lavv17]

lftp does not try to retrieve original URL when downloading of redirected URL breaks. The redirected URL is tried again instead. I don't know if it's feasible to change, in current architecture the original URLs are forgotten.

[AndrewSav]

I'll leave it with you then. It's more and more often nowadays to use cloud based blob storage which is protected by time based tokens. Not being able to use lftp to retrieve those is a bit of shame.

I'd also like to point out, that some redirects (like 307) should not be cached in principle as they are by design can give a different URL each time. If lftp caches those too (and not just 301) I would qualify this as a bug.

But of course whether to make improvements here or not is up to you.

[lavv17]

I'll fix the problem soon.

[AndrewSav]

Thank you, very much appreciated.

[lavv17]

Can you please take a look at #410?

[AndrewSav]

@lavv17 Is there a binary I can test it on? I'm not familiar with the code base enough to review the code changes.