quyin
commented
9 years ago just some notes:
- URLs ending with / without slash: technically they can serve different contents, but in practice often one redirects to the other. if we correctly deal with redirections this won't affect us much, so I'll leaning towards doing nothing to it.
- the root problem is that the 'curl' utility on most downloaders we currently use is too old and does not use a secure enough cipher honored by many today's websites.
the only way out would be to upgrade curl on these machines to a newer version, which seems non-trivial -- you'll need to manually, recursively upgrade all the underlying libraries used by curl.
one workaround is to use a recent, statically linked curl binary. I did find one: http://www.magicermine.com/demos/curl/curl/curl.html, however it's not actively maintained. building such a static binary also seems non-trivial.
right now I'm leaning towards the Ermine curl binary solution, mostly because it is simple. it's fairly recent so probably won't become obsolete soon. we might want to migrate the whole downloader pool to AWS or similar services some time.
Best Regards, Yin Qu (屈垠)
On Thu, Aug 6, 2015 at 3:15 PM, Andruid Kerne notifications@github.com wrote:
Assigned #21 https://github.com/ecologylab/BigSemanticsService/issues/21 to @quyin https://github.com/quyin.
— Reply to this email directly or view it on GitHub https://github.com/ecologylab/BigSemanticsService/issues/21#event-375704470 .
http://api.ecologylab.net/BigSemanticsService/metadata.json?url=http%3A%2F%2Fprezi.com%2F