search

anowell / mia

Experimental Algorithmia CLI (no longer the official CLI)
41 stars 5 forks source link

OSX CLI hangs reading responses in the 4k-16k range #12

Closed anowell closed 7 years ago

anowell commented 8 years ago

tl;dr: listing directories and downloading files hang in OSX if the result falls roughly between 4KB and 16KB. The cause isn't clear yet, but I'm following up with an upstream project to narrow it down.

story time

The predominate http library in Rust has been hyper which focuses mostly on low-level primitives, and by default it used openssl for https. This is pretty reasonable on Linux, but expecting openssl to be installed on OSX or Windows was less ideal given that both have their own native TLS stacks. Nevertheless, for the past year, using the CLI required openssl.

Fast forward, and the author of hyper created reqwest, a wrapper around hyper with more focus on the ergonomics and API stability you might expect from an http client. Additionally, it supports using the platform-native TLS stack by default.

The CLI made the switch from hyper to reqwest (as a very early adopter - I submitted PRs to allow me to use it before it was even officially released), and now OSX and Windows users don't need to install openssl to use the CLI. However, it appears to have regressed for 4-16k downloads from our API on OSX.

My first instinct is a bug somewhere in reqwest (i.e the way it combines hyper with the security-framework crate) or perhaps a bug in the security-framework crate. I have opened a detailed upstream issue to get to the bottom of it. That said, I have so far failed to repro this failure on any URL outside of our API, which reduces my confidence in that instinct.

If a fix isn't found relatively soon, the next best work-around is to downgrade the CLI back to the 1.x branch of the client to continue using openssl for a while, which also involves backporting the changes required to support data connectors.

anowell commented 7 years ago

lots of good details in that upstream issue, that can be summarized up into this commit message: https://github.com/sfackler/rust-security-framework/commit/4cd518d11a5c7e292a48c7ced15555e931bd0278

Basically, OSX's Secure Transport did some surprising things when called with a length that was longer than the amount of data it had buffered internally, causing it to block trying to read a socket even though it had already copied all the data needed. This fix/workaround has been included in the 1.0.0 release.