[clients/horizon] internal stream error from StreamTransactions

[vickiniu]

(horizon.Client).StreamTransactions returns the following error when streaming transactions by account ID. It seems to occur when significant time passes between transactions hitting the ledger (~15-20 seconds). The error is thrown by this line (https://github.com/stellar/go/blob/master/clients/horizon/client.go#L400) and causes StreamTransactions to stop streaming and exit.

Error: stream error: stream ID 1; INTERNAL_ERROR

Request:

GET /accounts/GDXTNUA3OSTCS3WBOCDEGOCMPK5DCKXR43N5334DHCSDOFCNIORCKGRJ/transactions?cursor=47185331052613632 HTTP/1.1                      Host: horizon-testnet.stellar.org                                                                                                          Accept: text/event-stream

Response Header:

HTTP/2.0 200 OK
Connection: close
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Cf-Ray: 4583f4c999c45158-SJC
Content-Type: text/event-stream; charset=utf-8
Date: Mon, 10 Sep 2018 18:29:04 GMT
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
Set-Cookie: __cfduid=dbb79172e1868de7279fa311ead5bc8da1536604125; expires=Tue, 10-Sep-19 18:28:45 GMT; path=/; domain=.stellar.org; HttpOnly
Vary: Origin
X-Ratelimit-Limit: 17200
X-Ratelimit-Remaining: 17191
X-Ratelimit-Reset: 2905

[vickiniu]

Also, running

curl -H "Accept: text/event-stream" "https://horizon-testnet.stellar.org/accounts/GDXTNUA3OSTCS3WBOCDEGOCMPK5DCKXR43N5334DHCSDOFCNIORCKGRJ/transactions?cursor=47184356095037440"

exits after a similar amount of time with the message

curl: (18) transfer closed with outstanding read data remaining

[bartekn]

Traffic to Horizon was going through CloudFlare. After switching off CF and adding a code that handles io.ErrUnexpectedEOF error (https://github.com/stellar/go/pull/663), it works fine. My best guess right now is that CF does some weird (and possibly incorrect) manipulation of SSE requests/responses in http/2. Let's track it in another GH issue.

[kr]

Nice find! Does this mean the horizon server is directly serving internet requests now? Are there any tasks we need to do to make sure that's safe? For example, https://blog.gopheracademy.com/advent-2016/exposing-go-on-the-internet/ lists several things that don't come out of the box with net/http and crypto/tls.

[bartekn]

Does this mean the horizon server is directly serving internet requests now?

No, it still goes through AWS ELB as we run multiple servers (both horizon.stellar.org and horizon-testnet.stellar.org).

Are there any tasks we need to do to make sure that's safe? For example, https://blog.gopheracademy.com/advent-2016/exposing-go-on-the-internet/ lists several things that don't come out of the box with net/http and crypto/tls.

I wasn't aware of all the things in this article (ex. crypto/tls configuration) so I think it's a good idea to check the existing code for these issues.

For maximum safety, I think it's a good idea to use well established third-party services like CloudFlare, AWS ELB, AWS WAF that tracks important security issues and are first to fix them in their products.

[bartekn]

FYI #663 is merged.