yesodweb / wai

Haskell Web Application Interface
MIT License
834 stars 262 forks source link

Difference in behaviour : warp vs warp-tls #114

Closed erikd closed 12 years ago

erikd commented 12 years ago

I was having some troubles with warp-tls giving me a "data: end of file" failure.

I tried debugging this myself, but wasn't able to make much progress. However, during my debugging I came up with a (relatively) simple test program that shows the problem (I'll add it as an attachment, warp-tls-test.hs).

When the test program is run as:

runghc warp-tls-test.hs

it sets up a warp server with a simple WAI Application and uses http-conduit read from the server. This works as expected.

However, when the test program is run as:

runghc warp-tls-test.hs tls

it sets up a warp-tls server instead of a warp server and uses http-conduit to access the server using HTTPS instead of HTTP. This fails.

My debugging suggests that the failure occurs when warp-tls is parsing the HTTP request, but I'm not making much progress debugging this further.

erikd commented 12 years ago
{-# LANGUAGE OverloadedStrings #-}

import Blaze.ByteString.Builder
import Control.Concurrent (forkIO, killThread)
import Control.Monad (unless)
import Control.Monad.IO.Class (liftIO, MonadIO)
import Control.Monad.Trans.Class (lift)
import Control.Monad.Trans.Resource
import Data.ByteString (ByteString)
import Data.Conduit (($$))
import Data.String
import Network.HTTP.Types
import Network.TLS
import Network.Wai
import Network.Wai.Handler.Warp
import Network.Wai.Handler.WarpTLS
import System.Environment

import qualified Data.ByteString.Char8 as BS
import qualified Data.ByteString.Lazy.Char8 as LBS
import qualified Data.CaseInsensitive as CI
import qualified Data.Conduit as DC
import qualified Data.Conduit.Binary as CB
import qualified Network.HTTP.Conduit as HC
import qualified Network.HTTP.Types as HT

main :: IO ()
main = do
    opts <- getArgs
    case opts of
        ["tls"] -> runTest True
        _ -> runTest False

runTest :: Bool -> IO ()
runTest tls = runResourceT $ do
    let (proto, testFunc) = if tls
        then ("https", runTestServerTLS)
        else ("http", runTestServer)
    let testServerPort = 3030
    _ <- allocate (forkIO $ testFunc testServerPort) killThread
    let url = proto ++ "://localhost:" ++ show testServerPort ++ "/"
    request <- lift $ HC.parseUrl url
    direct@(Result _ _ hdrs _) <- httpRun request
    let isWarp =
            case lookup "server" hdrs of
                Just s -> BS.isPrefixOf "Warp" s
                Nothing -> False
    unless isWarp $ error "No 'Server: Warp' header."
    liftIO $ printResult direct

--------------------------------------------------------------------------------

runTestServer :: Int -> IO ()
runTestServer port = do
    let settings = defaultSettings { settingsPort = port, settingsHost = "*6" }
    putStrLn "runTestServer"
    runSettings settings serverApp

runTestServerTLS :: Int -> IO ()
runTestServerTLS port = do
    let settings = defaultSettings { settingsPort = port, settingsHost = "*6" }
        tlsSettings = TLSSettings "test/certificate.pem" "test/key.pem"
    putStrLn "runTestServerTLS"
    runTLS tlsSettings settings serverApp

serverApp :: Request -> ResourceT IO Response
serverApp req= do
    liftIO $ putStrLn "serverApp"
    let text = BS.concat
            [ "  Method          : " , requestMethod req , "\n"
            , "  HTTP Version    : " , fromString (show (httpVersion req)) , "\n"
            , "  Path Info       : " , rawPathInfo req , "\n"
            , "  Query String    : " , rawQueryString req , "\n"
            , "  Server Name     : " , serverName req , "\n"
            , "  Server Port     : " , fromString (show (serverPort req)), "\n"
            , "  Secure (SSL)    : " , fromString (show (isSecure req)), "\n"
            , "  Request Headers :\n\n"
            , headerShow (requestHeaders req)
            , "\n"
            ]

    let respHeaders =
            [ (hContentType, "text/plain")
            , (hContentLength, fromString $ show $ BS.length text)
            ]
    let responseBS s h = ResponseBuilder s h . fromByteString
    return $ responseBS status200 respHeaders text

headerShow :: [HT.Header] -> ByteString
headerShow headers =
    BS.concat $ map hdrShow headers
  where
    hdrShow (f, v) = BS.concat [ "  ", CI.original f , ": " , v, "\n" ]

-- | Use HC.http to fullfil a HC.Request. We need to wrap it because the
-- Response contains a Source which we need to read to generate our result.
httpRun :: HC.Request (ResourceT IO) -> ResourceT IO Result
httpRun req = liftIO $ withManagerSettings settings $ \mgr -> do
    liftIO $ dumpHttpConduitRequest req
    HC.Response st hver hdrs bdyRes <- HC.http req mgr
    (bdy, finalizer) <- DC.unwrapResumable bdyRes
    bodyText <- bdy $$ CB.take 8192
    finalizer
    return $ Result (HT.statusCode st) hver hdrs $ BS.concat $ LBS.toChunks bodyText
  where
    settings = HC.def { HC.managerCheckCerts = \ _ _ -> return CertificateUsageAccept }

data Result = Result Int HT.HttpVersion [HT.Header] ByteString

printResult :: Result -> IO ()
printResult (Result status _ headers body) = do
    putStrLn $ "Status : " ++ show status
    putStrLn "Headers :"
    BS.putStr $ headerShow headers
    putStrLn "Body :"
    BS.putStrLn body

dumpHttpConduitRequest :: HC.Request m -> IO ()
dumpHttpConduitRequest req =
    let text = BS.concat
            [ "------- HttpConduit Request ------------------------------------------------------\n"
            , "Method          : " , HC.method req , "\n"
            , "Secure (SSL)    : " , fromString (show (HC.secure req)), "\n"
            , "Host Name       : " , HC.host req , "\n"
            , "Host Port       : " , fromString (show (HC.port req)), "\n"
            , "Path            : " , HC.path req , "\n"
            , "Query String    : " , HT.urlDecode False (HC.queryString req), "\n"
            , "Request Headers :\n"
            , headerShow (HC.requestHeaders req), "\n"
            ]
    in BS.putStr text

withManagerSettings :: HC.ManagerSettings -> (HC.Manager -> ResourceT IO a) -> IO a
withManagerSettings settings f = runResourceT $ do
    (_, manager) <- allocate (HC.newManager settings) HC.closeManager
    f manager
snoyberg commented 12 years ago

I get identical results:

ubuntu@ubuntu-snoyop-dell:~/Desktop$ runghc warp-tls-test.hs 
runTestServer
------- HttpConduit Request ------------------------------------------------------
Method          : GET
Secure (SSL)    : False
Host Name       : localhost
Host Port       : 3030
Path            : /
Query String    : 
Request Headers :

serverApp
Status : 200
Headers :
  Server: Warp/1.3.2
  Content-Type: text/plain
  Content-Length: 246
Body :
  Method          : GET
  HTTP Version    : HTTP/1.1
  Path Info       : /
  Query String    : 
  Server Name     : localhost
  Server Port     : 3030
  Secure (SSL)    : False
  Request Headers :

  Host: localhost:3030
  Accept-Encoding: gzip

ubuntu@ubuntu-snoyop-dell:~/Desktop$ runghc warp-tls-test.hs tls
runTestServerTLS
------- HttpConduit Request ------------------------------------------------------
Method          : GET
Secure (SSL)    : True
Host Name       : localhost
Host Port       : 3030
Path            : /
Query String    : 
Request Headers :

serverApp
Status : 200
Headers :
  Server: Warp/1.3.2
  Content-Type: text/plain
  Content-Length: 246
Body :
  Method          : GET
  HTTP Version    : HTTP/1.1
  Path Info       : /
  Query String    : 
  Server Name     : localhost
  Server Port     : 3030
  Secure (SSL)    : False
  Request Headers :

  Host: localhost:3030
  Accept-Encoding: gzip

Can you provide more information on your environment? For example, there's a bug with optimizations in GHC 7.6 which Vincent recently worked around for the tls package.

erikd commented 12 years ago

Running Debian Testing on x86-64.

GHC is from Debian:

Glasgow Haskell Compiler, Version 7.4.2, stage 2 booted by GHC version 7.4.1
Using binary package database: /usr/lib/ghc/package.conf.d/package.cache
Using binary package database: /home/erikd/.ghc/x86_64-linux-7.4.2/package.conf.d/package.cache
wired-in package ghc-prim mapped to ghc-prim-0.2.0.0-7d3c2c69a5e8257a04b2c679c40e2fa7
wired-in package integer-gmp mapped to integer-gmp-0.4.0.0-af3a28fdc4138858e0c7c5ecc2a64f43
wired-in package base mapped to base-4.5.1.0-6e4c9bdc36eeb9121f27ccbbcb62e3f3
wired-in package rts mapped to builtin_rts
wired-in package template-haskell mapped to template-haskell-2.7.0.0-2bd128e15c2d50997ec26a1eaf8b23bf

ghc-pkg list gives:

/var/lib/ghc/package.conf.d:
    Cabal-1.14.0
    array-0.4.0.0
    base-4.5.1.0
    bin-package-db-0.0.0.0
    binary-0.5.1.0
    bytestring-0.9.2.1
    containers-0.4.2.1
    deepseq-1.3.0.0
    directory-1.1.0.2
    extensible-exceptions-0.1.1.4
    filepath-1.3.0.0
    (ghc-7.4.2)
    ghc-prim-0.2.0.0
    (haskell2010-1.1.0.1)
    (haskell98-2.0.0.1)
    hoopl-3.8.7.3
    hpc-0.5.1.1
    integer-gmp-0.4.0.0
    old-locale-1.0.0.4
    old-time-1.1.0.0
    pretty-1.1.1.0
    process-1.1.0.1
    rts-1.0
    template-haskell-2.7.0.0
    time-1.4
    unix-2.5.1.1

/home/erikd/.ghc/x86_64-linux-7.4.2/package.conf.d:
    HUnit-1.2.5.1
    QuickCheck-2.5
    SHA-1.5.1
    aeson-0.6.0.2
    ansi-terminal-0.5.5
    asn1-data-0.7.1
    attoparsec-0.10.2.0
    attoparsec-conduit-0.5.0.2
    authenticate-1.3.1.1
    base-unicode-symbols-0.2.2.4
    base64-bytestring-1.0.0.0
    blaze-builder-0.3.1.0
    blaze-builder-conduit-0.5.0.1
    blaze-html-0.5.1.0
    blaze-markup-0.5.1.1
    byteorder-1.0.3
    bytestring-lexing-0.4.0
    case-insensitive-0.4.0.3
    cereal-0.3.5.2
    certificate-1.2.8
    cipher-aes-0.1.2
    clientsession-0.8.0.1
    conduit-0.5.2.4
    cookie-0.4.0.1
    cprng-aes-0.2.4
    cpu-0.1.1
    crypto-api-0.10.2
    crypto-conduit-0.4.0.1
    crypto-pubkey-types-0.1.1
    cryptocipher-0.3.5
    cryptohash-0.7.5
    css-text-0.1.1
    data-default-0.5.0
    date-cache-0.3.0
    dlist-0.5
    email-validate-0.2.8
    entropy-0.2.1
    failure-0.2.0.1
    fast-logger-0.3.1
    file-embed-0.0.4.5
    filesystem-conduit-0.5.0.1
    hamlet-1.1.1
    hashable-1.1.2.5
    hjsmin-0.1.2
    hspec-1.3.0
    hspec-expectations-0.3.0.2
    html-conduit-0.1.0.2
    http-conduit-1.6.1
    http-date-0.0.2
    http-types-0.7.3.0.1
    language-javascript-0.5.4
    largeword-1.0.3
    lifted-base-0.1.2
    mime-mail-0.4.1.2
    mime-types-0.1.0.0
    monad-control-0.3.1.4
    monad-logger-0.2.1
    mtl-2.1.2
    network-2.4.0.1
    network-conduit-0.5.0.2
    parsec-3.1.3
    path-pieces-0.1.2
    pem-0.1.1
    persistent-1.0.1.2
    persistent-template-1.0.0.2
    pool-conduit-0.1.0.3
    primitive-0.4.1
    pureMD5-2.1.2.1
    pwstore-fast-2.3
    random-1.0.1.1
    ranges-0.2.4
    regex-base-0.93.2
    regex-compat-0.95.1
    regex-posix-0.95.2
    resource-pool-0.2.1.1
    resourcet-0.4.0.1
    safe-0.3.3
    semigroups-0.8.4.1
    shakespeare-1.0.1.4
    shakespeare-css-1.0.1.5
    shakespeare-i18n-1.0.0.2
    shakespeare-js-1.0.0.6
    shakespeare-text-1.0.0.5
    silently-1.2.0.2
    simple-sendfile-0.2.7
    skein-0.1.0.9
    socks-0.4.2
    stm-2.4
    stringsearch-0.3.6.3
    strptime-1.0.8
    syb-0.3.7
    system-fileio-0.3.10
    system-filepath-0.4.7
    tagged-0.4.4
    tagsoup-0.12.8
    tagstream-conduit-0.5.3
    tar-0.4.0.0
    text-0.11.2.3
    tls-0.9.11
    tls-extra-0.4.6
    transformers-0.3.0.0
    transformers-base-0.4.1
    unix-compat-0.3.0.2
    unix-time-0.1.2
    unordered-containers-0.2.2.1
    utf8-light-0.4.0.1
    utf8-string-0.3.7
    vault-0.2.0.1
    vector-0.9.1
    void-0.5.8
    wai-1.3.0.1
    wai-app-static-1.3.0.2
    wai-extra-1.3.0.2
    wai-logger-0.3.0
    wai-test-1.3.0
    warp-1.3.2
    warp-tls-1.3.1
    xml-conduit-1.0.3.1
    xml-types-0.3.3
    xss-sanitize-0.3.2
    yaml-0.8.0.2
    yesod-1.1.1
    yesod-auth-1.1.1.1
    yesod-core-1.1.2
    yesod-default-1.1.0
    yesod-form-1.1.3
    yesod-json-1.1.0
    yesod-persistent-1.1.0
    yesod-platform-1.1.3
    yesod-routes-1.1.0
    yesod-static-1.1.0.1
    yesod-test-0.3.0.1
    zlib-0.5.4.0
    zlib-bindings-0.1.1.1
    zlib-conduit-0.5.0.1
snoyberg commented 12 years ago

What output do you get when you try and run the tls version?

erikd commented 12 years ago
erikd@pharoah > runghc test/warp-tls-test.hs tls
runTestServerTLS
------- HttpConduit Request ------------------------------------------------------
Method          : GET
Secure (SSL)    : True
Host Name       : localhost
Host Port       : 3030
Path            : /
Query String    : 
Request Headers :

warp-tls-test.hs: data: end of file
snoyberg commented 12 years ago

Just to try and localize further: what happens if you try and access the warp-tls server from your browser? In other words, is this a problem in http-conduit or warp-tls?

erikd commented 12 years ago

I have a separate http-conduit test that pulls a page from http://en.wikipedia.org/ and https://en.wikipedia.org/ and both work fine. I also tested the server with a web browser and it worked fine.

My debugging suggested that accessing warp-tls from a web browser was somehow different from accessing with http-conduit.

Since you can't reproduce it I should do some more debugging here.

erikd commented 12 years ago

Same problem on i386 Debian testing with pretty much the same package list.

Same problem on i386 Ubuntu Precise with pretty much the same package list.

iquiw commented 12 years ago

New tls-1.0.1 fixes this issue for me.

erikd commented 12 years ago

Yes, it no longer has the 'end of file' errror, but if I do the same request for a warp server and warp-tls server I still get one weird difference. The Wai.rawPathInfo field for a HTTP request is:

/

whereas with warp-tls its:

https://localhost:31080/

To me, it seems they should be the same.

snoyberg commented 12 years ago

That's likely correct behavior; it's probably an actual different in the HTTP request that was sent.

erikd commented 12 years ago

@snoyberg Yes, you're right. Closing this.

quetz commented 10 years ago

I'm having this issue too. warp-tls-2.0.0.1 warp-2.0.1 scotty-0.6.2

Looks like it happens when client closes keep-alive connection. (If I use http 1.0 or set Connection: close header there is no 'data: end of file' message).

malthe commented 10 years ago

I have this issue with a simple "Hello world!" handler:

{-# LANGUAGE OverloadedStrings #-}

import Network.Wai (responseLBS, Application)
import Network.Wai.Handler.Warp
import Network.Wai.Handler.WarpTLS (runTLS, tlsSettings)
import Network.HTTP.Types (status200)
import Network.HTTP.Types.Header

main = let port = 8080
           ds = setPort port $ setHost "0.0.0.0" defaultSettings
           ts = tlsSettings "certificate.pem" "key.pem"
       in do putStrLn $ "Listening on port " ++ show port
             runTLS ts ds app

app :: Application
app req f =
  f $ responseLBS status200 [
    (hContentType, "text/plain"),
    (hConnection, "close"),
    (hContentLength, "12")
    ] "Hello world!"

Note that it does not help with "Connection: close".

snoyberg commented 10 years ago

@malthe Can you give more details on what exactly is happening? This issue is long with a lot of history. It probably makes more sense to start a new issue with a full bug report.

malthe commented 10 years ago

Yes, that makes sense. I'll try to reproduce in a clean environment and open a new issue.

On Thursday, July 17, 2014, Michael Snoyman notifications@github.com wrote:

@malthe https://github.com/malthe Can you give more details on what exactly is happening? This issue is long with a lot of history. It probably makes more sense to start a new issue with a full bug report.

— Reply to this email directly or view it on GitHub https://github.com/yesodweb/wai/issues/114#issuecomment-49322264.