http.Server.Serve: "protocol not available" (OpenBSD 6.5)

peterljung commented 5 years ago

I have problem running the test on OpenBSD 6.5.

$ uname -a
OpenBSD nuc.lounge.se 6.5 GENERIC.MP#3 amd64
$ go version
go version go1.12.1 openbsd/amd64

This is probably related to something in OpenBSD. I have a very similar issue with another web framework. Could ephemeral port ranges have something to do with it?

$ doas ./aerotest                                                                                                        
Server running on: http://localhost:4000
--------------------------------------------------------------------------------
panic: set tcp 127.0.0.1:4000->127.0.0.1:38930: protocol not available

goroutine 21 [running]:
github.com/aerogo/aero.(*Application).serveHTTP(0xc000102000, 0xc0000a0d88)
    /home/peter/go/pkg/mod/github.com/aerogo/aero@v1.3.7/Application.go:418 +0x29f
created by github.com/aerogo/aero.(*Application).ListenAndServe
    /home/peter/go/pkg/mod/github.com/aerogo/aero@v1.3.7/Application.go:197 +0xe3

But maybe you have seen something similar on another platform?

I tried a simple example using http on the same machine and that works fine.

package main

import (
    "fmt"
    "log"
    "net/http"
)

type helloHandler struct{}

func (h helloHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "hello, you've hit %s\n", r.URL.Path)
}

func main() {
    err := http.ListenAndServe(":9999", helloHandler{})
    log.Fatal(err)
}

Any idea what this could be related to?

akyoto commented 5 years ago

Have never seen this panic before using Arch Linux on x86_64.

λ uname -a
Linux home 5.1.5-arch1-2-ARCH #1 SMP Mon May 27 03:37:39 UTC 2019 x86_64 GNU/Linux

I don't have an OpenBSD installation to test with, but if you're seeing the same problems in another framework then it might be something related to your system.

I don't want to be so quick on dropping responsibility onto someone else, though, so let's dig a little deeper.

The line referenced in the panic is this one:

// This will block the calling goroutine until the server shuts down.
// The returned error is never nil and in case of a normal shutdown
// it will be `http.ErrServerClosed`.
err := server.Serve(listener)

if err != http.ErrServerClosed {
    panic(err)
}

Via some google searches I found these threads, all returning the same error message:

The message itself is coming from a syscall code 42 on OpenBSD in go/src/syscall/zerrors_openbsd_amd64.go:

https://github.com/golang/go/blob/master/src/syscall/zerrors_openbsd_amd64.go#L1497

This leads me to searching again and finding those:

It looks like the problem is related to the TCP_USER_TIMEOUT socket option.

peterljung commented 5 years ago

Yes, you may be on to the problem ...

TCP socket options in linux include TCP_USER_TIMEOUT: http://man7.org/linux/man-pages/man7/tcp.7.html

TCP_USER_TIMEOUT (since Linux 2.6.37)

OpenBSD does not: https://man.openbsd.org/tcp

Is this a generic problem with the go port or http package? Or can this option be controlled in aero?

I minimal example which trigger the problem would be good to be able to send a bug to OpenBSD mailing lists if it is a general go port problem. I tried with the small HTTP example above but that did not trigger the issue ...

peterljung commented 5 years ago

gRPC package handles this in: https://github.com/grpc/grpc-go/blob/master/internal/syscall/syscall_nonlinux.go vs https://github.com/grpc/grpc-go/blob/master/internal/syscall/syscall_linux.go

Is aero dependent on gRPC in some way?

akyoto commented 5 years ago

depth isn't showing it:

λ depth github.com/aerogo/aero
github.com/aerogo/aero
  ├ compress/gzip
  ├ context
  ├ crypto/tls
  ├ errors
  ├ fmt
  ├ io
  ├ io/ioutil
  ├ mime
  ├ net
  ├ net/http
  ├ net/http/httptest
  ├ os
  ├ os/signal
  ├ path/filepath
  ├ sort
  ├ strconv
  ├ strings
  ├ sync
  ├ syscall
  ├ time
  ├ github.com/aerogo/csp
    └ strings
  ├ github.com/aerogo/http/ciphers
    └ crypto/tls
  ├ github.com/aerogo/http/client
    ├ bytes
    ├ compress/gzip
    ├ crypto/tls
    ├ fmt
    ├ io
    ├ io/ioutil
    ├ log
    ├ net
    ├ net/url
    ├ strconv
    ├ sync
    ├ unicode
    ├ github.com/aerogo/http/ciphers
    ├ github.com/akyoto/stringutils/convert
    └ github.com/json-iterator/go
      ├ bytes
      ├ encoding
      ├ encoding/base64
      ├ encoding/json
      ├ errors
      ├ fmt
      ├ io
      ├ math
      ├ math/big
      ├ reflect
      ├ sort
      ├ strconv
      ├ strings
      ├ sync
      ├ unicode
      ├ unicode/utf16
      ├ unicode/utf8
      ├ unsafe
      ├ github.com/modern-go/concurrent (unresolved)
      └ github.com/modern-go/reflect2 (unresolved)
  ├ github.com/aerogo/linter-performance
    ├ fmt
    ├ time
    ├ github.com/aerogo/http/client
    └ github.com/akyoto/color
      ├ fmt
      ├ io
      ├ os
      ├ strconv
      ├ strings
      ├ sync
      ├ github.com/mattn/go-colorable
        ├ bytes
        ├ io
        ├ os
        └ github.com/mattn/go-isatty
          └ golang.org/x/sys/unix
            ├ bytes
            ├ encoding/binary
            ├ net
            ├ runtime
            ├ sort
            ├ strings
            ├ sync
            ├ syscall
            ├ time
            └ unsafe
      └ github.com/mattn/go-isatty
  ├ github.com/aerogo/session
    ├ fmt
    ├ sync
    ├ time
    └ github.com/akyoto/uuid
      ├ bytes
      ├ crypto/md5
      ├ crypto/rand
      ├ crypto/sha1
      ├ encoding/binary
      ├ encoding/hex
      ├ errors
      ├ fmt
      ├ hash
      ├ io
      ├ net
      ├ os
      ├ strings
      ├ sync
      └ time
  ├ github.com/aerogo/session-store-memory
    ├ errors
    ├ sync
    └ github.com/aerogo/session
  ├ github.com/akyoto/color
  ├ github.com/akyoto/hash
    └ github.com/zeebo/xxh3
      ├ math/bits
      ├ unsafe
      └ golang.org/x/sys/cpu
        ├ encoding/binary
        └ runtime
  ├ github.com/akyoto/stringutils/unsafe
    ├ reflect
    └ unsafe
  └ github.com/json-iterator/go
61 dependencies (42 internal, 19 external, 0 testing).

The biggest problem for me in debugging this is not having a physical system to test with. I'll try to download Virtualbox.

In the meantime, could you try downloading the git repo, using the repo via replace and removing the following lines from the source:

Only remove the second if the first didn't work. If removing the timeouts from the first link works, I'm honestly going to be surprised. These timeouts are important and exist for a reason. Without these timeouts, you will have a lot of idle connections and your app might eventually crash. This might be circumvented by putting something like nginx in front of the server, but if the server is the main server / actual endpoint of the client then you definitely would want to have these timeouts.

akyoto commented 5 years ago

Found the culprit, it's https://github.com/aerogo/aero/blob/v1.3.11/Listener.go#L32

connection.SetKeepAlivePeriod(keepAlivePeriod)

Aero on OpenBSD

akyoto commented 5 years ago

Fixed in 1.3.12 via https://github.com/aerogo/aero/commit/079571728d466eb915cc2d9d596a20aa45bdb341:

Aero 1.3.12 on OpenBSD

peterljung commented 5 years ago

Super thanks!

I suppose you need to handle timeouts in the application code to avoid triggering the TCP timeout.

akyoto commented 5 years ago

I might be wrong, but I believe there is a default TCP keep-alive interval considering that SetKeepAlive(true) is returning no errors on OpenBSD.

aerogo / aero

http.Server.Serve: "protocol not available" (OpenBSD 6.5) #10