hoisie / web

The easiest way to create web applications with Go
MIT License
3.67k stars 702 forks source link

Not sure sockets are being closed correctly... #128

Open Altonymous opened 11 years ago

Altonymous commented 11 years ago

I'm not sure if this is a me, web.go, or go issue... but I was doing a little light load testing and discovered that under a little load I started getting Connection reset by peer errors.

Here's a gist of my setup: https://gist.github.com/Altonymous/7bab9ee8caa6da368883

Some things to note. the machine doing the siege is different than the machine serving the requests.

The limits setup on the machine serving the requests are as follows: $ launchctl limit cpu unlimited unlimited filesize unlimited unlimited data unlimited unlimited stack 8388608 67104768 core 0 unlimited rss unlimited unlimited memlock unlimited unlimited maxproc 1064 1064 maxfiles 10240 120000

$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 10240 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1064 virtual memory (kbytes, -v) unlimited

hoisie commented 11 years ago

Can you try using Go's http package (the code below should work).

I get similar errors when running on my mac. I can try running this benchmark on linux tomorrow.

As far as I know, unless you add a Connection: close header, Go's http server will keep the connection open. My guess is that it's hitting the file descriptor limit.

package main

import (
  "encoding/json"
  "net/http"
)

// structs
type Reading struct {
  Id   string `json:"id"`
  Name string `json:"name"`
}

func main() {
http.HandleFunc("/machines/", func(w http.ResponseWriter, r *http.Request) {
  // Setup readings
  readings := prepareReadings()

  // return readings
  w.Write([]byte(readingsToString(readings)))
})
  http.ListenAndServe(":3000", nil)
}

func readingsToString(readings []Reading) string {
  data, err := json.Marshal(readings)
  if err != nil {
    panic(err)
  }

  return string(data)
}

func prepareReadings() []Reading {
  var readings []Reading
  for i := 1; i <= 1; i++ {
    readings = append(readings, Reading{Name: "Thing"})
  }

  return readings
}
Altonymous commented 11 years ago

It may be hitting the limit. That's why I included my settings. Unless its another setting I'm missing. If you are getting the same error using their libraries I'm sure ill get more of the same. If it is a core library issue do you want to open an issue since you will probably be able to explain it better?

Altonymous commented 11 years ago

I swapped over to running it on linux, and don't seem to have the same problems. It looks like I may have found an issue related to Mac's only. :(

Altonymous commented 11 years ago

Oh I forgot, I also updated to 1.1beta of golang. So it could be that as well.

hoisie commented 11 years ago

This seems related:

https://code.google.com/p/go/issues/detail?id=66

Altonymous commented 11 years ago

Well I've moved all my stuff to ec2 to continue my testing and I'm running ubuntu there as well. I'm getting the error again.... :(

Altonymous commented 11 years ago

I used the code you generated above, installed rc3, and dropped it out onto my aws instance.

The problem still appears to be happening. I'm not sure how to determine what the cause of the issue is at this point.

Here are the limits for my process...

[api #3312 -- limits]
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             59479                59479                processes
Max open files            4999999              4999999              files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       59479                59479                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

When I 'lsof | wc -l', I dont' get above 1000. Generally in the ~800-850 range.

When I 'watch --interval=2 'netstat -tuna |grep "SYN_RECV"|wc -l'', I am generally in the ~130-250 range.