Prevent too many parallel requests on the same webserver

It's possible to issue enough "random" requests against url-expander at the same time, such that it then itself issues parallel requests to the same target web server, potentially overloading it.

Test case

A simple server to count concurrently handled requests (this be a remote server hosting the short link redirects):

> cat us.go 
package main

import "net/http"
import "sync/atomic"
import "time"
import "fmt"

func main() {
    app := App{reqs: 0}
    http.HandleFunc("/", app.expand)
    http.ListenAndServe(":7777", nil)
}

type App struct {
    reqs uint32
}

func (a *App) expand(w http.ResponseWriter, r *http.Request) {
    rq := atomic.AddUint32(&a.reqs, 1)
    fmt.Printf(">> %d\n", rq)
    time.Sleep(1 * time.Second)
    rq = atomic.AddUint32(&a.reqs, ^uint32(0))
    w.Write([]byte(`{error: "foo"}`))
    fmt.Printf("<< %d\n", rq)
}

Running this and url-expander on the same machine, one way to simulate the problem is as follows.

# 8080 is the url-expander
# 7777 is the us server / our fake target web server
> seq 10 | parallel --verbose -j10 curl --silent "http://localhost:8080/api/expand\?url=http://localhost:7777/{}"

The us server output then shows something along the lines of:

>> 1
>> 2
>> 3
>> 4
>> 5
>> 6
>> 7
>> 8
>> 9
>> 10
<< 9
<< 8
<< 7
<< 6
<< 5
<< 4
<< 3
<< 2
<< 1
<< 0

Unfortunately, I couldn't find any way to grab the remote IP of a once established connection to build up a "live map" of currently executed work. Maybe resolving it upfront using net.LookupIP is just fine (but, unlike Java, it seems Go does no caching in this regard.)

Of course, this issue is not critical in controlled environments; rather just something to consider when exposing url-expander to the public.

vanekjar / urlexpander

Prevent too many parallel requests on the same webserver #2

Test case