NewSingleHostReverseProxy, pls do a HostClient

[hiqsociety]

having plenty of problems with NewSingleHostReverseProxy can u check? with http2 (h2) plenty of issues.

rp, _ := reverseproxy.NewSingleHostReverseProxy("https://123.123.123.123:808/") //<- is this correct?
  h2client, _ := client.NewClient()
  h2client.SetClientFactory(factory.NewClientFactory(
      config.WithDialer(standard.NewDialer()),
      config.WithTLSConfig(&tls.Config{InsecureSkipVerify: true}),
  ))
  rp.SetClient(h2client)
  rp.SetDirector(func(req *protocol.Request) {
  //req.SetRequestURI(string(reverseproxy.JoinURLPath(req, rp.Target))) //<- what is this actually?
  req.SetRequestURI("example.com/")
  req.Header.SetHostBytes("example.com") //<- not sure if this is supposed to work too, in fact it's strange if it's not entered. redirect multiple times
  req.URI().SetScheme(httpScheme) //<- this doesnt really work, this redirects multiple times too
  //req.Options().Apply([]config.RequestOption{config.WithSD(true)}) //<- is this needed?
})

i cant get it to work. please get the below HostClient{} feature up as I've spent too much time testing NewSingleHostReverseProxy.

basically please have a h2 implementation of below. thx.

    proxyClient = &fasthttp.HostClient{
        Addr: "/tmp/somet.socket",
        //Addr: "127.0.0.1:8080",
        Dial: func(addr string) (net.Conn, error) {
            return net.Dial("unix", addr)
        },
        IsTLS: false,

        MaxIdleConnDuration:           30 * time.Second,
        ReadBufferSize:                64 * 1024,
        WriteBufferSize:               64 * 1024,
        ReadTimeout:                   30 * time.Second,
        WriteTimeout:                  30 * time.Second,
        DisableHeaderNamesNormalizing: true,
        DisablePathNormalizing:        true,

    }

[Duslia]

What is the error?

[li-jin-gou]

I thinl you can use https://www.cloudwego.io/docs/hertz/reference/config/ and as it is two frames, I don't think we need to align them and we provide a similar configuration.

[hiqsociety]

@Duslia

1. keeps redirecting back to the setscheme uri (redirection error), which doesnt work properly
2. no idea how to use properly. can u try out what i mean?
3. this part works perfectly

   rp, _ := reverseproxy.NewSingleHostReverseProxy("https://123.123.123.123:808/ but i the host and uri section is using ip address not the ones i've set. and becomes problem of 1.

[hiqsociety]

@li-jin-gou pls provide a similar function because maybe then you'll see why it's not working as expected with http2

[Duslia]

@Duslia

1. keeps redirecting back to the setscheme uri (redirection error), which doesnt work properly
2. no idea how to use properly. can u try out what i mean?
3. this part works perfectly

rp, _ := reverseproxy.NewSingleHostReverseProxy("https://123.123.123.123:808/ but i the host and uri section is using ip address not the ones i've set. and becomes problem of 1.

Sry, I don't know your question. Please construct a minimum complete and reproducible example for us. If you can't understand the docs, maybe you can read the source code. It's simple, too. Now this way of communication is inefficient.

[Duslia]

If the result is unexpected, please follow the default bug template to open an issue. The template is:

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Hertz version:

Please provide the version of Hertz you are using.

Environment:

The output of go env.

Additional context

Add any other context about the problem here.

[hiqsociety]

@Duslia

error

2022/12/08 19:06:14.827526 transport.go:68: [Error] HERTZ: Error=accept tcp [::]:443: accept4: too many open files
2022/12/08 19:06:14.827675 hertz.go:70: [Error] HERTZ: Receive close signal: error=accept tcp [::]:443: accept4: too many open files

reverse proxy "main.go"

package main

import (
        "crypto/tls"
        "context"
        "time"
        "log"
        "fmt"

//      "github.com/hertz-contrib/http2"
        config "github.com/hertz-contrib/http2/config"
        "github.com/cloudwego/hertz/pkg/common/utils"
        "github.com/cloudwego/hertz/pkg/protocol/consts"
        "github.com/cloudwego/hertz/pkg/protocol"
        "github.com/hertz-contrib/http2/factory"
        //"github.com/cloudwego/hertz/pkg/app/middlewares/client/sd"
        //config2 "github.com/cloudwego/hertz/pkg/common/config"
        "github.com/cloudwego/hertz/pkg/app/client"
        "github.com/cloudwego/hertz/pkg/app"
        "github.com/cloudwego/hertz/pkg/network/standard"
        "github.com/cloudwego/hertz/pkg/app/server"
        "github.com/hertz-contrib/reverseproxy"
)

func main() {
        cfg := &tls.Config{
                MinVersion:       tls.VersionTLS12,
                CurvePreferences: []tls.CurveID{tls.X25519, tls.CurveP256},
                CipherSuites: []uint16{
                        tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                },
        }
        cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
        if err != nil {
                fmt.Println(err.Error())
        }
        cfg.Certificates = append(cfg.Certificates, cert)
        cfg.NextProtos = append(cfg.NextProtos, "h2")

        h := server.New(server.WithHostPorts(":443"), server.WithALPN(true), server.WithTLS(cfg))
        h.AddProtocol("h2", factory.NewServerFactory(
                config.WithReadTimeout(time.Minute),
                config.WithDisableKeepAlive(false)))

        //h := server.New(server.WithHostPorts(":80"))

/*
        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://127.0.0.1:8081/")

        h2client, _ := client.NewClient()
        h2client.SetClientFactory(factory.NewClientFactory(
                config.WithDialer(standard.NewDialer()),
                config.WithTLSConfig(&tls.Config{InsecureSkipVerify: true}),
        ))

        rp.SetClient(h2client)
*/
        //      h.GET("/", rp.ServeHTTP)

        h.Use(func(cc context.Context, ctx *app.RequestContext) {
                //log.Printf("fwV = %v", fw)
                /*
                if fw == 0 {
                        ctx.Hijack(bcznet.HijackHandlerHertz)
                        return
                }
                */
                log.Printf("ipAddrS = %s", ctx.ClientIP())
                log.Printf("oldCTX = %s",ctx)

                ProxyHandlerHertz(cc, ctx)
        })

        h.GET("/json", func(c context.Context, ctx *app.RequestContext) {
                log.Printf("ipaddr = %s", ctx.ClientIP())
                ctx.JSON(consts.StatusOK, utils.H{"ping": "pong2"})
        })
        h.Spin()
}

func ProxyHandlerHertz(cc context.Context, ctx *app.RequestContext) {

//        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://127.0.0.1:8081/")

        h2client, _ := client.NewClient()
        h2client.SetClientFactory(factory.NewClientFactory(
                config.WithDialer(standard.NewDialer()),
                config.WithTLSConfig(&tls.Config{InsecureSkipVerify: true}),
        ))

        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://127.0.0.1:8081/")
//      rp.SetClient(cli)
        rp.SetDirector(func(req *protocol.Request){
//              req.SetRequestURI(string(reverseproxy.JoinURLPath(req, rp.Target)))
//              req.Header.SetHostBytes(req.URI().Host())
//              req.Options().Apply([]config2.RequestOption{config2.WithSD(true)})

//              log.Printf("\n\n")
//              log.Printf("value = %s = %s = %s",string(reverseproxy.JoinURLPath(req, rp.Target)),rp.Target,req)
        })

        rp.SetClient(h2client)

        log.Printf("newCTX = %s",ctx)
        rp.ServeHTTP(cc, ctx)

}

backend server "server.go"

package main

import (
        "crypto/tls"
        "context"
        "time"
        "log"
        "fmt"

//      "github.com/hertz-contrib/http2"
        config "github.com/hertz-contrib/http2/config"
        "github.com/cloudwego/hertz/pkg/common/utils"
        "github.com/cloudwego/hertz/pkg/protocol/consts"
        //"github.com/cloudwego/hertz/pkg/protocol"
        "github.com/hertz-contrib/http2/factory"
        //"github.com/cloudwego/hertz/pkg/app/middlewares/client/sd"
        //config2 "github.com/cloudwego/hertz/pkg/common/config"
        //"github.com/cloudwego/hertz/pkg/app/client"
        "github.com/cloudwego/hertz/pkg/app"
        //"github.com/cloudwego/hertz/pkg/network/standard"
        "github.com/cloudwego/hertz/pkg/app/server"
        //"github.com/hertz-contrib/reverseproxy"
)

func main() {
        cfg := &tls.Config{
                MinVersion:       tls.VersionTLS12,
                CurvePreferences: []tls.CurveID{tls.X25519, tls.CurveP256},
                CipherSuites: []uint16{
                        tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                },
        }
        cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
        if err != nil {
                fmt.Println(err.Error())
        }
        cfg.Certificates = append(cfg.Certificates, cert)
        cfg.NextProtos = append(cfg.NextProtos, "h2")

        h := server.New(server.WithHostPorts(":8081"), server.WithALPN(true), server.WithTLS(cfg))
        h.AddProtocol("h2", factory.NewServerFactory(
                config.WithReadTimeout(time.Minute),
                config.WithDisableKeepAlive(false)))

        //h := server.New(server.WithHostPorts(":80"))

/*
        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://127.0.0.1:8081/")

        h2client, _ := client.NewClient()
        h2client.SetClientFactory(factory.NewClientFactory(
                config.WithDialer(standard.NewDialer()),
                config.WithTLSConfig(&tls.Config{InsecureSkipVerify: true}),
        ))

        rp.SetClient(h2client)
*/
        //      h.GET("/", rp.ServeHTTP)

        h.Use(func(cc context.Context, ctx *app.RequestContext) {
                //log.Printf("fwV = %v", fw)
                /*
                if fw == 0 {
                        ctx.Hijack(bcznet.HijackHandlerHertz)
                        return
                }
                */
                //log.Printf("ipAddrS = %s", ctx.ClientIP())
                log.Printf("oldCTX = %s",ctx)

                ProxyHandlerHertz(cc, ctx)
        })

        /*
        h.GET("/json", func(c context.Context, ctx *app.RequestContext) {
                log.Printf("ipaddr = %s", ctx.ClientIP())
                ctx.JSON(consts.StatusOK, utils.H{"ping": "pong2"})
        })
        */
        h.Spin()
}

func ProxyHandlerHertz(cc context.Context, ctx *app.RequestContext) {

        ctx.JSON(consts.StatusOK, utils.H{"ping": "pong2"})
        return
        /*

        //        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://127.0.0.1:8081/")

        h2client, _ := client.NewClient()
        h2client.SetClientFactory(factory.NewClientFactory(
                config.WithDialer(standard.NewDialer()),
                config.WithTLSConfig(&tls.Config{InsecureSkipVerify: true}),
        ))

        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://54.242.245.230:808/")
        //      rp.SetClient(cli)
        rp.SetDirector(func(req *protocol.Request){
                req.SetRequestURI(string(reverseproxy.JoinURLPath(req, rp.Target)))
                req.Header.SetHostBytes(req.URI().Host())
                req.Options().Apply([]config2.RequestOption{config2.WithSD(true)})

                //              log.Printf("\n\n")
                //              log.Printf("value = %s = %s = %s",string(reverseproxy.JoinURLPath(req, rp.Target)),rp.Target,req)
        })

        rp.SetClient(h2client)

        log.Printf("newCTX = %s",ctx)
        rp.ServeHTTP(cc, ctx)
        */
}

[Duslia]

There is no need to create a client for each request. Just use the same one.

[hiqsociety]

@Duslia sorry can u help out? i really wish to test faster. pls help check what's wrong with the code and provide a working one. it's the client side main.go issue. pls advise how it should be.

seriously, i dont understand what u mean.

[Duslia]

You create a client each time and this will use all the fd. Just initialize it in global. Actually, what you ask is basic questions. I think you should read the document carefully or the source code and have some debug ability.

[hiqsociety]

@Duslia can you pls run this and help change it so this works? it's supposed to show "show this please"

it runs two servers, :8081 (this works) and :443 (this doesnt work, it's supposed to show the :8081 message)

package main

import (
        "crypto/tls"
        "context"
        "time"
        "log"
        "fmt"

//      "github.com/hertz-contrib/http2"
        config "github.com/hertz-contrib/http2/config"
        "github.com/cloudwego/hertz/pkg/common/utils"
        "github.com/cloudwego/hertz/pkg/protocol/consts"
        "github.com/cloudwego/hertz/pkg/protocol"
        "github.com/hertz-contrib/http2/factory"
        //"github.com/cloudwego/hertz/pkg/app/middlewares/client/sd"
        //config2 "github.com/cloudwego/hertz/pkg/common/config"
        "github.com/cloudwego/hertz/pkg/app/client"
        "github.com/cloudwego/hertz/pkg/app"
        "github.com/cloudwego/hertz/pkg/network/standard"
        "github.com/cloudwego/hertz/pkg/app/server"
        "github.com/hertz-contrib/reverseproxy"
)

var (   
        h2client, _ = client.NewClient()
)

func init() {
        h2client.SetClientFactory(factory.NewClientFactory(
                config.WithDialer(standard.NewDialer()),
                config.WithTLSConfig(&tls.Config{InsecureSkipVerify: true}),
        ))
}

func main() {
        cfg := &tls.Config{
                MinVersion:       tls.VersionTLS12,
                CurvePreferences: []tls.CurveID{tls.X25519, tls.CurveP256},
                CipherSuites: []uint16{
                        tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                        tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                },
        }
        cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
        if err != nil {
                fmt.Println(err.Error())
        }
        cfg.Certificates = append(cfg.Certificates, cert)
        cfg.NextProtos = append(cfg.NextProtos, "h2")

go func(){
        h := server.New(server.WithHostPorts(":8081"), server.WithALPN(true), server.WithTLS(cfg))
        h.AddProtocol("h2", factory.NewServerFactory(
                config.WithReadTimeout(time.Minute),
                config.WithDisableKeepAlive(false)))
        h.Use(func(cc context.Context, ctx *app.RequestContext) {
                ctx.String(200, "show this please")
        })
        h.Spin()
}()

        h := server.New(server.WithHostPorts(":443"), server.WithALPN(true), server.WithTLS(cfg))
        h.AddProtocol("h2", factory.NewServerFactory(
                config.WithReadTimeout(time.Minute),
                config.WithDisableKeepAlive(false)))

        //h.Use(rp.ServeHTTP(ProxyHandlerHertz))
        ///*
        h.Use(func(cc context.Context, ctx *app.RequestContext) {
                //need to do some processing here so this part is needed.
                //log.Printf("fwV = %v", fw)
                /*
                if fw == 0 {
                        ctx.Hijack(bcznet.HijackHandlerHertz)
                        return
                }
                */
                ///*
                log.Printf("ipAddrS = %s", ctx.ClientIP())
                log.Printf("oldCTX = %s",ctx)

                ProxyHandlerHertz(cc, ctx)
        })
        //*/

        h.GET("/json", func(c context.Context, ctx *app.RequestContext) {
                log.Printf("ipaddr = %s", ctx.ClientIP())
                ctx.JSON(consts.StatusOK, utils.H{"ping": "pong3"})
        })
        h.Spin()
}

func ProxyHandlerHertz(cc context.Context, ctx *app.RequestContext) {
       //need to set another request uri based on some preprocessor logic.
       //e.g. if ctx.Host() == "example1.com" { backend from somewhere else and ctx.SetRequestURI to something else

        rp, _ := reverseproxy.NewSingleHostReverseProxy("https://127.0.0.1:8081/")
        rp.SetClient(h2client)
        rp.SetDirector(func(req *protocol.Request){
//              req.SetRequestURI(string(reverseproxy.JoinURLPath(req, rp.Target))) //<- this part is needed actually
//              req.Header.SetHostBytes(req.URI().Host())
//              req.Options().Apply([]config2.RequestOption{config2.WithSD(true)})
//              log.Printf("\n\n")
//              log.Printf("value = %s = %s = %s",string(reverseproxy.JoinURLPath(req, rp.Target)),rp.Target,req)
        })

        log.Printf("newCTX = %s",ctx)
//      rp.ServeHTTP(cc, ctx)
}

[hiqsociety]

@Duslia u missed this can u help out? just updated the code. h.Use() and routing through a reverse proxy that does some processing and diversion based on ctx.Host and then setrequestURI and backend server ip and port

need dynamic backend ip and backend ctx.Host() set inside proxyhandler

[Duslia]

There is no error output after I run example code

[hiqsociety]

it's not code running error, when you visit :443, it doesnt output "show this please". means it's not getting response from 8081

[Duslia]

Please show your request and the command you execute. You just give me the code, but don't give me how to reproduce it.

[hiqsociety]

@Duslia go to https://127.0.0.1:8081/ (produce "show this please", works)

go to https://127.0.0.1:443/ (expected "show this please", results "404...")

go to 443 should show "show this please"

what's the problem with the reverse proxy? i would like to modify values within the h.Use() and also within the reverse proxy as the backend ip and Host() information will be dynamically modified based on

1. client ip address
2. Host()
3. request URI()

this is the only issue i have now. please help. thx

normally it's done within fasthttp.HostClient() function.

[Duslia]

You commented out the director function, so the reverseproxy won't take effect. Please look the source code