client.CloseConns() question

[lucianchickenlover]

Hey, I'm using this to scrape websites. First off, it's an amazing tool. But I've got a question. So, I'm using rotating proxies with it. I create HTTP clients beforehand and then just serve them on threads for some performance gain.

client.CloseConnections()
client.CloseIdleConnections()

It doesn't seem to be enough to force a proxy change on the request, if u know what I mean.

client.setproxy() obviously won't help here as I only have one ip. I've also used the client option of name "DisAlive"

[gospider007]

Hey, I'm using this to scrape websites. First off, it's an amazing tool. But I've got a question. So, I'm using rotating proxies with it. I create HTTP clients beforehand and then just serve them on threads for some performance gain.

client.CloseConnections()
client.CloseIdleConnections()

It doesn't seem to be enough to force a proxy change on the request, if u know what I mean.

client.setproxy() obviously won't help here as I only have one ip. I've also used the client option of name "DisAlive"

Does your rotating proxy change the IP every time a new connection is created, or does it only change the IP after a certain period of time?

[gospider007]

Hey, I'm using this to scrape websites. First off, it's an amazing tool. But I've got a question. So, I'm using rotating proxies with it. I create HTTP clients beforehand and then just serve them on threads for some performance gain.

client.CloseConnections()
client.CloseIdleConnections()

It doesn't seem to be enough to force a proxy change on the request, if u know what I mean.

client.setproxy() obviously won't help here as I only have one ip. I've also used the client option of name "DisAlive"

You can refer to the following code to find the problem. When the return value of resp.IsNewConn() is true, this request is a newly established connection. If it is false, it reuses the previous connection. Since the rotating proxy only switches the IP when creating a new connection, you only need to determine whether this request reuses the connection to find the bug

for i := 0; i < 2; i++ {
    resp, err := requests.Get(nil, "https://myip.top")
    if err != nil {
        log.Panic(err)
    } else {
        log.Printf("send num: %d, new conn: %v", i, resp.IsNewConn())
    }
}

[lucianchickenlover]

Thank you, I will try that later and come back with a response! I gotta say I love the entire library, makes my life EASY.

One more question?

• Is defer resp.Close() still needed? I see it is there but not mentioned at all so you probably take care of it internally and just exposed it for whoever might have a use for it?

Edit:

Alright.

2023/11/13 22:37:37 send num: 0, new conn: true, ip: 109.158.152.98  
2023/11/13 22:37:37 send num: 1, new conn: false, ip: 109.158.152.98

For your example code. After using ForceCloseConns(), it creates a new connection. This is exactly what I wanted, thank you again! You're a godsent sir.

[gospider007]

Thank you, I will try that later and come back with a response! I gotta say I love the entire library, makes my life EASY.

One more question?

• Is defer resp.Close() still needed? I see it is there but not mentioned at all so you probably take care of it internally and just exposed it for whoever might have a use for it?

Edit:

Alright.

2023/11/13 22:37:37 send num: 0, new conn: true, ip: 109.158.152.98  
2023/11/13 22:37:37 send num: 1, new conn: false, ip: 109.158.152.98

For your example code. After using ForceCloseConns(), it creates a new connection. This is exactly what I wanted, thank you again! You're a godsent sir.

The name of resp.Close() is now changed to resp.CloseBody() due to a semantic error. Please see here for details: https://github.com/gospider007/requests/issues/2 Regarding the issue of resp.CloseBody(), it is automatically executed by default. It will only not be actively executed under the following circumstances:

1. websocket protocol
2. sse protocol
3. stream in requestsOption equals true These three types of requests are all streaming requests and body and conn will not be automatically managed. When resp.IsStream() is true, one of the above three situations is met. At this time, you need to manually execute resp.CloseBody()

[lucianchickenlover]

I will be closing this if you don't mind. Both your new features are amazing and the new fingerprinting "tests" were amazing examples! Keep up the good work, hands down best library I've used.