[Bug]: failed to do request

[emretulek]

TLS client version

v1.3.11

System information

Windows Server 2019 64 bit Intel(R) Xenon(R) E-2136 CPU @ 3.30GHz Nodejs v16.17

failed to do request: Post "https://xxxxxxxx.xxx/": unexpected EOF

I get the following error very often when using proxy. this issue sometimes causes the code to freeze without error.

After doing some research, I came across the following topic. is there a parameter that i can apply for tls client? And does it solve the problem?

https://stackoverflow.com/a/19006050

Issue description

I get this error very often when using proxy.

failed to do request: Get "https://www.example.com/": EOF

Steps to reproduce / Code Sample

let tlsClientLibrary = ffi.Library('./tls-client-windows-64-1.3.11.dll', {
    'request': ['string', ['string']],
    'getCookiesFromSession': ['string', ['string']],
    'addCookiesToSession': ['string', ['string']],
    'freeMemory': ["void", ['string']],
    'destroyAll': ['string', []],
    'destroySession': ['string', ['string']]
});

const requestPayloadDefault = {
    "tlsClientIdentifier": "chrome_105",
    "followRedirects": false,
    "insecureSkipVerify": false,
    "withoutCookieJar": false,
    "withDefaultCookieJar": false,
    "isByteRequest": false,
    "catchPanics": false,
    "additionalDecode": null,
    "withDebug": false,
    "forceHttp1": false,
    "withRandomTLSExtensionOrder": true,
    "timeoutSeconds": 30,
    "timeoutMilliseconds": 0,
    "sessionId": "my-session-id",
    "proxyUrl": "",
    "isRotatingProxy": true,
    "certificatePinningHosts": {},
    "headers": {},
    "headerOrder": [
        "accept",
        "user-agent",
        "accept-encoding",
        "accept-language"
    ],
    "requestUrl": "",
    "requestMethod": "GET",
    "requestBody": "",
    "requestCookies": []
}

let requestPayload = JSON.parse(JSON.stringify(requestPayloadDefault));
    requestPayload.requestUrl = `https://www.xxx.com/`;
    requestPayload.requestMethod = "GET";
    requestPayload.proxyUrl = proxy;
    requestPayload.sessionId = makeid(32);
    requestPayload.tlsClientIdentifier = options.tlsClientIdentifier;
    requestPayload.headers = {
        'User-Agent': headers['user-agent'],
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
        'Accept-Language': headers['accept-language'],
        'Accept-Encoding': 'gzip, deflate, br',
        'Cookie': headers['cookie'],
        'Upgrade-Insecure-Requests': '1',
        'Sec-Fetch-Dest': 'document',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-Site': 'cross-site',
        'Sec-Fetch-User': '?1',
        'Te': 'trailers',
        'Pragma': 'no-cache',
        'Cache-Control': 'no-cache',
        'Connection': 'close',
    }

    return await new Promise((resolve, reject) => {
        tlsClientLibrary.request.async(JSON.stringify(requestPayload), (err, response) => {
            resolve(JSON.parse(response))
        });
    }).then((response) => {
        return response;
    });

[bogdanfinn]

@emretulek could you please share more information about the actual content of the used variables? otherwise your provided code to reproduce the issue does not even run ...

requestPayload.tlsClientIdentifier = options.tlsClientIdentifier;
'User-Agent': headers['user-agent'],
'Accept-Language': headers['accept-language'],
'Cookie': headers['cookie'],

and did you try it with the latest 1.5.0 release?

[emretulek]

I'm using a residental proxy, I have two requests linked together. Both requests exit from the same ip among themselves.

random one of the following versions

let tlsClientIdentifier = [
    "chrome_103",
    "chrome_104",
    "chrome_105",
    "chrome_106",
    "chrome_107",
    "chrome_108",
    "chrome_109",
    "chrome_110",
    "chrome_111",
    "chrome_112"
];

random chrome useragent may not be version compatible with tlsClientidentifier. "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"

usually a fixed accept language "accept-language": "en-US,en;q=0.5"

and string cookies "cookie": "key=316D605E-C079-45EB-A9F0-2941BD75765B; key2=ZKhBvgALAAEHSbR95Y"

I will try the latest version as soon as possible.

[bogdanfinn]

@emretulek when i use your provided code and just grab some residential proxies everything works fine for me.

Any other hints how to reproduce your issue? i mean somehow you are able to do it

[emretulek]

I spawn child processes with promiseAll by a main task, each of which makes 10 * 2 = 20 requests in groups of 2. The number of children can sometimes reach 100. I also get some errors when I use node-fetch instead of tls-client, of course I don't get that many errors. It's around 1/10. In addition, the requests I make with node-fetch are somehow timeout, tls-client can freeze without responding, which causes the number of children in the background to swell.

The proxy I'm using is pretty slow and problematic, I'm aware of that, but when I use it with node-fetch, the flow doesn't break. Here are some errors I get with fetch: request to https://xxxxxx.xxx/ failed, reason: Client network socket disconnected before secure TLS connection was established request to https://xxxxxx.xxxl/ failed, reason: socket hang up

[emretulek]

@emretulek could you please share more information about the actual content of the used variables? otherwise your provided code to reproduce the issue does not even run ...

requestPayload.tlsClientIdentifier = options.tlsClientIdentifier;
'User-Agent': headers['user-agent'],
'Accept-Language': headers['accept-language'],
'Cookie': headers['cookie'],

and did you try it with the latest 1.5.0 release?

I would like to state that I did all the final tests with version 1.5.0.

[flying-spider]

Hello, have you solved your problem? I also have the same problem, which is strange. It has been running normally for 3 months, but it only started to appear today

[flying-spider]

@emretulek当我使用您提供的代码并获取一些住宅代理时，一切对我来说都很好。

还有其他提示如何重现您的问题吗？我的意思是你能以某种方式做到这一点 I have found that this is fine if the proxy IP does not carry an account and password, such as https://127.0.0.1:8888

[Probabilities]

anyone found a fix for this that works with proxies with auth?

I am having the same issue but only happens when using a lot of concurrent requests. I am using an ipv6 proxy

[RKYGit]

Anynone found a fix that works with proxies please ?

[Probabilities]

Anynone found a fix that works with proxies please ?

No but I know what is causing the problem. TLS client does not close all connections when they are done with, so with proxies that have a thread limitter it will easily fill up the amount you have. The simple fix is to close the connect after every request but I am unsure how to do so.

Also I found out it only happens with proxies with username:password auth. (only ipv6 from what I have tested)