[SR-14677] FoundationNetworking web request error on Debian/TwisterOS

[swift-ci]

Previous ID	SR-14677
Radar	rdar://problem/78698174
Original Reporter	MacGregor (JIRA User)
Type	Bug

Environment

Raspberry PI 4, 4gb ram, Debian/TwisterOS, swift5.1

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 0 | |Component/s | Foundation | |Labels | Bug | |Assignee | None | |Priority | Medium | md5: f2166f3f1cc9ddf3c5fe1e91bb8d9745

Issue Description:

I am using the following code to get the content from a website. On macOS this works fine, but on the Pi 4 this only throws \<Error Domain=NSURLErrorDomain Code=-1 "(null)">. ** I also tried several other ways to download the website content, but they are all throwing the same error on the pi while working fine on the Mac.

Anny ideas how to fit that issue ?

import Foundation **

#if canImport(FoundationNetworking)

import FoundationNetworking

#endif

#if os(Linux)

import Glibc

#else

import Darwin.C

#endif

let url = "https://www.google.com"

if let url = URL(string: url) {

        **do** {

            **let** contents = **try** String(contentsOf: url)

            print(contents)

        } **catch** { print(error) }

else { print("error") }

[typesanitizer]

@swift-ci create

[0xfeedface1993]

Recently, when developing Swift programs on Linux, I encountered a situation where the same network request worked fine with the curl command but immediately failed when using URLSession, showing the error <Error Domain=NSURLErrorDomain Code=-1 "(null)">. It drove me crazy.

The scenario is as follows: at a certain point in time, a series of network requests need to be executed. The first network request works perfectly, but the second one starts to fail with the previously mentioned error, and some other requests also fail immediately, with most of the errors being NSURLErrorDomain Code=-1001 timeouts.

In order to investigate the issue, I switched to using the async-http-client framework for testing these network requests on a small scale, and unsurprisingly, all the network requests succeeded! But does this mean I have to refactor all my code to use async-http-client?

I was truly frustrated with this issue and started various attempts, including examining the open-source implementation of URLSession, which Apple graciously made available. Finally, I found a solution. Let me share it with you:

// Or modify it to your preferred encoding strategy
request.addValue("*", forHTTPHeaderField: "Accept-Encoding")

Yes, it's as simple as adding an accepted encoding to the header.

Why? Oh my goodness!

There are actually three reasons behind this:

1. URLSession was not interacting properly with libcurl.
2. libcurl turned out to be too fragile.
3. The web server was not using a compression strategy.

Here's the analysis:

NSURLErrorDomain Code=-1 is actually NSURLErrorUnknown, originating from CFURLSessionEasyCodeABORTED_BY_CALLBACK (CURLE_ABORTED_BY_CALLBACK). Clearly, it's an error reported by libcurl, meaning that somewhere, incorrect parameters or operations were passed to curl.

// Sources/FoundationNetworking/URLSession/libcurl/MultiHandle.swift
case (CFURLSessionEasyCodeABORTED_BY_CALLBACK, _):
    return NSURLErrorUnknown // Or NSURLErrorCancelled if we're in such a state

The culprit was in the set(automaticBodyDecompression:) method, where the request header was being set to Accept-Encoding="". Fortunately, most well-behaved web servers follow the rules, so my first network request worked fine. The problem arose when the web server returned an unencoded or differently encoded response, causing libcurl to throw in the towel.

// `swift-corelibs-foundation/Sources/FoundationNetworking/URLSession/libcurl/EasyHandle.swift`
func set(automaticBodyDecompression flag: Bool) {
    if flag {
        "".withCString {
            try! CFURLSession_easy_setopt_ptr(rawHandle, CFURLSessionOptionACCEPT_ENCODING, UnsafeMutableRawPointer(mutating: $0)).asError()
        }
        try! CFURLSession_easy_setopt_long(rawHandle, CFURLSessionOptionHTTP_CONTENT_DECODING, 1).asError()
    } else {
        try! CFURLSession_easy_setopt_ptr(rawHandle, CFURLSessionOptionACCEPT_ENCODING, nil).asError()
        try! CFURLSession_easy_setopt_long(rawHandle, CFURLSessionOptionHTTP_CONTENT_DECODING, 0).asError()
    }
}

CFURLSessionOptionACCEPT_ENCODING is essentially CURLOPT_ACCEPT_ENCODING, which sets the "Accept-Encoding" header in the network request.

// CoreFoundation/URL.subproj/CFURLSessionInterface.c
CFURLSessionOption const CFURLSessionOptionACCEPT_ENCODING = { CURLOPT_ACCEPT_ENCODING };

Everywhere this method was called, it was set to true, indicating that this functionality was designed without considering situations where the encoding strategy might not match.

I had previously seen issues with FTP services failing with URLSession, and I believe it's for the same reason.