curl / curl-for-win

Reproducible curl binaries for Linux, macOS and Windows
https://curl.se/windows/
MIT License
686 stars 207 forks source link

What is the Unicode "feature"? #54

Closed eabase closed 11 months ago

eabase commented 11 months ago

I'm comparing the native Windows curl with the latest release from here and I see one thing missing, the Unicode feature.

What is that exactly, and when is it needed?

curl 8.3.0 (x86_64-w64-mingw32) libcurl/8.3.0 OpenSSL/3.1.2 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.56.0 ngtcp2/0.19.1 nghttp3/0.15.0
Release-Date: 2023-09-13
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd

#--------------------------------------------------------

curl 8.0.1 (Windows) libcurl/8.0.1 Schannel WinIDN
Release-Date: 2023-03-20
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets
vszakats commented 11 months ago

Windows offers a "unique" way (compared to *nix operating systems) for apps to handle Unicode characters. It needs a lot of internal complexity. The curl Unicode feature implements some of this. Support is still incomplete. It also breaks compatibility with non-Unicode curl builds.

The Unicode feature means that curl talks to the OS via its "wide" Win32 API and "wide" CRT calls. These support the whole Unicode character range. To talk to these APIs, curl needs to convert strings back and forth between UTF-16, UTF-8, and potentially other 8-bit codepages. curl dependencies might have their own ways to exchange strings with them in Unicode. curl need to adapt to these as well.

We track Unicode issues/discussions here: https://github.com/curl/curl/issues?q=label%3Aunicode https://github.com/curl/curl/discussions?discussions_q=is%3Aopen+label%3Aunicode