Project: Support Cronet Go binding

klzgrad commented 2 years ago

Any Go developers who are interested in using the core of naiveproxy in Go natively, please see my proposal and comment below:

The core of naiveproxy is namely Chromium's net stack. It's possible to expose its interface in the style of BSD socket in a library libnaive.so or libnaive.a (or more accurately libchromiumsocket.a), which then could be hooked up with a Go program via FFI. The benefit of doing this: no socks5 round trip overhead for using Chromium's net stack, and more versatile to link directly with Go's net stacks at socket/connection level. The impact of doing this: a better TLS fingerprint parrot than utls with similar Go development friendliness.

To achieve this goal:

I can provide the code and build system to produce the proposed libnaive.a and corresponding C headers.
An interested Go developer to discuss the interface definition for the Go FFI.
Said Go developer develops Go binding and example integration showing Chromium net stack working in Go natively.
[x] Cronet's official Android build is for Java, but here the build is native and intended for use from Go. ~~Do you @nekohasekai think it's better to use Cronet from Java or Go?~~ I'll take it as a yes.
[x] The current build produces a shared library. Do you @nekohasekai think it's better to use a static library to link with Go?
[x] My Cronet build on Android still doesn't work.
[x] My Cronet build doesn't work with ThinLTO.
[x] ~~I'm considering finalizing it to -connect-authority-override in the next version. Really hard to name this one.~~ Currently it's decided as -connect-authority, as connect sufficiently defines the scope of this header, the leading - denotes its internal nature, and the suffix -override doesn't seem absolutely necessary and the lack of it doesn't introduce confusion.
[x] Record optimal static linking ldflags for each arch
[x] Add CI test case for cornet example and go example
[x] Document diff from upstream
[x] Write a specification of protocol used in Naiveproxy
[x] Add -network-isolation-key header to smuggle it in and configure stream isolation.

nekohasekai commented 2 years ago

Chromium already has cronet for calling from platforms like android, but it's basically https.

I'm not familiar with C/C++, but if you're done, I'll try it.

klzgrad commented 2 years ago

~~Cronet only provides a request-response API. One example:~~ https://developer.android.com/guide/topics/connectivity/cronet/reference/org/chromium/net/UrlRequest.Builder.html#setHttpMethod(java.lang.String) ~~the CONNECT method is not supported. I don't think there's any other way to create bidirectional sockets. Another proof is Cronet does not support WebSocket.~~

Cronet does support bidirectional "sockets". Several years ago when I checked, it was being used from gRPC with odd support status (ios only, or whatever, https://source.chromium.org/chromium/chromium/src/+/main:components/grpc_support/README.md: Currently Cronet (//components/cronet/ios) is the only consumer of this API). But it does support it. grpc-java does so via org.chromium.net.ExperimentalBidirectionalStream, grpc does it via third_party/objective_c/Cronet/bidirectional_stream_c.h (note the "objective_c", so it was designed for ios, but the cronet API is in C, which means this path may work but it was never tried). The reason I did not find it is because ExperimentalBidirectionalStream is not documented in https://developer.android.com/guide/topics/connectivity/cronet but it is documented in https://source.chromium.org/chromium/chromium/src/+/main:components/cronet/android/api/src/org/chromium/net/ExperimentalBidirectionalStream.java.

If you use Cronet's ExperimentalBidirectionalStream:

Its traffic behavior goes through the same TLS code path with NaiveProxy and regular URL requests, so, same TLS behaviors. But on higher levels, internally it goes through BidirectionalStreamSpdyImpl, which is somewhat different from NaiveProxy's SpdyProxyClientSocket and http stream paths of regular URL requests. I'm just admitting imperfect parrot here, because I can't audit them exhaustively now within the large Chromium net stack.
ExperimentalBidirectionalStream will need some care to avoid 1-RTT latency from waiting for HTTP response.
ExperimentalBidirectionalStream is not exactly a socket. Some slightly more work is needed to handle its HTTP CONNECT logic. Auth headers.
This happens in Java or C. It probably doesn't make sense to run heavy I/O between Java and Go at the same time. Assuming I/O heavy work happens in Go, then the work is to link Cronet native API to Go. The C API looks like this: https://source.chromium.org/chromium/chromium/src/+/main:components/grpc_support/include/bidirectional_stream_c.h. I will build a C++ test case to try this out before going with using Cronet's bidirectional stream.

The more ambitious goal of providing a general Go connection API with better TLS parroting seems better achieved with building a Go FFI with Cronet native API (if it works and is without major restrictions from the original Chromium stack). This requires more work, some design review, and more testing.

The less ambitious goal is to provide a specific Go API for running the logic of NaiveProxy inside Go address space without socks5 round trips, as a proxy backend along side other backends. This requires less work and is easier.

Recap of NaiveProxy

## The changes made to Chromium * Build minimization c647d3d36 base: Remove JNI function on Android b3a476823 dns: Fix iwyu 0ac6ca70c libc++: Disable exceptions and RTTI 543d0bdbe url: Remove perfetto tracing ecd4991eb base: Disable trace event 093898ac6 base: Don't fix Y2038 problem with icu be723d44a net, url: Remove icu 1c639b07d build: Disable Android java templates 1c3067640 build: Disable build_with_chromium 27de40aa2 base: Add Android stubs 6750e63ed net: Add Android stubs 1fa442a90 build: Remove tests and minimize 79fecf9f8 cert: Use builtin verifier on Android and Linux 8708bc15e cert: Add SystemTrustStoreStaticUnix * OpenWrt build support e3ef20e8f Fix OpenWrt build with use_allocator_shim=false 189411143 allocator: Improve MIPS coverage of spinlocks 61512a294 debug: Fix uClibc macro condition 7e3a06f14 third_party: Fix missing sgidefs.h for Musl 7104046e2 base: Do not forward declare stat64 for Musl 7105b1eb1 base: Fix narrowing casting for Musl eea4cbe33 base: Disable __close overloading for Musl 5cfc3960a process: Remove use of mallinfo for Musl d43ad9ef1 base: Remove use of mallinfo for Musl 489ba5d82 udp: Fix mmsghdr struct initializer for Musl 2c0f6ff2e dns: Support Musl 07dedde21 build: Add OpenWrt toolchains de5f76436 build: Support MIPS -mtune= flag 9df671fe0 build: Support ARM build without FPU 2c44fd6e3 build: Support -mcpu= on ARM and ARM64 a8a02e327 libc++: Guard C++20 atomic type aliases e543ca2e0 lss: Avoid naming conflict in fstatat64 * Android build support c6606de1d Fix android build * CI/CD 92976967d Add continuous integration and tests 0d649f1cd Add build scripts a8c41f623 Add source import tool a38b058f0 build: Handle empty pkgconfig in sysroot b62139c30 build: Add sysroot creator script 03f0fcfaa build: Force determinism in official build 8c4e850f8 Add .gitignore * Docs d07b4a5c4 Add example config.json a754cb6ba Add README bc5196118 Add LICENSE * Naive itself: bf42527be Add initial implementation of Naive client, a hopefully complete specification of what it does: * http_proxy_socket.cc: an HTTP proxy frontend that accepts HTTP requests from clients (features: CONNECT method only; detects client opt-in of payload padding using `padding` header in the HTTP CONNECT request; advertise padding capability using `padding` header in the HTTP CONNECT response) * socks5_server_socket.cc: a SOCKS5 proxy frontend that accepts SOCKS5 requests from clients * naive_connection.cc: a bidirectional pipe that connects a client socket accepted from a frontend (socks5, http, redir) with Chromium's `ClientSocketHandle` which is initiated towards the requested proxy origin (features: payload padding encapsulation and decapsulation; ) * naive_proxy.cc: Configures Chromium proxy settings, TLS params, parallelism with network isolation keys; A socket server that accepts TCP connections and moves them to backend * naive_proxy_bin.cc: Configures Chromium ClientSocketPool limits, config parsing, configures logging, builds URLRequestContext * naive_proxy_delegate.cc: Detects downstream and upstream padding capability and enables Fast Open logic in H2/H3 code in Chromium net stack. * redirect_resolver.cc: Deprecated, don't care * Changes of Chromium required by Naive 910fcc50f debug: Fix obsolete max check 0198ac540 quic: Add support for HTTP/3 CONNECT Fast Open 5f91e9d2d h2: Pad RST_STREAM frames a9bdea492 h2: Add support for HTTP/2 CONNECT Fast Open b95497e4b h2: Reduce warnings about RST on invalid streams 6fe5dc101 socket: Force tunneling for all sockets 3c48a78a0 socket: Allow higher limits for proxies f8e486bfd socket: Add RawConnect method bcd5d09f7 cert: Handle AIA response in PKCS#7 format * Chromium itself 1c761bbba Import chromium-100.0.4896.60 Apparently much of Naiveproxy itself can be implemented with less and cleaner Go code. ## Key changes and uses of the Chromium net stack, summary * Adds support in Chromium's H2 and H3 code to support HTTP CONNECT "Fast Open", i.e. does not wait for the response after sending the HTTP CONNECT request and sends the payload immediately. * Pads H2's RST_STREAM frames to slightly increase packet length entropy. * "Handles AIA response in PKCS#7 format", otherwise it complains about issues dealing with Let's Encrypt's certificates, but it hasn't been thoroughly investigated whether this is necessary. * Exposes a "RawConnect" function that creates Chromium ClientSocketHandle directly. * Configures Chromium's proxy setting, QUIC stack versions, logging, extra headers, DNS resolver static mapping,

klzgrad commented 2 years ago

Cronet's BidirectionalStream:

With proxy setting via myproxy.com, a BidirectionalStream to https://example.com will send TLS and HTTP headers to example.com, before sending user agent data to example.com. This does not work with the model where the user agent (browser) requires a socket to send TLS and HTTP headers to example.com for E2E encryption. (Cronet is the intermediary, not user agent)
Without proxy settings, a BidirectionalStream to myproxy.com cannot use the CONNECT method to create HTTP tunnels, because the authority field can only be myproxy.com. Patching it to support this use case is very messy and not realistic, because the entire concept of "stream" in Chromium does not consider tunnels.

So the use case BidirectionalStream can support is something like WebSockets, where the origin is represented in other fields.

In NaiveProxy both the client intermediary and server intermediary assume HTTP/2 CONNECT tunneling. A proxy system using Cronet's BidirectionalStream won't work with Caddy's forwardproxy (without abusing HTTP headers).

TBC

klzgrad commented 2 years ago

Patching it to support this use case is very messy and not realistic Cronet's BidirectionalStream won't work with Caddy's forwardproxy (without abusing HTTP headers)

I managed to find a not very messy way to smuggle the authority in: Pass a _real_authority HTTP header and use its value to override the authority when constructing the CONNECT header, and delete this fake header so it is not transmitted. This way a CONNECT tunnel can be set up, and then there is Cronet bidirectional stream API for simple read and write. With this, Cronet can be used as a substitute the "core" of Naiveproxy without the "major restrictions" mentioned above.

Architecturally Cronet and Naiveproxy both use the Chromium net core, but Cronet uses BidirectionalStream and Naiveproxy uses SpdyProxyClientSocket/QuicProxyClientSocket. BidirectionalStream and SpdyProxyClientSocket are also very similar. Naiveproxy is a more "lightweight" use of the Chromium net core because it can make more assumptions and use more simplifications. Cronet exposes more complete control interfaces.

In terms of the API as a product, Cronet is a better option because its API is better maintained. So unless more major restrictions are discovered (e.g. parrot detectors), it's better to use Cronet as basis to build proxy apps.

With that said, it's still better to use my fork of Chromium to build Cronet, because the build is minimized and supports OpenWrt and other niceties.

	Official Cronet release	My Cronet fork
Supported platforms	Android, iOS	Linux, Openwrt, Windows, Android, MacOS
Supported API	Java, ObjC	C
Usage	No CONNECT tunnels	Supports CONNECT tunnels

@nekohasekai The following is an example of using Cronet bidirectional stream:

naiveproxy-cronet-test-linux-x64.tar.xz from https://github.com/klzgrad/naiveproxy/releases/tag/cronet-test

# Note this does not use Chromium's build system
g++ bidi_example.cc libcronet.100.0.4896.60.so
# SSLKEYLOGFILE is saved at /tmp/keys so Wireshark can use it.
LD_LIBRARY_PATH=$PWD ./a.out https://my-caddy-forwardproxy.com "Basic $(printf user:pass | base64)"

Remaining questions:

nekohasekai commented 2 years ago

I have just tried

https://gist.github.com/nekohasekai/dda9ba499332433ba7a94f5f3d690a53

$ LD_LIBRARY_PATH=$PWD go run .
[0503/130502.548494:ERROR:cert_verify_proc_builtin.cc(603)] No net_fetcher for performing AIA chasing.
2022/05/03 13:05:02 on_stream_ready_callback
2022/05/03 13:05:02 on_response_headers_received, negotiated_protocol= h2
2022/05/03 13:05:02 :status:302
2022/05/03 13:05:02 cache-control:no-cache
2022/05/03 13:05:02 pragma:no-cache
2022/05/03 13:05:02 content-length:149
2022/05/03 13:05:02 content-type:text/html; charset=utf-8
2022/05/03 13:05:02 expires:-1
2022/05/03 13:05:02 location:https://www4.bing.com/?form=DCDN
2022/05/03 13:05:02 p3p:CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
2022/05/03 13:05:02 set-cookie:SUID=M; domain=.bing.com; expires=Wed, 04-May-2022 05:05:02 GMT; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:MUID=2B8BC37FAEB966EC2616D2E6AFB167E3; domain=.bing.com; expires=Sun, 28-May-2023 05:05:02 GMT; path=/; secure; SameSite=None
2022/05/03 13:05:02 set-cookie:MUIDB=2B8BC37FAEB966EC2616D2E6AFB167E3; expires=Sun, 28-May-2023 05:05:02 GMT; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:_EDGE_S=F=1&SID=241DA119413A6A0C1EBCB08040326B77; domain=.bing.com; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:_EDGE_V=1; domain=.bing.com; expires=Sun, 28-May-2023 05:05:02 GMT; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:SRCHD=AF=NOFORM; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:SRCHUID=V=2&GUID=FEA003B5EF0546E1B9F9EC8783811781&dmnchg=1; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:SRCHUSR=DOB=20220503; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:SRCHHPGUSR=SRCHLANG=zh-Hant; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:_SS=SID=241DA119413A6A0C1EBCB08040326B77; domain=.bing.com; path=/
2022/05/03 13:05:02 x-snr-routing:1
2022/05/03 13:05:02 strict-transport-security:max-age=31536000; includeSubDomains; preload
2022/05/03 13:05:02 x-cache:CONFIG_NOCACHE
2022/05/03 13:05:02 accept-ch:Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
2022/05/03 13:05:02 x-msedge-ref:Ref A: 03C48A42AE58420984C42B2E9D1FB30E Ref B: HKBEDGE0513 Ref C: 2022-05-03T05:05:02Z
2022/05/03 13:05:02 date:Tue, 03 May 2022 05:05:01 GMT
2022/05/03 13:05:02 on_read_completed
2022/05/03 13:05:02 <html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www4.bing.com/?form=DCDN">here</a>.</h2>
</body></html>

2022/05/03 13:06:05 on_failed
2022/05/03 13:06:05 net error  -365

Seems to work, but looks like it requires a lot of wrappers to use in go

nekohasekai commented 2 years ago

~~When I use bidirectional_stream_write, it just returns 1 and then segmentation fault.~~

Fixed

klzgrad commented 2 years ago

looks like it requires a lot of wrappers to use in go

I don't know how many is a lot. 100-200 lines of code seem reasonable.

There can be something to do on the C side to make things nicer. I can think of this https://github.com/golang/go/commit/918396b3e11cbb65bf7ef2423a4379e7373828cb _GoString_ which helps a bit. What are the specific complexities in wrapping it? I can check if there are other changes in C that makes things easier in Go.

nekohasekai commented 2 years ago

I don't know how many is a lot. 100-200 lines of code seem reasonable.

I am writing some wrapper like this:

nekohasekai commented 2 years ago

Unless we have a code generator for that idl, I guess

Maybe https://chromium.googlesource.com/chromium/src.git/+/HEAD/mojo/README.md

klzgrad commented 2 years ago

Chromium's Mojo does not support Go bindings.

I kind of get what you're trying to do. cronet.idl exports 100~200 functions. An automatic binding generator wouldn't produce code as nice as that of manually written, which is not a small undertaking but also achievable with a few days of work. And once written, it's also a complete and general Go binding on its own, for the official Cronet library.

I would suggest starting with the important part of the API first so to prove it works, in terms of replacing libnaive.so with native use of Cronet. Even though it can be fun to engage in completionism, as I can imagine, PublicKeyPins is not needed for proving the above concept.

nekohasekai commented 2 years ago

Does the port in real-authority effect? I always get "Client sent an HTTP request to an HTTPS server."

klzgrad commented 2 years ago

https://github.com/klzgrad/naiveproxy/blob/cronet-test/src/net/spdy/spdy_http_utils.cc#L110

I changed it to _real_authority some time ago.

nekohasekai commented 2 years ago

Now requests to 443 works, but requests with _real_authority port 80 always fails directly with -9 error. what's wrong?

And if I call write before on_response_headers_received I will get the same error

klzgrad commented 2 years ago

Net errors are defined here https://source.chromium.org/chromium/chromium/src/+/main:net/base/net_error_list.h. But ERR_UNEXPECTED is not informative.

Try Wireshark with /tmp/keys to see what happens on the wire.

nekohasekai commented 2 years ago

Try Wireshark with /tmp/keys to see what happens on the wire.

The client sent the h2 reset (reason cancel)

Chromium is rejecting the connection?

klzgrad commented 2 years ago

Probably didn't wait for the stream to finish before destroying it.

nekohasekai commented 2 years ago

I confirmed that bidirectional_stream_cancel or bidirectional_stream_destroy is not called before on failed

klzgrad commented 2 years ago

https://source.chromium.org/search?q=ERR_UNEXPECTED%20file:spdy&sq=&ss=chromium%2Fchromium%2Fsrc

https://source.chromium.org/chromium/chromium/src/+/main:net/spdy/bidirectional_stream_spdy_impl.cc;l=127?q=ERR_UNEXPECTED%20file:spdy&ss=chromium%2Fchromium%2Fsrc https://source.chromium.org/chromium/chromium/src/+/main:net/spdy/bidirectional_stream_spdy_impl.cc;l=418?q=ERR_UNEXPECTED%20file:spdy&ss=chromium%2Fchromium%2Fsrc

I have very little information to work with. Show code or read code.

nekohasekai commented 2 years ago

https://github.com/SagerNet/cronet-go/blob/main/bidirectional_stream.go

LD_LIBRARY_PATH=$PWD go run -v ./example https://myproxy.com user:pswd https://bing.com

klzgrad commented 2 years ago

Ok, there's too much code. I'm overwhelmed.

There's another way to get debug info:

  Cronet_Engine_StartNetLogToFile(cronet_engine, "/tmp/netlog.json", true);
  // do stream stuff
  Cronet_Engine_StopNetLog(cronet_engine);

View the saved netlog.json at https://netlog-viewer.appspot.com/. Don't upload the file elsewhere as it contains traffic data.

You should be able to locate where ERR_UNEXPECTED is generated.

nekohasekai commented 2 years ago

No valid information, like cancel was called right after write...

t=700 [st=  1]    HTTP2_SESSION_SEND_DATA
                  --> fin = false
                  --> size = 256
                  --> stream_id = 1
t=701 [st=  2]    HTTP2_SESSION_SEND_RST_STREAM
                  --> description = ""
                  --> error_code = "8 (CANCEL)"
                  --> stream_id = 1

nekohasekai commented 2 years ago

https://github.com/SagerNet/cronet-go/blob/main/example/main.go

In these two examples, the first one fails when requesting plain http, but the second one succeed, is this related to thread safe?

nekohasekai commented 2 years ago

No matter if the connection is successful or not, it always ends with on failed -9, never on succeed, maybe it's not the net error unexpected?

klzgrad commented 2 years ago

The error is returned from here https://source.chromium.org/chromium/chromium/src/+/main:components/grpc_support/bidirectional_stream.cc;l=291

It means bidirectional_stream_read() is called when there is no data ready for read.

Read API doc:

/* Reads response data into |buffer| of |capacity| length. Must only be called
 * at most once in response to each invocation of the
 * on_stream_ready()/on_response_headers_received() and on_read_completed()
 * methods of the bidirectional_stream_callback.
 * Each call will result in an invocation of the callback's
 * on_read_completed() method if data is read, or its on_failed() method if
 * there's an error. The callback's on_succeeded() method is also invoked if
 * there is no more data to read and |end_of_stream| was previously sent.
 */
GRPC_SUPPORT_EXPORT
int bidirectional_stream_read(bidirectional_stream* stream,
                              char* buffer,
                              int capacity);

Make do with the build for now. I'll create debug builds for easier debugging in the future.

nekohasekai commented 2 years ago

Everything works now

https://github.com/SagerNet/cronet-go/commit/1826de40472374be74bfe0e3ea92e4a3b7a0183e

nekohasekai commented 2 years ago

It seems that there are no exported symbols in android's so library (the file size is also small)

Maybe by:

klzgrad commented 2 years ago

I noted this issue in https://github.com/klzgrad/naiveproxy/issues/282#issuecomment-1114805925.

klzgrad commented 2 years ago

Only bidirectional_stream.go needs LDFLAGS with libcronet. Others don't need it, or they are just adding duplicate ldflags to linker invocation.

Static linking:

bidirectional_stream.go:
// #cgo LDFLAGS: -fuse-ld=lld -Wl,--as-needed libcronet_static.a -ldl -lpthread -lrt -lresolv -latomic -lm
sudo apt install clang-15 lld-15
CGO_LDFLAGS_ALLOW="-fuse-ld=lld" CC="clang-15" go build example/main.go

This produces working but less optimal binaries than libcronet.so (optimized with LTO and other stuff). The optimal ldflags are complicated and platform-dependent. But this produces a single binary compared to two binaries with libcronet.so (LD_LIBRARY_PATH=$PWD can be saved with -Wl,-rpath,$ORIGIN).

nekohasekai commented 2 years ago

I made a simple packaging tool to create (fake) statically linked binaries. It's basically tar zstd and then unpack once at runtime and execvp: https://github.com/SagerNet/sing/blob/main/cli/libpack/main_linux.go

$ go build -v -o cronet-example ./example
github.com/sagernet/cronet-go/example

$ libpack -i cronet-example
INFO[0000] [libpack]: >> libpthread.so.0                
INFO[0000] [libpack]: >> libm.so.6                      
INFO[0000] [libpack]: >> libgcc_s.so.1                  
INFO[0000] [libpack]: >> libcronet.so                   
INFO[0001] [libpack]: >> libc.so.6                      
INFO[0001] [libpack]: >> cronet-example                 
INFO[0003] >> /home/sekai/Projects/cronet-go/cronet-example 

$ file  cronet-example
cronet-example: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

$ ./cronet-example
INFO[0000] [example]: libcronet 101.0.4951.41           
FATA[0000] [example]: missing args

nekohasekai commented 2 years ago

Static build works: https://github.com/SagerNet/cronet-go/commit/d3387b5665e31ad95c24eca48d3c2f9a8477dfc9

Edit: It's still dynamically linked unless -static is added in ldflags (reports -latomic not found, but works fine after removing it)

klzgrad commented 2 years ago

Would not recommend binary packing like UPX. It's reinventing something at least, in the linker infra, and may have problems with security measures against self modifying binaries. It could work, but I doubt the aesthetic is worth the effort.

Static linking of libcronet but not the rest of the common libraries. E.g. I don't think you can not link with libandroid.so on Android. But as long as the common libraries have no version and availability issues, they can be depended upon with dynamic linking.

nekohasekai commented 2 years ago

Static linking of libcronet but not the rest of the common libraries

As long as the dynamic linking to cronet is successful, only need to package libcronet.so into apk/lib/arch.

~~By the way, android doesn't have pthread.~~ Included in libc.

As well gomobile has some problems, it's old and the android part doesn't seem to be maintained anymore.

nekohasekai commented 2 years ago

The version of aarch64-linux-android21-clang in NDK is 12.0.8 and I don't know how to do static linking

nekohasekai commented 2 years ago

/* Returns true if the |stream| was successfully started and is now done
 * (succeeded, canceled, or failed).
 * Returns false if the |stream| stream is not yet started or is in progress.
 */
GRPC_SUPPORT_EXPORT
bool bidirectional_stream_is_done(bidirectional_stream* stream);

ld reports bidirectional_stream_is_done not found

klzgrad commented 2 years ago

bidirectional_stream_is_done was never implemented https://github.com/chromium/chromium/commit/06eab3e8e64eacef30f15299ee7c3ad32df2840b#diff-9bb8e6ea79473fee571b5eb74e0959307330d19e9eb9f5a36cf05a6736e5291bR197

It was derived from isDone() in components/cronet/android/java/src/org/chromium/net/impl/CronetBidirectionalStream.java.

CronetBidirectionalStream.isDone() is only used in test cases, so not super useful. I think it can be ignored.

klzgrad commented 2 years ago

I don't know how it's used to build Android apps. If the end product is something like libgojni.so in an apk, then using libcronet.so is quite appropriate. Static linking producing single binaries is for releases like this https://github.com/v2fly/v2ray-core/releases.

nekohasekai commented 2 years ago

Generating a release like v2ray requires disabling CGO, which is not possible with cronet.

Maybe need another get-clang.sh

klzgrad commented 2 years ago

I don't know much about v2ray build, but I don't see an inherent reason or difficulty why CGO is required to be disabled. According to this guy, https://github.com/v2ray/discussion/issues/754#issuecomment-650682512 CGO makes cross compile difficult. I don't see any specifics of where it is difficult, but there is definitely something. Then it has to showcase a Go app where cross compile works with Cronet CGO.

klzgrad commented 2 years ago

Ok, how about this: Use Cronet to recreate naiveproxy in pure Go (except libcronet.so) and see how many platforms Go can build for. As I summarized in "Recap of NaiveProxy" above, I suspect the actual logic of naiveproxy can be expressed in 100-200 lines of code in Go (besides protocol libraries). This should happen in cronet-go repo to see what would happen if a third-party developer tries to use cronet-go for cross compile.

nekohasekai commented 2 years ago

see how many platforms Go can build for

The answer is only host arch can build, we need cross compiler for each target (and sysroot)

https://github.com/SagerNet/cronet-go/commit/d46f60149f67ab9e50a60d88556db20dbc9c12f0

nekohasekai commented 2 years ago

How to set insecure-concurrency from cronet?

Riatre commented 2 years ago

I don't see any specifics of where it is difficult, but there is definitely something.

Drive by comment on "the specifics":

CGO has to be used together with Go's own build system, which in turn calls $CC and $LD to build C codes. It also does some insane things (parsing gcc stderr to do compile-time introspection on C struct/union/enum). Therefore, a typical CGO_ENABLED=1 go build kinda requires a cross compiler to be presented. And... people are hesitated to pull in a cross compiler for whatever reason.

In this case, since libcronet has to be built with another build system, as long as we ship one prebuilt .a per arch, most of the above does not directly apply (it should be possible to convince cgo to never call $CC in our package, and lld is mostly a hassle free cross-linker), HOWEVER:

In Go's build system, it is possible to detect whether cgo is enabled and do different things. Some popular packages include a "faster" variant with C dependency, and use a pure Go fallback in case cgo is disabled, but break if cgo is enabled but broken (e.g. without a working cross compiler). This could happen in transitive dependencies.
We need the libc.so/libpthread.so/... or at least their dynsym only stub of the target platform to -lc -lpthread -latomic <...>. This is okay for Android (just download NDK), painful for *-linux-gnu, and may drive people nuts for OpenWRT.

We could try to bundle required dynsym stubs along with prebuilt .a-s, similarly to what Zig does, but it is definitely not trivial.

klzgrad commented 2 years ago

it should be possible to convince cgo to never call $CC

What does this mean? There are some struct definitions in bidirectional_stream_c.h, which cronet-go imports. CC would be needed to get its layout, no?

transitive dependencies

I see, the problem is not creating a perfect cross compile tool chain for building a binary, but instead creating a cross tool chain that can be transitively included with other packages and not conflict with other CGO packages with possibly their own cross tool chain?! It seems cross toolchains don't really coexist with each other, and there has to be one single universal tool chain that handles everything?

painful for *-linux-gnu, and may drive people nuts for OpenWRT

This part is a solved problem in naiveproxy. The solution has three parts: a cross compiler (a single clang binary), sysroots creator, the flags to build it. There is some work to put the sysroots in better packaging so easier to use, and extract build flags from chromium for external use.

Riatre commented 2 years ago

CC would be needed to get its layout, no?

Yes. I mean it is possible if we come up with a struct-free FFI. I agree it would be horrible.

the problem is not creating a perfect cross compile tool chain for building a binary

The problem is Go developers prefer CGO_ENABLED=0 over installing and configuring a cross toolchain. Even if you make a perfect toolchain, you still need to convince those developers that "v2ray has to be built with CGO_ENABLED=0" is a myth and if they do A then B then C it could work.

Remember that Go is its own build system, in this build system, cross toolchains are configured by setting environment variables before invoking go build . It could not be bundled in a Go package, so there is no way to magically make our toolchain available.

And yes, the one toolchain has to be able to build all CGO dependencies of a project.

Actually, in 2022 we do have something close to "the perfect cross toolchain" (with a few rough edges), and people do try to use it with cgo: https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho. Maybe we could promote that.

nekohasekai commented 2 years ago

Even if you make a perfect toolchain, you still need to convince those developers

I'm just talking about the fact that CGO-enabled projects can't be directly cross-compiled successfully to that many targets, but no myth.

There are already projects like xgo that create cross-compilation environments with gcc to help with compilation, but not perfect.

the perfect cross toolchain

https://ziglang.org/learn/overview/#zig-ships-with-libc

It only comes with musl for linux, glibc for linux and windows, apparently not working out of the box for openwrt/darwin/bsd/...

By the way, currently static linking libcronet depends on llvm-15 which has not been stably released yet

klzgrad commented 2 years ago

Issues related to cross compiling cronet-go should continue in https://github.com/SagerNet/cronet-go/issues/1. This issue is used for tracking the C part of libcronet. Cronet-go is the Go part.

In short it's doable because it's only adapting existing cross tool chain in naiveproxy to cgo. There will be a showcase showing how it's done.

nekohasekai commented 2 years ago

The existing api only provides switches for http2 and quic, while http1 is always enabled. I think we need a mandatory setting, otherwise quic naive client will not work as expected.

klzgrad commented 2 years ago

http1 is always enabled

It's not? Bidirectional stream doesn't allow http endpoint. Didn't understand your issue. Please rephrase.

nekohasekai commented 2 years ago

Sorry, this came up on url request, it doesn't appear to be the problem. (quic seems to almost never be used when using url requests.)

klzgrad commented 2 years ago

https://github.com/klzgrad/naiveproxy/releases/tag/cronet-test6

I admit, CGO on Windows is hard. But it's done.

The hardest part is this:

Chromium uses clang-cl.exe, but CGO officially only supports GCC/MinGW on Windows. See https://github.com/golang/go/issues/17014.

CGO hardcodes GCC options incompatible with clang-cl, so an extra clang.exe is required (Chromium only provides clang-cl.exe).
We need CGO to link LLVM bitcode from Chromium, so ld cannot work and lld is required.
CGO passes GCC options incompatible with lld, so an extra lld wrapper is required to remove those options.
I didn't figure out a way to make the whole pipeline use lld-link-wrapper cleanly:
- -fuse-ld=lld --ld-path=lld-link-wrapper reports --ld-path is an unknown argument.
- -fuse-ld=.../lld-link-wrapper.exe creates garbled linker path. So uses a hack to rename lld-link.exe to lld-link-old.exe which is called from the wrapper "lld-link.exe".
lld-13 does not work with bitcode produced by Chromium's lld-15. So copies clang-13 from environment to Chromium's LLVM bin directory, and uses clang-13 together with lld-15. cgo_ldflags="-fuse-ld=lld $cgo_ldflags"

Tested build mode: dynamic executable linking with dynamic cronet library, dynamic executable linking with static cronet library.

Not tested build mode: dynamic library linking with static cronet library, static executable linking with static cronet library.

Known deficiencies: PIE executables on x86 Ubuntu and OpenWrt (except Android) segfault, so PIE is disabled on those platforms. I still haven't found out why.

klzgrad / naiveproxy

Project: Support Cronet Go binding #282