Any plan of Android support ?

NghiaTranUIT commented 3 years ago

I actually dig around Android world to find the equivalent function of Method Swizzling, but I couldn't find any useful.

It seems that Kotlin doesn't support it, and Java have Reflection, but it doesn't seem what I'm looking for.

Do you have any suggestion or code example 🤔 @jkyeo

djbe commented 3 years ago

Very curious about this as well (just in general, for all my Android colleagues).

From some quick searching, these seem to be the current solutions:

Android Studio has a built-in Network Profiler
There are libraries for in-app network debugging such as Chuck, but they require the user making all calls via specially configured OkHttp clients (by adding an interceptor).
Then there are Android tools such as HttpCanary which just do the VPN (or proxy) dance to intercept all traffic (like Charles does).

For something like Atlantis or Bagel to work, the libraries need to be able to monitor all calls made by the system (in the app), either by using some built-in mechanism, or by swizzling (replacing the system implementations).

A possible solution may be option 2, by having the developer add interceptors to each "client" they want to monitor, something like Atlantis.registerClient(...).

NghiaTranUIT commented 3 years ago

Thanks for the detailed hint @djbe. I will continue researching and soon support Atlantis on Android 👍

aminerol commented 3 years ago

in Android, they have what they call dynamic instrumentation where you can set up hooks on methods. not sure if this is what it's needed or the equivalent for Method Swizzling in android but there are many libraries abstracting this concept such as AndHook

djbe commented 3 years ago

Right, so a colleague of mine made an Android implementation for Bagel, credits to @dieter115!

Right now the code is embedded in an internal library we use in our Android projects. It consists of:

Interceptor and listener autodiscovery, code can be found here. It's very similar to the iOS code.
The interceptor is added to the client(s) the developer wants to monitor: interceptor creation and registration
The service discovery is started on app onCreate() using BagelNetworkDiscoveryManager.registerService(context), the equivalent of Atlantis.start()

Now we'd love to switch to Proxyman for our Android devs (using Atlantis). We can probably tweak our implementation to match the Atlantis protocol, or would you rather do this yourself?

NghiaTranUIT commented 3 years ago

@djbe It sounds good. I'm not familiar with Android development, so it might take more time to do it by myself 😸

If you don't mind, you can do and open the PR. I'm happy to help you with the integration 😄

djbe commented 3 years ago

Is there a high level documentation on how Atlantis works, and maybe where it differs from Bagel? I noticed Atlantis for example compresses all sent data using gzip

NghiaTranUIT commented 3 years ago

@djbe Unfortunately, there is no high-level documentation, but I can quickly describe here:

As soon as you launch Atlantis, Atlantis will do "Method Swizzling" for all NSURLSession methods: It's the same logic with Bagel, but in Swift. Code: https://github.com/ProxymanApp/atlantis/blob/main/Sources/NetworkInjector.swift
When Atlantis receives the request/response. It will construct the message (https://github.com/ProxymanApp/atlantis/blob/89702b9586fcbed2e99a5e896821461bc700272c/Sources/Atlantis.swift#L298)
Then, sending it to macOS via Bonjour Service (https://github.com/ProxymanApp/atlantis/blob/89702b9586fcbed2e99a5e896821461bc700272c/Sources/Transporter.swift#L104)
gzip is optioned, but recommend. Proxyman for macOS will check if it's a gzip or not.

If you're going to implement the Android, you can implement your own "Method Swizzling" in Android, then construct a message with the given Request/Response and pass it to Atlantis with this func (https://github.com/ProxymanApp/atlantis#1-my-app-uses-c-network-library-and-doesnt-use-urlsession-nsurlsession-or-any-ios-networking-library)

In this way, Atlantis will handle the rest (Construct the message and send it to Proxyman for macOS) 👍

dieter115 commented 3 years ago

Hey Guys I have been working on a POC for Android. And I made it possible to connect my application (Android phone) to a Proxyman socket on my macbook. I wrote your message ,device , project , ... objects in kotlin and I am trying to send this data gzipped json or normal json over this socket. I used Wireshark to check the packages and connection and they were all sent to the right socket. They don't seem to appear in the Proxyman app ... Is there a format the message and / or it's fields need to have to be recognized by Proxyman.

I tried it with escape slashes

{"buildVersion":"1.0-dev","content":"{\"device\":{\"model\":\"OnePlus,IN2023; Android/30\",\"name\":\"OnePlus 8 Pro\"},\"project\":{\"bundleIdentifier\":\"be.dietervaesen.bageltester.dev\",\"name\":\"ProxyManPOC\"}}","id":"be.dietervaesen.bageltester.dev-OnePlus,IN2023; Android/30","messageType":"connection"}

and without

{"buildVersion":"1.0-dev","content":{"device":{"model":"OnePlus,IN2023; Android/30","name":"OnePlus 8 Pro"},"project":{"bundleIdentifier":"be.dietervaesen.bageltester.dev","name":"ProxyManPOC"}},"id":"be.dietervaesen.bageltester.dev-OnePlus,IN2023; Android/30","messageType":"connection"}

NghiaTranUIT commented 3 years ago

Hey @dieter115, If you connect to Proxyman Socket at localhost:9090, and send the Atlantis data, it won't work.

You might have someway to consume Bonjour Service from Proxyman app. From what I googled, there is Network Service Discovery from Android, which is allowed you to discover Proxyman and send the data to. Ref: https://jaanus.com/implementing-bonjour-across-ios-and-android/

Here is the Bonjour Service from Proxyman app

        static let netServiceDomain = ""
        static let netServiceType = "_Proxyman._tcp"
        static let netServiceName = "Proxyman"
        static let netServicePort: Int32 = 10909

let uniqueServiceName = "\(netServiceName)-\(UUID().uuidString)"
let netService = NetService(domain: netServiceDomain, type: netServiceType, name: uniqueServiceName, port: netServicePort)

From your JSON, the second one (without escape) looks correct. Basically, it's a JSON, we don't need to escape it.

However, in order to send the message to Proxyman, you might construct the message like Atlantis does. Code: https://github.com/ProxymanApp/atlantis/blob/87107de8c2881dd86b6b53da6519a93048f40ed3/Sources/Transporter.swift#L115-L118

The message will contain two parts:

Reserve xx bytes (Int(MemoryLayout<UInt64>.stride)) and fill it will the length of the JSON message
The actual JSON message in byte.

The reason is that Proxyman doesn't know when the message is completed (The message can be spliced into small chunks when sending in TCP layer). Thus, the first part is important to know where the JSON Message is done.

dieter115 commented 3 years ago

hey @NghiaTranUIT I'm indeed connected with the service with port 10909 and I found it through Network Service Discovery. I already used this part in my Bagel POC. I create a socket with its IP and port and send packages through it. I'm already sending the length of the json data and then the json itself each as its own package/message. But what you mean is that you send 1 message with length of data and the date itself combined as 1 package?

NghiaTranUIT commented 3 years ago

@dieter115 you can see the code how the message is constructed https://github.com/ProxymanApp/atlantis/blob/87107de8c2881dd86b6b53da6519a93048f40ed3/Sources/Transporter.swift#L115-L118

Yes, it’s one message that consists of two parts.

NghiaTranUIT commented 3 years ago

I think that you can roughly translate to Kotlin. Please let me know if you need some help 😊

dieter115 commented 3 years ago

Hey @NghiaTranUIT I reworked my code so it sends 1 message instead of 2 different messages. I used this code :

Sending size of message as a byte array

val outPutStream = socket.getOutputStream()
                    val messageByteArray =Gson().toJson(proxymanRequestMessage).toByteArray()

                    val output = ByteArrayOutputStream()
                    output.write(messageByteArray.size.toLong().toByteArray())
                    output.write(messageByteArray)
                    outPutStream.write(output.toByteArray())

This sends a message that looks like this I think this is also how Atlantis does it ? But I also tried other formats for the size to see if they would work.

Sending size of message as a string or int

...
output.write(messageByteArray.size.toString().toByteArray())
...

or

...
output.write(messageByteArray.size)
...

These seem to make Proxyman stop.

NghiaTranUIT commented 3 years ago

buffer.append(&lengthPackage, length: Int(MemoryLayout<UInt64>.stride))

@dieter115 It means the buffer will reserve 8 bytes to store the lengthPackage, which is in bytes too.

Here is how Proxyman app get the Atlantis message

First of all, Proxyman will read 8 bytes on the stream and parse it to UInt64 to know the message length

    let byLength = Int(MemoryLayout<UInt64>.stride)
    connection.receive(minimumIncompleteLength: byLength, maximumLength: byLength) {[weak self] (data, _, _, error) in
        guard let strongSelf = self else { return }

        if let data = data {
            // Cast byte to UInt64
            var length: UInt64 = 0
            _ = withUnsafeMutablePointer(to: &length) { (length) -> Int in
                data.copyBytes(to: UnsafeMutableBufferPointer(start: length, count: MemoryLayout<UInt64>.stride))
            }
...

After getting the length, Proxyman tries to read the message

connection.receive(minimumIncompleteLength: length, maximumLength: length)

From what I see in your code

output.write(messageByteArray.size.toString().toByteArray())
output.write(messageByteArray.size)

It looks like your write a length without reserving 8 bytes. and it must be an Integer or UInt64

So if you code is already working with Bagel, it should work with Proxyman app

dieter115 commented 3 years ago

@NghiaTranUIT The first approach I tried was using the same code as with Bagel. But here I send 2 messages over the socket 1 with length of message and 1 with the message data. It works with Bagel but Proxyman doesn't seem to show my sent data.. Like you can see in the picture of the message I sent.. The length of the message reservers / takes 8 bytes.

So like Bagel

val outPutStream = socket.getOutputStream()
                        val messageByteArray =Gson().toJson(proxymanRequestMessage).toByteArray()

                        //first sent length of message to bagel
                         outPutStream.write(messageByteArray.size.toLong().toByteArray())
                        //sent actual message as a bytearray
                        outPutStream.write(messageByteArray)

                        outPutStream.flush()

Force it as 1 message like Atlantis https://github.com/ProxymanApp/atlantis/issues/28#issuecomment-822344148

Or is it possible that the app (buildConnectionMessage) only shows up in Proxyman after you sent a Traffic message for that app? Is there a way to debug if Proxyman is receiving anything?

djbe commented 3 years ago

Quick update: @dieter115 and I were able to get a PoC working with Proxyman today, the issue had nothing to do with the message size / byte reservation.

After using Wireshark and temporarily disabling gzip compression in Atlantis, we found the issue: as a side effect of using Codable, it'll automatically base64 encode any Data property. This is something that Proxyman does that's quite different from Bagel. Besides that, we found some other small differences, but that was the main one (it applies to content, requestBody, responseBody, etc...).

We found another difference, where I have no idea why Proxyman changed this from Bagel:

Bagel sends a message at the start of a request (with only the request content), and a second message at the end of a request with the full info (request, response, etc...)
Proxyman only expects a message at the end of a request.

If we send traffic messages with the same ID to Proxyman like we did with Bagel, it'll appear twice in Proxyman (see screenshot). I've created a separate issue for this, as it would be a general improvement to handle this.

NghiaTranUIT commented 3 years ago

Thanks for your input @djbe

as a side effect of using Codable, it'll automatically base64 encode any Data property.

Yes, it's. Codable automatically encoded the raw data to base64. I intentionally use it for easier implementation.

Bagel sends a message at the start of a request (with only the request content), and a second message at the end of a request with the full info (request, response, etc...)

You're right. It's the main difference between Bagel and Proxyman.

Instead of sending two parts (Request and Response) of the 1 request, Atlantis accumulates it. and send ONCE when it's done. It increases the performance of your app has a large number of traffic that sending to Proxyman.

Here is the function that you can manually send a request and response to Proxyman: https://github.com/ProxymanApp/atlantis/blob/5ca79443ec6d68b6fa5b6b76b8ddf3070e69b45a/Sources/Atlantis%2BManual.swift#L18-L30

dieter115 commented 3 years ago

@NghiaTranUIT Me and my colleague David made it possible to send packages to Proxyman. But I still have a question.. I was making the feature where you can limit the Mac's you connect with. I copied this from my Bagel implementation but then I saw a difference. When I find bagel in the network discovery code it gives me a name like "macbook Dieter" but when i discover Proxyman services I get names like Proxyman-DA9E3C12-74DC-4865-9690-AA55FD351690 did you guys ever encounter something similar ?

djbe commented 3 years ago

Right so the difference with Bagel is that Bagel sets the service name to the hostname, whereas Proxyman sets the service name to some random identifier (like @dieter115 posted).

We've tried switching to using InetAddress' getHostName() and getCanonicalHostName(), but neither work well. It usually just returns the host's IP address instead of a nice network name. I'm not sure there's any solution to this, Android's reverse DNS lookup seems to be quite buggy.

NghiaTranUIT commented 3 years ago

Glad to know you finally send a request to Proxyman 🥇 @djbe @djbe 👍

You're right about the Proxyman Service Name, which is Proxyman-uuid. I intentionally added the suffix to prevent NSNetServicesCollisionError if we have multiple Proxyman apps in the same network.

Please try this beta build, I improve the Service Name to Proxyman-<host_name> as Bagel does. (I believe that hostname is unique for each machines in the same network, so there is no Collision Error anymore)

https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_2.26.0_Better_Bonjour_Service_Name.dmg

djbe commented 3 years ago

Quick update: tried that beta build, and got this as a service name:

Proxyman-ptr-4xa9d8d5k2butaudew8.18120a2.ip6.access.telenet.be

Whereas the laptop's name is MacBook-Pro-van-dieter-2, so no idea where it's getting that name. At first glance seems like a combination of random identifier, ipv6 address, and ISP (telenet).

NghiaTranUIT commented 3 years ago

@djbe can you clarify how you get this name? 🤔 It'd be great if we have a sample code that I can test out

I tested with Discovery app, which lists all available Bonjour services.

In the latest build, the hostname has a UUID as a suffix
In the beta build, it has a nicer name

dieter115 commented 3 years ago

hey @NghiaTranUIT it depends on the network it seems ! So you were right !! On the network in the office I get right hostname

at home I get this

djbe commented 3 years ago

What @dieter115 forgot to mention was that the device name / hostname / ... when going to "Certificates > iOS > Atlantis" is correct. Is Proxyman using different code to set that service name (in the beta build) compared to that view?

NghiaTranUIT commented 3 years ago

In the beta build, I use Host().current.name, which returns a Host/Computer name in the network.

It's the same result when executing the following code on the Terminal app

$ hostname
Nghias-Mac-mini.local

Ref: https://developer.apple.com/documentation/foundation/host/1416949-name

I suppose that the hostname might be changed when you're using a corporate network? 🤔

messi commented 2 years ago

I am excited to read that @dieter115 and @djbe found a way to bring Atlantis to Android. What's the progress on this work? Is there any plan to make this official? And if not, is there any way to profit of that work and get some code snippets? :)

sjmueller commented 2 years ago

We're also interested in Android support. What's really confusing is that the Proxyman repo README advertises Android, yet it is absent in Atlantis?

NghiaTranUIT commented 2 years ago

@sjmueller Atlants is written by Swift, so it only supports iOS, macOS, tvOS, watchOS.

Regarding the readme, it means Proxyman can capture from Android Devices (Without using the Atlantis Framework). Please check out this doc: https://docs.proxyman.io/debug-devices/android-device

For the Android Emulator, Proxyman also provides the automatic script to do the boring part for you (Override HTTP Proxy and install the certificate), ref: https://docs.proxyman.io/debug-devices/android-device/automatic-script-for-android-emulator

dieter115 commented 2 years ago

@messi Sorry for the late reply but we made it work and are using it in our Android projects. At the moment I haven't found a way to catch all the network traffic but I made something that works with Retrofit and Okhttp. It catches all the api calls and sends the requests/responses to Proxyman. If you want I can put in on a seperate repo? If thats ok for @NghiaTranUIT .

NghiaTranUIT commented 2 years ago

Yes, please @dieter115 🥇 I will test again and link to your repo, so Atlantis-Android can work with Proxyman 🙌

gtmn commented 1 year ago

@dieter115, I am eager to explore your Android solution. Are there still any plans for its publication? @NghiaTranUIT, would it be possible to have it in a public repository under the ProxymanApp organization for easy access?

NghiaTranUIT commented 1 year ago

@gtmn Atlantis is Open source, you can reimplement it in Android if it's possible.

I don't have plan to port to Android by myself, but I'm happy to accept the new PR or Repo.

gtmn commented 1 year ago

@NghiaTranUIT, I understand. I just wanted to suggest that it would be great if @dieter115 could contribute his implementation directly in a repo under the ProxymanApp organization. So a solution for Android could live next to the original Atlantis for iOS 😊

NghiaTranUIT commented 1 year ago

yes, it's possible. Please share with me the Repo URL 👍

dieter115 commented 1 year ago

Hey Guys , sorry for the late response. Have been busy for the last year... I would like to make it public ... At the moment it only works with Retrofit. I'd like to make some more time to tear it away from the repo where it is in at the moment

NghiaTranUIT commented 1 year ago

It's awesome new @dieter115 👍 You can publish it on your own Repo, then we can refer it 👍

dieter115 commented 1 year ago

@NghiaTranUIT I did the initial commit on this repo https://github.com/dieter115/proxyman-atlantis-android. At the moment it works with an interceptor you can set to Retrofit. Did you already check if it is possible to work with unique id's for the packages ? So we can first send all the request data for call and when it succeeds / fails we can send and update the response data?

ProxymanApp / atlantis

Any plan of Android support ? #28