Does this support inspecting Firestore grpc calls?

[ColtonIdle]

Sorry if this is the wrong place for this question but there isn't a "Discussions" tab on this repo.

I'm trying to inspect firebase firestore traffic which uses grpc. Would this be possible using Mediator?

[devkanro]

Mediator not support tls gRPC traffic now, I am refactoring the proxy make it support tls.

If the firebase firestore traffic use the gRPC with tls, Mediator can't work with it.

[ColtonIdle]

Okay. I will try with charles. Thank you!

[devkanro]

I am pleased to inform you that Mediator will support gRPC/TLS requests in the next release.

We are currently testing the feature, so if you are interested in it, please leave a comment here and I will provide you with a beta version.

[ColtonIdle]

Do you know if it works with Firestore? If so, I'd pay money for that. Let me know!!!!!

[devkanro]

@ColtonIdle

I don't know if Mediator works with Firestore, you can try it.

In theory, Mediator works with any gRPC service that supports service reflection.

Before applying Mediator to Firestore, you first need to confirm the following:

1. Does Firestore use gRPC as an API?
2. Does Firestore support service reflection?

We expect to add support for gRPC servers that do not support service reflection in the next version, but the premise is that you must have the proto source code of the API, or a binary description file.

[devkanro]

@ColtonIdle After some searching in github, the good news is firestore using gRPC.

You can found the protos here: https://github.com/googleapis/java-firestore/blob/main/proto-google-cloud-firestore-v1/src/main/proto/google/firestore/v1/firestore.proto

But we don't know does Firestore support service reflection? If firestore don't support it, you can wait our next release.

[devkanro]

@ColtonIdle Another good news here!

After 1.4.2 version, you can decode the gRPC message by proto source file.

You can download firestore proto files from here.

Configure the proto root in mediator server rule follow the document.

Then you can decode the firestore api, don't forgot install the Mediator Root Certificate.

Check the ReadMe for more information.

[ColtonIdle]

@devkanro thanks for the instructions. I gave this a try on my android device just now.

I set up my devices WiFi to have a manual proxy set to my machine and now I can see traffic on mediator. I tried to install the mediator root certificate, but the file doesn't download wehn i click the link on my android device.

[ColtonIdle]

Also please let me know if this is how I would setup the proto rule

[ColtonIdle]

please also let me know if theres a way to donate to the project. thanks

[devkanro]

I set up my devices WiFi to have a manual proxy set to my machine and now I can see traffic on mediator. I tried to install the mediator root certificate, but the file doesn't download when i click the link on my android device.

This seems odd because the Echo Service shows the address you accessed the certificate (/mediatorRoot.cer), and in theory when accessing the certificate path, the Echo Service will not be used.

Also please let me know if this is how I would setup the proto rule.

If you don't need rewrite the host, do not check the rewrite checkbox

Other settings are correct

[devkanro]

Can you download it from your PC or Mac? then transfer to your Android.

[ColtonIdle]

Okay. Downloading to my mac worked. Going to email the file to myself.

Accessing gmail on my android device with the proxy setup doesn't seem to work though which is weird. Ideas?

[devkanro]

Accessing gmail on my android device with the proxy setup doesn't seem to work though which is weird. Ideas?

Mediator only support the gRPC request in proxy, when you are in a non-gRPC scenario, the proxy should not be enabled.

[devkanro]

Another weird thing is that Echo Service doesn't print Request Host like GET http://192.168.1.1:8888/mediatorRoot.cer

I don't know why your android is showing like this.

[ColtonIdle]

yeah. thats what i expected. all non-frpc traffic just works. but it doesn't.

it looks like once i set my proxy up to my mac with mediator running, i just lose my internet. i disabled the proxy. installed the CA certificate and installed it on my device. connected back to the proxy and can see that traffic is showing in mediator

[devkanro]

Notice the path after GET string do not include the host.

[ColtonIdle]

if I visit http://192.168.1.168:8888/mediatorRoot on my mac I see

[ColtonIdle]

i do see traffic from my android device. let me know if you think i can debug this somehow. also. let me know if you have a donation/patreon/github sponsor setup. thank you. i really would like to help make firestore data inspection a reality.

[ColtonIdle]

I have used both charles proxy and proxyman in the past. I wonder if any of their port numbers are messing with mediator. i made sure that both charles and proxyman are shutdown though and that my android device is not pointed to either of those port numbers.

[devkanro]

yeah. thats what i expected. all non-frpc traffic just works. but it doesn't.

it looks like once i set my proxy up to my mac with mediator running, i just lose my internet. i disabled the proxy. installed the CA certificate and installed it on my device. connected back to the proxy and can see that traffic is showing in mediator

Ummm, maybe in the next version I can implement transparent transmission for non-grpc traffic.

[devkanro]

@ColtonIdle You should not set proxy in your wifi, the Java gRPC client proxy should be setting by those code:

ManagedChannelBuilder.forAddress("foo.barapis.com", 9000)
    .usePlaintext()
    .proxyDetector {
        HttpConnectProxiedSocketAddress.newBuilder()
            .setTargetAddress(it as InetSocketAddress)
            .setProxyAddress(InetSocketAddress("<YOUR PC/MAC IP>", 8888))
            .build()
    }
    .build()

[ColtonIdle]

Interesting. Typically I set the proxy for charles and proxyman in my wifi settings.

I will try including your snippet above in my android App class

[devkanro]

Interesting. Typically I set the proxy for charles and proxyman in my wifi settings.

I will try including your snippet above in my android App class

Yes, because most gRPC client will not use the system proxy setting, you need config it by your self.

[ColtonIdle]

ManagedChannelBuilder.forAddress("firestore.googleapis.com", 443)
  .usePlaintext()
  .proxyDetector {
    HttpConnectProxiedSocketAddress.newBuilder()
      .setTargetAddress(it as InetSocketAddress)
      .setProxyAddress(InetSocketAddress("192.168.1.168", 8889))
      .build()
  }
  .build()

(i changed mediator to use :8889)

I dont see any traffic in mediator now. I wonder if firestore.googleapis.com is not the right address

[devkanro]

I found the FirestoreClient can accept a FirestoreStub.

You can build the FirestoreStub with

val channel = ManagedChannelBuilder.forAddress("firestore.googleapis.com", 443)
  .uuseTransportSecurity()
  .proxyDetector {
    HttpConnectProxiedSocketAddress.newBuilder()
      .setTargetAddress(it as InetSocketAddress)
      .setProxyAddress(InetSocketAddress("192.168.1.168", 8889))
      .build()
  }
  .build()

val stub = FirestoreStub(channel)

val client = FirestoreClient(stub)

[ColtonIdle]

[devkanro]

Ummmm...repleace with

val stub = FirestoreGrpc.newStub(channel)

[devkanro]

And

val client = FirestoreClient.create(stub)

[ColtonIdle]

Getting closer!

[devkanro]

Ummm...it should be a create method here https://github.com/googleapis/java-firestore/blob/main/google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/FirestoreClient.java#L180

[ColtonIdle]

I think I technically use https://github.com/firebase/firebase-android-sdk but I thought it just deferred to java-firestore. hmm

[ColtonIdle]

https://github.com/firebase/firebase-android-sdk/blob/a50da9f8416ed35df3b1f80f05e5e91c440e6aa1/firebase-firestore/src/main/java/com/google/firebase/firestore/core/FirestoreClient.java

[devkanro]

It seem an wrapper with auth info for FirestoreClient in firestore-java

[devkanro]

Need to find some way to create FirestoreClient with our channel or stub

[ColtonIdle]

okay. i will keep looking. let me know if you find anything. so CLOSE!

[devkanro]

Try to see if there is a configuration for the proxy in FirebaseFirestoreSettings

[ColtonIdle]

It doesn't look like it unfortunately

[ColtonIdle]

Just 4 options available in firestore settings

[ColtonIdle]

Created a feature request on firebase android. maybe someone there can help.

[devkanro]

Create some breakpoints, to find how the firebase stub created in firebase-android-sdk

[ColtonIdle]

Yeah. I'm going to do that first thing tomorrow. about 4 hours past my bedtime. 😭 💤

[devkanro]

Hahaha I'm going to have some lunch too

[ColtonIdle]

looks like its created here. but yeah. thanks again for the help. i feel like im so close to having some way of inspecting firestore calls lol. its so hard not having ANY insight

[devkanro]

Maybe you can clone their code and change it from here https://github.com/firebase/firebase-android-sdk/blob/c35626a475cfcfc702586a0df09718b11a7427d3/firebase-firestore/src/main/java/com/google/firebase/firestore/remote/GrpcCallProvider.java#L109

Then compile one yourself

[ColtonIdle]

Thanks. I have never had to compile an external library like this myself... but hopefully I can make it work. I hope that firestore can also add this. We'll see how my feature request goes. 😄

[devkanro]

I built one for you. check it.

maven.zip

Unzip in your ~/.m2, merge the repository dir.

Add mavenLocal to your build.gradle repository.

Use the com.google.firebase:firebase-firestore:24.3.1-SNAPSHOT

You can found the FirebaseFirestoreSettings class have new property named 'proxy', set it to <YOU-PC-IP>:8888.

[devkanro]

The source code is here https://github.com/devkanro/firebase-android-sdk/commit/e9208e1961964da08a599f5318986072754356c6

[ColtonIdle]

Downloaded, placed in .m2, hit merge instead of replace. added mavenLocal() and added that dep SNAPSHOT you posted. Set the proxy on firestoreSettings... building and running now. 🤞

[ColtonIdle]

java.lang.IllegalStateException: The proxy address 192.168.1.168:8888 is not resolved