Hermes debugger is unavailable on all of our devices and machines

Nantris commented 2 years ago

I'll start by saying I don't believe this is related to expo-community-flipper as it began affecting builds that previously worked, on Flipper versions that previously worked - but it affects all of our machines, (iOS and Android, Linux and macOS,) and previous git commits, previous builds, Expo 45, Expo 46, previous Flipper versions - nothing fixes the issue - so I really don't know what to guess at as far as causes.

I've commented on a relevant issue on the Flipper repo, but it seems like we're out of luck if it comes to hoping on Facebook for a solution. Is anyone else experiencing this?

https://github.com/facebook/flipper/issues/3983

Edit: I noticed the ReadMe doesn't include SDK 46 on the list of verified versions. I'm guessing that's just an oversight though?

jakobo commented 2 years ago

I noticed the ReadMe doesn't include SDK 46 on the list of verified versions. I'm guessing that's just an oversight though?

Yeah 😅 I haven't found a great way to keep the table up to date w/ expo releases. The latest release 46.0.0 should work with SDK 46, and if not let's work on debugging it.

You mentioned this is happening on Android as well, ruling out the usual Podfile shenanigans. The number one thing I'd do is make sure the flipper app's version matches your package.json version. Unfortunately, lots of flipper minor semver releases are backwards incompatible (pre 1.0) and it's easy to get bitten by this.

Nantris commented 2 years ago

Thanks so much for your reply @jakobo!

The latest release 46.0.0 should work with SDK 46, and if not let's work on debugging it.

I'm sure it does work. I can't get SDK 45 to work either, but only for Hermes, so I can't imagine it's this plugin - which always worked great for us!

Unfortunately the react-native-flipper version matches the Flipper desktop app version, but no luck with Hermes to be found anywhere. I just keep hoping some new Flipper version will fix things, but it never does. The fact it affects across machines and Flipper versions and commits is baffling to say the least.

jakobo commented 2 years ago

Yeah it does seem strange. Is your host a Windows or a Mac? There's some networking annoyances when it comes to Windows + Hermes.

There's also a PR at https://github.com/jakobo/expo-community-flipper/pull/20 if you want to try this changeset. There were some changes in 0.69.2 that we didn't catch; though they're mostly iOS related.

Nantris commented 2 years ago

Unfortunately it's affecting both Linux (connecting to Android) and macOS (connecting to iOS.) I never got Hermes debugger working on Windows as seems to be standard experience.

But the issue affects SDK 45 also, so I really doubt it's related to anything on the Expo-Community-Flipper side. I just don't know where else to ask or what else to try besides accepting that debugging is going to have to take place via log statements only from now on, which I really can't accept =/

There's some networking annoyances when it comes to Windows + Hermes.

Can you point me to some further reading, or possibly provide a very brief overview if you don't know of any further reading? I'm really grasping at straws here and wondering if perhaps there could be some networking issue causing this even though none of my affected machines are using Windows.

Other possibly related issues (but probably not actually related - I'm grasping at straws):

https://github.com/expo/expo/issues/18542

jakobo commented 2 years ago

Totally. Here's the kinda short version of the horror in windows networking as a baseline, and then I'll follow up on some non-networking stuff.

Under WSL

The fundamental problem with WSL is that the Windows Subsystem for Linux is set up as a NAT, not as a Bridge. This means that your Ubuntu under windows is given an IP with its gateway being the host system. For example, when I run ip a on WSL Ubuntu I get the following for my machine:

5: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:08:c0:2e brd ff:ff:ff:ff:ff:ff
    inet 172.30.57.145/20 brd 172.30.63.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe08:c02e/64 scope link
       valid_lft forever preferred_lft forever

The 172.x is a Class B private address, similar to how most home networks have a 192.168.x.y (Class C) private address. Unfortunately, this comes to a head when you launch a server such as Metro via Expo and then ask your Android Simulator to connect to localhost. Windows doesn't think to forward localhost to the WSL instance, because (rightly so) that's not local at all as far as IP routing is concerned. There's a lot of IP Table / Forwarding workarounds, but the most reliable is a utility called WSLHostPatcher which uses the Windows Detours framework to let WSL listen on Windows' 0.0.0.0 broadcast interface. Doing this lets localhost requests find the Metro server inside of WSL and everything works wonderfully.

To my knowledge, things with Hermes + Android + WSL2 are working for users, as well as on-device builds against WSL using the expo dev client.

Under Windows Native

Windows Native is currently hell (see #4). There's several upstream issues, and the majority of them are related to use the Hermes runtime which is either (A) choking on a websocket over IPv6, (B) can't find the devtoolsFrontendUrl inside of localhost:8081/json for whatever reason, or (C) a completely unknown issue that's related to the Flipper app itself when running on Windows.

The Windows Native one has me absolutely stumped; my bet is that it's something in the Flipper Windows app related to networking & localhost. It's only a partially educated guess though given the sheer number of things that go wrong with Windows Networking. I don't know if Windows Flipper uses Electron, but using electon's windows networking woes as a baseline definitely feels like it's because Windows networking doesn't act like anyone else's networking.

I'm not sure it's Windows fault either. We have:

A device connecting to Windows forwarding requests to Metro (possibly inside of WSL)
A Flipper app on Windows connecting to Metro (possibly on localhost, or possible inside of WSL)
That same Flipper app is also connected to the device because breakpoints

It's just a lot of moving parts. The RN's work of moving Flipper into React-Native & Metro will reduce some of this pain, but we're probably a few releases off before we see those benefits.

Trying to Fix Hermes and Get Debugging Back

Moving on to the Hermes debugger up-and-leaving I'd clear the Metro cache first. Going nuclear on the caches will make for a slow start, but this is kind of the power it off/on approach. There's a set of commands in the Expo docs that will get rid of the caches.

The next thing that I would try is checking the metro endpoint for Hermes. You should be able to load http://localhost:8081/json once your app is started and see the device. If the device is there, great it's the Flipper app. If we get back an empty array [] then less-great, it's anything from Hermes not enabling its debugger to Metro not telling anyone there's a Hermes client connected. In both of these cases, they're likely upstream issues in Flipper.

Finally, and this will hurt, try turning of Hermes. Hermes and JSC use completely different code pathways, and it could legitimately be something about Hermes that simply doesn't exist in JSC land. I'm not a fan of JSC in development in Hermes in production, but gosh-darn debugging via log statements is awful.

So much for kinda short. That's everything I've got on the pain of Hermes and Networking to date.

Nantris commented 2 years ago

@jakobo wow thank you so much for your detailed reply! That was too much to absorb in one reading, but I'll probably read over this a few more times as I continue the insane troubleshooting saga.

I don't know if Windows Flipper uses Electron

It does indeed use Electron.

I wonder if disabling IPv6 might resolve my issue. That will be the next thing I try.

There's a set of commands in the Expo docs that will get rid of the caches.

Unfortunately the outcome is the same.

The next thing that I would try is checking the metro endpoint for Hermes. You should be able to load http://localhost:8081/json once your app is started and see the device.

Is the default port 8081? We don't seem to have anything running on there. I did find the JSON on port 19000 though.

I wonder if you see anything obviously wrong here?

[
  {
    "id": "2-3",
    "description": "devclient.app.mobile",
    "title": "Hermes React Native",
    "faviconUrl": "https://reactjs.org/favicon.ico",
    "devtoolsFrontendUrl": "devtools://devtools/bundled/js_app.html?experiments=true&v8only=true&ws=%5B%3A%3A1%5D%3A19000%2Finspector%2Fdebug%3Fdevice%3D2%26page%3D3",
    "type": "node",
    "webSocketDebuggerUrl": "ws://[::1]:19000/inspector/debug?device=2&page=3",
    "vm": "Hermes"
  },
  {
    "id": "2--1",
    "description": "devclient.app.mobile",
    "title": "React Native Experimental (Improved Chrome Reloads)",
    "faviconUrl": "https://reactjs.org/favicon.ico",
    "devtoolsFrontendUrl": "devtools://devtools/bundled/js_app.html?experiments=true&v8only=true&ws=%5B%3A%3A1%5D%3A19000%2Finspector%2Fdebug%3Fdevice%3D2%26page%3D-1",
    "type": "node",
    "webSocketDebuggerUrl": "ws://[::1]:19000/inspector/debug?device=2&page=-1",
    "vm": "don't use"
  },
  {
    "id": "3-3",
    "description": "devclient.app.mobile",
    "title": "Hermes React Native",
    "faviconUrl": "https://reactjs.org/favicon.ico",
    "devtoolsFrontendUrl": "devtools://devtools/bundled/js_app.html?experiments=true&v8only=true&ws=%5B%3A%3A1%5D%3A19000%2Finspector%2Fdebug%3Fdevice%3D3%26page%3D3",
    "type": "node",
    "webSocketDebuggerUrl": "ws://[::1]:19000/inspector/debug?device=3&page=3",
    "vm": "Hermes"
  },
  {
    "id": "3--1",
    "description": "devclient.app.mobile",
    "title": "React Native Experimental (Improved Chrome Reloads)",
    "faviconUrl": "https://reactjs.org/favicon.ico",
    "devtoolsFrontendUrl": "devtools://devtools/bundled/js_app.html?experiments=true&v8only=true&ws=%5B%3A%3A1%5D%3A19000%2Finspector%2Fdebug%3Fdevice%3D3%26page%3D-1",
    "type": "node",
    "webSocketDebuggerUrl": "ws://[::1]:19000/inspector/debug?device=3&page=-1",
    "vm": "don't use"
  }
]

Limited section of Flipper's startup logs

[conn] secure server listening at port:  
8088
[conn] secure server (ws) listening at port:  
9088
[conn] insecure server listening at port:  
8089
[conn] insecure server (ws) listening at port:  
9089
[conn] Browser server (ws) listening at port:  
8333
[conn] Flipper server state -> started 
AndroidLogListener.start -> success 
AndroidCrashWatcher.start -> success 
[conn] Secure websocket connection attempt: app-dev-client on emulator-5554. Medium 1. CSR: /data/user/0/devclient.app.mobile/files/sonar/

And adb reverse --list makes me think that the ports are being connected

❯ adb reverse --list                                                                                    
host-19 tcp:8088 tcp:8088
host-19 tcp:8089 tcp:8089
host-19 tcp:9088 tcp:9088
host-19 tcp:9089 tcp:9089

Finally, and this will hurt, try turning of Hermes. Hermes and JSC use completely different code pathways, and it could legitimately be something about Hermes that simply doesn't exist in JSC land. I'm not a fan of JSC in development in Hermes in production, but gosh-darn debugging via log statements is awful.

I would happily move to JSC at this point, but unfortunately we can't, because React Navigation now uses React Reanimated 2 which requires Hermes if you want to debug it. It's the reason we moved to Hermes to begin with - which means that truly we are out of luck unless Facebook staff decides to care - basically a complete halt to our mobile development work.

Nantris commented 2 years ago

Finally got Hermes Debugger to stop saying not supported! Because it mysteriously began needing METRO_SERVER_PORT defined like env METRO_SERVER_PORT=19000 ./flipper - I'd never have figured that out without your comment, and I am eternally grateful!

I mean, out of nowhere this happened, on all machines, on all Flipper versions. Wtf? Top 5 strangest issues I've seen in all my years. And it seems to affect no one else?

But, of course that only got us to a new error:

"Metro is connected but no Hermes apps were found."

beating-head-against-a-pole-looney-tunes

But at least I have hope again!

jakobo commented 2 years ago

https://fbflipper.com/docs/custom-ports/

As I was reading, I was about to share that link until I saw the metro port issue got resolved. The good news is, we're moving into purely Flipper related issues now. It's building, the port's being found, it's progress! 🎉 Now we just need to figure out how to tell Hermes where to look.

Try forwarding 8097 (hermes) in case the build is still using the old hermes port
Run through the troubleshooting
Open up a bug in the flipper repo and follow their bug template. This now smells like the websocket ipv6 woes

Nantris commented 2 years ago

Got Hermes working! My final challenge now seems to be getting it to work in a way that allows me to run it from a desktop icon, but that's small potatoes!

Seriously I'd have spent at least another full day on this without your help, I am so grateful!

Basically the fix:

~~Disabled IPv6 on my machine (not sure if this was important, but it reduces the entries in localhost:19000/json from 4 to 2.)~~ Don't do this - it breaks the emulator, but only when you restart it
env METRO_SERVER_PORT=19000 ./flipper

I can't imagine why running the same command from a desktop shortcut (KDE) would get Hermes to stop saying unsupported, but not to actually connect. Flipper is f***ing weird!

Nantris commented 2 years ago

Fully functional shortcut for KDE users.

Note the "work path" must be set for Hermes to actually connect!

Also note that the app must be reloaded once from in-app using Ctrl+M (for Android) and then choosing "Reload." After that, you should have Hermes connected and all of the source files should be accessible in the debugger.

jakobo commented 2 years ago

Awesome. Would you mind if I pinned this? I'm sure others will run into the same issues trying to get Hermes working

Nantris commented 2 years ago

Sure thing; spread the wealth!

And once more, thank you!

Nantris commented 2 years ago

Update: Don't disable IPv6.

Android Emulator fails to start if it can't connect to Android Studio via IPv6, and it turns out not to be important anyway.

Nantris commented 2 years ago

For macOS you can run the following, once Flipper.app is installed:

METRO_SERVER_PORT=19000 open -a Flipper

If anyone knows how to create a shortcut, or even better to add this to dock, I'd be eager to learn!

jakobo / expo-community-flipper