search

httptoolkit / httptoolkit

HTTP Toolkit is a beautiful & open-source tool for debugging, testing and building with HTTP(S) on Windows, Linux & Mac :tada: Open an issue here to give feedback or ask for help.
https://httptoolkit.com
2.71k stars 71 forks source link

[Bug]: Desktop app stuck on "This is taking longer than normal" #530

Open H-Maa opened 9 months ago

H-Maa commented 9 months ago

Has this been reported before?

Repro steps

Open desktop app, either new install or standalone (tried newest and 1.13.0)

How often does this bug happen?

Every time

The desktop OS you're using

Windows 10

Details of other apps/devices

No response

Error screenshot

No response

Any other info?

Error log:

--- Launching HTTP Toolkit desktop v1.14.7 ---
INFO: Initialising UI (version 504213ce08f02af2c7faec2e9681e7dbabf3874d)
INFO: Account store initialized
INFO: UI store initialized
INFO: Proxy settings loaded
INFO: API store initialized
Config checked in 15 ms
Certificates setup in 3 ms
Standalone server started in 4 ms
Server started in 9 ms
Total startup took 31 ms
INFO: Previous server version was 1.14.7
INFO: Reporting error: Error: Failed to initialize application [object Object]
INFO: Server initialization failed TypeError: Failed to fetch
INFO: Server initialization failed TypeError: Failed to fetch
INFO: Error: API error during TriggerUpdate: fetch failed with 'Failed to fetch'
pimterry commented 9 months ago

Hmm, OK. What's supposed to happen is:

The logs you've shared there show that the server thinks it's up and running (Total startup...) but the UI request to 45456 isn't working somehow... But with no obvious errors appearing on the server side, so it doesn't seem like the proxy itself is failing to start - it seems like the request is not being received at all.

The API error during TriggerUpdate also means that a separate server update request to 45457 failed as well, also with no details on the server side.

Is there anything unusual about this machine? Is it an enterprise managed device, or are you running in a particularly locked down account? If the app isn't able to listen or connect locally on those ports then this would happen.

To investigate further, it would be helpful if you could start up HTTP Toolkit to reproduce this, and then:

acedogblast commented 6 months ago

I have a very similar issue. I have cloned the UI repo, ran npm install then npm start. Instead of the desktop electron app I am using firefox to 127.0.0.1:8080. It shows the HTTP Toolkit loading screen but after a while gives me the "This is taking longer than normal" text. There is no error messages on the terminal that I have running npm start. This is all that is on the terminal:

> npm-run-all --parallel --print-label start:server start:web

[start:web   ] 
[start:web   ] > start:web
[start:web   ] > env-cmd -f ./automation/ts-node.env.js webpack-dev-server --config ./automation/webpack.dev.ts
[start:web   ] 
[start:server] 
[start:server] > start:server
[start:server] > npm-run-all server:setup server:start
[start:server] 
[start:server] 
[start:server] > server:setup
[start:server] > ts-node -P ./automation/tsconfig.json ./automation/setup-server.ts
[start:server] 
[start:server] Downloaded server already up to date.
[start:server] 
[start:server] > server:start
[start:server] > cross-env OCLIF_TS_NODE=0 node .httptoolkit-server/httptoolkit-server/bin/run start
[start:server] 
[start:web   ] <i> [webpack-dev-server] Project is running at:
[start:web   ] <i> [webpack-dev-server] Loopback: http://127.0.0.1:8080/
[start:web   ] <i> [webpack-dev-server] Content not from webpack is served from '/home/trc/httptoolkit-ui/public' directory
[start:web   ] <i> [webpack-dev-server] 404s will fallback to '/index.html'
[start:server] Config checked in 9 ms
[start:server] Certificates setup in 6 ms
[start:server] Standalone server started in 6 ms
[start:server] Server started in 18 ms
[start:server] Total startup took 39 ms
[start:web   ] webpack 5.89.0 compiled successfully in 10011 ms
[start:web   ] No issues found.

Running netstat -tulpn shows that the 3 ports are open and running:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      27700/webpack       
tcp        0      0 127.0.0.1:45457         0.0.0.0:*               LISTEN      27773/HTTP Toolkit  
tcp        0      0 127.0.0.1:45456         0.0.0.0:*               LISTEN      27773/HTTP Toolkit  
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:5037          0.0.0.0:*               LISTEN      2898/adb            
tcp6       0      0 ::1:631                 :::*                    LISTEN      -                   
tcp6       0      0 :::1716                 :::*                    LISTEN      1895/kdeconnectd    
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:48829           0.0.0.0:*                           -                   
udp        0      0 127.0.0.53:53           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:631             0.0.0.0:*                           -                   
udp6       0      0 :::5353                 :::*                                -                   
udp6       0      0 :::1716                 :::*                                1895/kdeconnectd    
udp6       0      0 :::51156                :::*                                -

I have then tried to open 127.0.0.1:45456 which does not give me a 403 error but just a blank screen. Opening 127.0.0.1:45457 does give me the "Invalid CORS headers" message.

pimterry commented 6 months ago

Can you see anything in the network tab of the browser console @acedogblast? That should tell you how the browser is trying to connect to your local server, and what's going wrong.

In general, you should see a "Mock session started..." line in the server output as soon as the UI connects, so the server definitely isn't receiving a successful request for some reason.

I use the UI from Firefox all the time, and that's definitely supported, so there must be something else going on here.

acedogblast commented 6 months ago

I see a repeating POST and OPTION requests on the network monitor to http://127.0.0.1:45456/start. I have attached a screenshot and the HAR file. 127.0.0.1_Archive [24-03-12 15-45-06].har.gz Screenshot_20240312_154804

EDIT: The console has Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://127.0.0.1:45456/start. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 204.. Seems to be the problem.

Here is a screen shot of the browser console: Screenshot_20240312_160023

pimterry commented 6 months ago

It seems that the server you're connecting to here is blocking the request via CORS. The most likely reason to that is that the server is running as a production build - in which case it only accepts app.httptoolkit.tech as a client for security reasons. See here for the configuration for this.

That shouldn't happen though, and I've just checked and it definitely runs in dev mode by default in the normal UI npm start setup.

The server detects production builds by looking for a HTTPTOOLKIT_SERVER_BINPATH env var, which is set by part of the production app launcher process. Is that set in your terminal for some reason?

One thing that could cause this is if you're launching the UI from an HTTP Toolkit-intercepted terminal. Is that possible? I've just checked and the env var does propagate so that this is indeed set in those terminals too (I'll fix that now). You probably don't want to do that though - I haven't tested but I bet it'll cause all sorts of other problems (intercepted intercepted traffic...?) and it would definitely cause this issue.

Regardless of the reason, if you make sure that variable is unset in your terminal then that should avoid any CORS issues like this. Does that work for you?

acedogblast commented 6 months ago

Running echo $HTTPTOOLKIT_SERVER_BINPATH gives me nothing and I have no memory of ever creating such ENV. Is it possible that the npm packages I am using is causing this problem? I am running node v20.11.1 and npm 10.2.4 inside of a Ubuntu 22.04 LTS laptop.

EDIT: I also am running Firefox 122.0.1 from the Ubuntu Snap store/repo.

pimterry commented 6 months ago

If you open app.httptoolkit.tech in Firefox does that work? Or at least fail in a different way? (e.g. showing a 403 rejection in the network tab, instead of a CORS error)

If so, that would confirm it - for some reason, your local server is running as a production build, and it's using CORS to refuse connections from your localhost UI.

One possibility is that you're actually still running the real HTTP Toolkit server somewhere from the normal app, and it's connecting to that instead of the server you're starting here. I'm not sure why that would happen since it's making a localhost connection and only one server can listen on that port on localhost (unless you're running part of this in some isolated environment, like Docker or a virtual machine or something?) but you can probably confirm that by looking at netstat and the ps tree, and comparing the pids to check that the server listening on the 45456 and 45457 ports is a child of the process you're launching from your terminal, not a server launched by HTTP Toolkit elsewhere.

acedogblast commented 6 months ago

Opening app.httptoolkit.tech does work.

pimterry commented 6 months ago

Ok - your server is definitely running as a production build. Still no idea why though.

On the other points from above:

Is there any Docker/VMs/other involved here?

If you look at netstat, find the pid listening on this port, and then look at the PS tree (e.g. ps -aufx), is that the correct process that you expect?

What happens if you clone the server (from https://github.com/httptoolkit/httptoolkit-server/), npm install and npm start that, and then just run npm run start:web in this repo (to launch the UI without the server)?

acedogblast commented 6 months ago

I do not have any other servers/VMs/Docker container on the laptop. Netstat and PS tree tells me that only httptoolkit from the git clones are operating on the associated ports.

When running the server (fresh git clone) and ui separate I have the same problem of 127.0.0.1:8080 not working but app.httptoolkit.tech working so no difference.

pimterry commented 6 months ago

Ok, being able to reproduce this in a fresh checkout is very helpful though! Given that, we can debug this.

Can you add a console.log to print ALLOWED_ORIGINS and MOCKTTP_ALLOWED_ORIGINS at the end of src/constants.ts? If you restart the server, what do those print?

It would also be interesting to try just removing this block and this block and then see if that fixes this for 127.0.0.1:8080.

It will be informative to test with those disabled briefly, to confirm that this is definitely to the cause, but :warning: in general you should not run the server with those CORS settings totally disabled :warning: as it effectively allows any local process to access the API - that means potentially any website you open in your browser can start a proxy and launch programs using it on your machine (very unlikely to actually happen but very bad). So don't forget to undo those changes and restart the server once you've tested that :smile:

acedogblast commented 6 months ago

This is what I get when appending to constants.ts

console.log(ALLOWED_ORIGINS)
console.log(MOCKTTP_ALLOWED_ORIGINS)
[
  /^https?:\/\/localhost(:\d+)?$/,
  /^http:\/\/local\.httptoolkit\.tech(:\d+)?$/,
  /^https:\/\/app\.httptoolkit\.tech$/
]
[
  /^https?:\/\/localhost(:\d+)?$/,
  /^http:\/\/local\.httptoolkit\.tech(:\d+)?$/,
  /^https:\/\/app\.httptoolkit\.tech$/,
  'chrome-extension://oeehdgfohghfelggpifolochpnkdmpog'
]
pimterry commented 6 months ago

Oh, duh, I see what the problem is!

What happens if you open localhost:8080 in Firefox instead of 127.0.0.1:8080?

They are the same address, but they're not the same browser origin, and only the former is allowed to connect to this API.

Does that work for you?

There's no particular reason for this to be blocked, other than loading the UI by IP not being tested until now. I'll update the server to allow that too in the next release.

acedogblast commented 6 months ago

Using localhost:8080 still results in the same issue even after the 2 blocks you mentioned.

The server does now give me an output of the mock session starting after removing those 2 blocks.

pimterry commented 6 months ago

I think you're going to need to debug into the cors module itself, and see what origin your firefox is sending, and why it's being rejected. Localhost should clearly match the patterns above, and this is generally working everywhere else (I, other contributors, and plenty of people self-hosting all use this setup all the time) so there's something quite unusual going on but I'm not sure what it could be.

isOriginAllowed in node_modules/cors/lib/index.js is where all the action is happening. It's very likely that that's the code that's rejecting your requests for some reason.

One other possible cause is the cors-gate module, which is used to add some additional restrictions on top to tighten this. I don't think that should be the cause here, but you might want to add some logging in there too to check what's up, if cors doesn't quickly lead you to a reasonable explanation. One interesting note I've just seen in their readme:

https://github.com/mixmaxhq/cors-gate?tab=readme-ov-file#firefox-and-the-origin-header.

Firefox does not set the Origin header on same-origin requests for same-origin requests, as of version 53.

That's interesting since it might relate to why it's not working in your case specifically when it does work elsewhere. But that doesn't actually explain this, since the port is part of the origin so this should never be same-origin anyway :shrug:

acedogblast commented 6 months ago

OK I will look into. Have you been able to reproduce the issue on your side?

pimterry commented 6 months ago

Have you been able to reproduce the issue on your side?

No, unfortunately. It works fine on my machine in both Firefox (v123) and Chrome (v121) when using localhost:8080. I've just added a patch in the server here as well to allow 127.0.0.1 and that's now working correctly for me too, also in both browsers.

acedogblast commented 6 months ago

I just tested on a different linux computer which works. The only reason why it is different is that the laptop (which did not work) is using a Snap version of Firefox where are the other computer is using a deb version of Firefox. I have then installed the deb version of Firefox and it now works on the laptop. Therefor it is not an issue with HTTP Toolkit but just another snap limitation. Thank you for your help.

pimterry commented 6 months ago

Hmmm, that's interesting but I'm also using snap Firefox and it works fine, and I don't think it's likely to be the cause (snap certainly shouldn't affect how CORS works!).

Did you look at the cors internals at all? Are there any clues there? Even if you have a working solution now, it would be very helpful to know exactly what's going on to make sure that others don't hit the same issue in future.