Use one federated socket for communication between clients and Companion

[aduh95]

Initial checklist

• [X] I understand this is a feature request and questions should be posted in the Community Forum
• [X] I searched issues and couldn’t find anything (or linked relevant results below)

Problem

Currently on uploads from a remote source (e.g. Google Drive), Uppy is first sending a POST request to Companion to get an idea, and then use this ID to open a WebSocket. The issues with this approach are:

• Browsers limit the number of WebSockets a tab can have opened, if the limit is set to something over the browser limit (usually 200), Uppy crashes without any real possiblity of catching that.
• Because there are two distinct requests, both get queued on the RateLimitedQueue, artificially slowing down the upload.
• Having two requests split in two the queuing logic, and forces us to set priority to avoid timeout on sockets.

Solution

Instead, it could subscribe to an event stream (https://web.dev/eventsource-basics/) using only one request. That should simplify the uploader code, and also speeds up the actual upload.

Alternatives

N/A

[Murderlon]

Unlikely to impact us but interesting trivia:

Warning When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100) — MDN

[aduh95]

It turns out we are using the WebSocket to send pause and resume, so Server-Sent API doesn't seem to fit our use case. We talked about it during the Uppy Call today, and we thought we could instead have one WebSocket shared for all remote files, and use it for all communication between Companion and Uppy (instead of currently making first a HTTP call to get the token, and then use the token to create a new socket per file).

[Murderlon]

What about still using the Server-Sent API but make pause/resume an HTTP request?

[aduh95]

What about still using the Server-Sent API but make pause/resume an HTTP request?

That's definitely possible, but why not use a WebSocket since it seems to fit our use-case better?

[Murderlon]

I'm just wondering, as before the Server-Sent API seemed to suit is better to drop WebSockets, but now that we have to send pause/cancel those benefits don't hold true anymore? Just saying if it has better benefits, and HTTP requests is an option, then it may be worth the thought.

[aduh95]

Having to send separate requests for pause/resume/cancel has its own drawbacks (mainly more latency for each action, more things for the RateLimitedQueue, more stuff that could break if the network conditions are not ideal, etc.), I think we should not dismiss them. The reasons for preferring Server-Sent events apply for a quite limited use-case – tbf we don't even know if corporate proxies blocking WebSockets are still a thing.

[mifi]

according to https://en.wikipedia.org/wiki/WebSocket

The communications are usually done over TCP port number 443 (or 80 in the case of unsecured connections), which is beneficial for environments that block non-web Internet connections using a firewall

I think the only thing a a firewall can do is to cut off long-running requests. maybe that can cause problems for websockets. but it should be the same for SSE?

Or an advanced firewall using deep neural networks can analyse the packets and look at the pattern and timing of how packets arrive without looking at the actual (encrypted TLS) data. Then it can drop connections that look like they are not part of a normal HTTP request-response cycle. I've read that certain states like China's national GFW implement this kind of AI in order to block any traffic that it cannot flag as "safe". Apparently hat's how China detects and blocks, throttles or slows down VPN connections even without knowing the encrypted content of the connection