Conditional Websocket

[theodorDiaconu]

Problem

There are types of apps where you don't need a persistent connection to the webserver, or you may only needed in some routes or components in your app. This limits to some extent the scalability of a Meteor app because it consumes resources that aren't needed.

This opens the path to make your Meteor instance act only as an HTTP API, allowing same RPC pattern, but making the request HTTP.

This pattern, combined with SSR, can allow us to host our js/css/public inside a CDN. And make Meteor only serve the initial rendering and later handle HTTP requests.

Solution

As solution I proposed https://github.com/cult-of-coders/fusion. Which does the following:

• Doesn't automatically open a connection
• Provides an API to open-close connection
• Provides an API to handle opening of connections in certain areas, and then close it when it's no longer needed.
• Creates a server-side route "/__meteor" that accepts RPC calls
• Allows authentication in this matter as you would normally have with this.userId inside methods, because it sends the Resume Token on every request and it gets verified.
• Gives ability to conditionally open the websocket when needed and morphs Meteor.calls to work via DDP

As much as I preferred to make this live as a stand-alone package. The current implementation of ddp-client doesn't allow us to. So we can either make DDP-Client hackable, by allowing it to delay it's connection via Meteor.settings.public and properly exporting DDP so it can be hacked by other packages, either we put it in the core (my preferred option)

The license of my package (idea) is MIT so you have the freedom to do whatever you want with it.

Additional Extensions

• We can even make subscriptions work in a hackish way with this approach. By requesting them via HTTP, call the publish handler server-side, return the docs, then emulate "added" events to the local collections. However, this is unsafe, because unsubscribing can become very chaotic and hard to do properly especially when multiple subscriptions are involved with matching documents.

Foreseen Problems

I believe this approach may have some hidden implications especially for global publications (clientVersions, loginConfiguration). I'm not yet aware of the full impact, but I strongly believe this is the future.

[theodorDiaconu]

@klaussner @benjamn I need this in order to further iterate on Apollo integration. Can you please confirm that if I start doing this, it's gonna get merged ? Also if you have any tips about implementing/testing this, do let me know. I want to give a helping hand and merge https://github.com/cult-of-coders/fusion to the core.

I was thinking allowing an env variable to define this behavior: CONDITIONAL_WEBSOCKETS=true or a Meteor setting? Any tips are welcome!

[drone1]

Hm, no reply from Meteor on this issue?

I don't see a Meteor app can scale to millions of users without conditional DDP connections (or exorbitant server costs). Is this something Meteor's added support for in 1.8 or what's the latest on this?

Would love to learn more. Thanks a lot.

[theodorDiaconu]

@drone1 they can't do everything and it's all right. We have a fully working solution that helped us scale massively. Adding it to the core is optional and not very important to be frank. It already works and is production and battle tested.

[drone1]

Of course they can’t, but to me it does seem like a fundamental limitation, so I was hoping for some explanation – mostly because I was hoping to be wrong – I was hoping it's been solved 1.8 somewhere and I missed it! Thanks.

On Thu, Apr 25, 2019 at 01:45 Theodor Diaconu notifications@github.com wrote:

@drone1 https://github.com/drone1 they can't do everything and it's all right. We have a fully working solution that helped us scale massively. Adding it to the core is optional and not very important to be frank. It already works and is production and battle tested.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/meteor/meteor-feature-requests/issues/221#issuecomment-486527856, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQIEPGA3AJBVC6L6IA3TALPSFAOVANCNFSM4EEB25KQ .

[raphaelarias]

I believe there are some improvements regarding DDP performance coming right? I saw one or two pull requests about batching DDP.

I found them: https://github.com/meteor/meteor/pull/10478 https://github.com/meteor/meteor/pull/10120

[theodorDiaconu]

@raphaelarias this is less about DDP and more about using Meteor without websockets with all the greatness it offers

[raphaelarias]

@theodorDiaconu Yes, you're right and I understand.

Personally, scalability of Meteor still concerns me, internally we're moving a lot of our hooks to a standalone nodejs server plus some increased usage of serverless functions. Then our Meteor apps are responsible only for the user interactions.

To have the WebSocket constantly connected in our case is somewhat okay because literally every route we load something different, but it's nice to have as many options as possible to customize Meteor to your own needs.

We disconnect from Meteor if the user is not using the app for more than 60s.