Web socket support

[kingdun3284]

Is there any road map on supporting web socket? Or any suggested temporary solution for now?

[rustamwin]

If do you want socket.io based - socket-controllers ;)

[LoicPoullain]

We can add it in the roadmap (August release will focus on adding social auth though).

Websockets use a different protocol from http and so it will take some time to implement. The idea would to use the same Context object for controllers but with a different Context.request property. In this way, we will be able to keep hooks that use ctx.user, ctx.session and ctx.state.

I'll think on how to implement a temporary solution for now. Is your frontend in JS (or TS)?

[kingdun3284]

My frontend is TS

[LoicPoullain]

Ok so we could go with socket.io then. I'll take at look today on what can be done. 👍

[LoicPoullain]

A workaround could be this one:

index.ts

import * as socketIO from 'socket.io';

// ...

async function main() {
  // ...

  const app = createApp(AppController);

  const httpServer = http.createServer(app);
  const io = socketIO(httpServer);

  io.on('connection', socket => {
    socket.on('chat message', msg => {
      console.log('message: ' + msg);
      // Use foal.services.get(MyService) to call a service
    });
  });

  const port = Config.get('port', 3001);
  httpServer.listen(port, () => {
    console.log(`Listening on port ${port}...`);
  });
}

main();

It's pretty ugly, but it's the only thing I can propose for now.

[LoicPoullain]

Depending on the app you're creating, there is also the polling technique that can be another workaround.

[osman-mohamad]

any news about supporting socket.io out of the box ?

[LoicPoullain]

@osman-mohamad I won't have time to implement something before 2020.

Just to know, in which case would you like to use socket.io / websockets. Are there particular features that like / love ?

[Edo78]

@LoicPoullain personally I'm looking into socket.io integration to achieve realtime collaboration otherwise I'll have to move and adapt my db to firebase (or have to use my db AND firebase)

[osman-mohamad]

@LoicPoullain some thing like https://laravel.com/docs/5.7/broadcasting

[damien-white]

I think that supporting native WebSockets would be far better of an idea. Socket.io was created a long time ago in order to support browsers that had not yet had proper support for native WebSockets.

This is no longer the case. There is a much better library simply called "ws" that I use for actual WebSockets. I don't see why socket.io is the default go-to here. I would strongly recommend at least entertaining the idea of native WebSockets instead of the hacky version also known as "socket.io".

WebSockets are natively supported these days within all modern browsers. There is really no longer a need for socket.io. It's also far more simple to work with the "ws" library IMHO than "socket.io". You also don't lock people into using "socket.io-client" on their frontend if you're using actual WebSockets. If you go with "socket.io" you are automatically locking everybody into having to use the "socket.io-client" library on their frontends -- whether they want to use this library or not.

Edit: I wanted to point out that "ws" is potentially also actually far more popular than "socket.io". The following is a graph that shows monthly npm downloads of all the "major" WebSocket-related libraries:

https://www.npmtrends.com/socket.io-vs-sockjs-vs-websocket-vs-ws

These download trends might not be the best metric, but it does have some merit to it. I could be wrong, here, and this metric may mean nothing to you. Still, I figured it was worth including.

[dengue8830]

I started a topic here. Personally I use socketio because

• it have a client for java and swift. So important for your native apps
• support for disconnect detection and auto reconnect. Otherwise you have to write your own boilerplate for web, java and swift using the basic ws libs for each one.
• support for proxies and load balancers

Socket.io is very old yes but it uses ws behind the scenes and have a lot of boilerplate battle tested. So I prefer that instead to re invent the wheel.

Also I think a framework should support both (would be ideal) or at least ws.

We can check how nestjs solved this problem.

[damien-white]

@dengue8830 You make some very good and valid points. Also, a good point RE: how Nest.js solved the problem. I completely forgot about their implementation. That would be like the best of both worlds. Really nice idea, in my opinion.

[LoicPoullain]

Thank you for sharing your experience on the subject.

Here is a how I see things at the moment based on your comments and my own research:

Requirements / Goal

Build apps with real-time collaboration.

Solution

To achieve this, the framework must allow to receive and broadcast notifications using WebSockets.

Constaints

• The solution should be as simple as using Foal's HTTP controllers. The architecture should not be too complicated or overengineered.
• Foal's authentication, authorization and validation hooks should work with the solution.
• The solution should work with proxies and load balancers.
• The solution should be scalable and work with Foal's stores and sessions.

Library comparison

Both librairies are widely used (> 3 000 000 downloads per week).

Pros of ws	Pros of socket.io
Number of dependencies: 0 (ws) vs 40 (socketio)	Auto-reconnection support
node_modules size: 0,1 MB (ws) vs 2,9 MB (socketio)	Disconnection detection
Cumplies with WebSockets protocol. No need to add socketio client library.	Handles proxies & load balancers
Network traffic	Namespaces and rooms
Maybe easier to implement with Foal's stores.

Addditional link

https://stackoverflow.com/questions/10112178/differences-between-socket-io-and-websockets/38558531#38558531

[damien-white]

I think you raise some excellent points. My only real "concern" here, then, would be authenticating Websocket connections before allowing them to upgrade as this is a rather expensive operation to perform.

For libraries, I know that I had initially advocated for the "ws" library, but if Socket.IO is easier to implement and is easier to integrate into the framework, I would not be opposed to choosing Socket.IO.

I am indifferent, really, as to what lib is chosen. As long as Wesocket clients can be authenticated properly (ideally with the auth "baked in", but I understand that is, perhaps, much easier said than done) and the framework supports Websockets, I would be happy.

FoalTS is a fantastic framework. The one and only thing that is currently stopping me from using it is the fact that it does not support Websockets. It's much more simple for me to spin up a custom express or koa server and handle all the details myself.

I would love to be able to use FoalTS and I would even be willing to help out with integrating websockets support if you are planning to implement support for them.

[jonathanroze]

Any news about WebSockets in FoalTS ? :)

[LoicPoullain]

@jonathanroze This feature will probably be implemented this year. But this is a tough subject that involves many parts of the framework so it will take some months to be coded and tested. It won't be available this spring for sure.

[LoicPoullain]

Aim

Be able to build web apps with real-time communication.

Here are three possible use cases:

• A client uploads to the server a file or complex data that needs time to be processed. The client receives regular updates on the status of the processing and is notified when it is complete.
• A specific user needs to be informed of a change on the server. For example, this could be an authorization that has been withdrawn from them.
• A group of users need to be informed of a change on the server. This can be for example to refresh a list of items displayed on the interface when a new item has been added on the server.

Problem with the Current Version

The HTTP protocol does not allow real-time communication. The only solution to simulate it at present is to use HTTP long-polling, but this technique is not very practical and does not meet all needs.

To solve this, Foal will have to support another protocol: WebSocket.

The problem is that Websocket and HTTP are two very different protocols. So we won't be able to reuse our controllers and hooks as they are by simply adding a new parameter or a small feature.

As an example of difference, HTTP supports authentication management through cookies. Websockets don't.

Solution Philosophy

However, even though these protocols are very different, the framework should provide a Websocket architecture similar to what Foal's HTTP architecture already looks like (hooks, controllers, etc).

The Websocket integration should also provide the ability to reuse things that are already provided by the framework (@UserRequired hook, etc).

Expected Features

Basic

This section describes the minimum features required to have a working Websocket solution.

• The Websocket architecture has routes, controllers (and sub-controllers) and hooks.
• When a request is received from the client, the server can respond to this particular request with a payload (like with a mere HTTP request).
• The server can send a message to a client at any time (using different routes).
• The server can broadcast a message to all clients.
• Clients can register for a particular room. The server can broadcast a message to the members of a given room.
• The solution works with proxies and load balancers.
• The solution works with multiple server instances. For example, if a server A tries to broadcast a message to all clients, it will send the message to all clients connected with server A and to all clients connected with server B.
• There is a robust and clear way to handle errors thrown or rejected in the hooks, controllers and services.
• The server and client have support for disconnect detection and auto reconnection.

Advanced

This section presents advanced features that are more difficult to implement but that may be of greater interest.

• We can re-use most of the existing HTTP hooks using an adapter.
• If the user was already logged in before (using cookies or a bearer token), they continue to be logged in when switching from HTTP to Websocket. The solution should not be vulnerable against CSRF attacks during the switch.
• Sessions (or pseudo-sessions) are supported. This means, for example, that a user ID can be attached to the socket object.
• Websocket controllers and hooks should be easy to test. In particular, we would like to be able to list in the correct order all messages that were emitted (with their respective recipient(s) and route).

Outstanding issues

• How do we manage socket destruction? What happens if the session expires during the handshake or after the connection is established?

Choice of Technology (ws vs socket.io)

Websocket integration will use socket.io under the hood. The reasons for this choice are the following:

• The concept of routes can be easily simulated using socket.io events.
• There is a system of aknowlegments to send a response to a specific request.
• Support for rooms.
• Support for proxies and load balancers.
• Support for multiple server instances.
• Support for disconnect detection and auto reconnection.
• Easy assignment of additional data to a socket object.

[LoicPoullain]

Specification

Architecture

Websocket controller

import { SocketIOService, UseWebsocketSessions, ValidatePayload } from '@foal/socket.io';

export class WebSocketController extends SocketIOController {
  // Optional
  options = { path, adapter, etc }; // socket.io options when creating the connection.

  subControllers = [
    wsController('users ', UserController)
  ];

  // Optional
  async onConnection(ctx: WebsocketContext) {
    // Do stuff
  }

  @EventName('create product')
  @UseWebsocketSessions()
  @ValidatePayload(/* ... */)
  createProduct(ctx: WebsocketContext, payload: any) {
    // Do stuff
  }
}

Client code

socket.emit('create product', { name: 'Jane Doe' });

src/index.ts

async function main() {
  const app = await createApp(AppController);
  const serviceManager = new ServiceManager();

  const httpServer = http.createServer(app, { serviceManager });
  const port = Config.get('port', 'number', 3001);
  httpServer.listen(port, () => displayServerURL(port));

  // Instanciate, init and connect websocket controllers.
  await serviceManager.get(WebSocketController).attachHttpServer(server);
}

Websocket context interface

class WebsocketContext<User = any, ContextSession = Session | undefined, ContextState = any> {
  state: ContextState = {} as ContextState;
  user: User;
  session: ContextSession;
  socket: Socket;
  eventName: string;
  payload: any;
}

Websocket hook

// Without a post funciton
@WebsocketHook((ctx: WebsocketContext, services: ServiceManager) => {

})

// With a post function
@WebsocketHook((ctx: WebsocketContext, services: ServiceManager) => {
  return (response: WebsocketResponse | WebsocketErrorResponse | undefined) => {

  }
})

Optional response

@EventName('create product')
createProduct(ctx: WebsocketContext, payload: any) {
  // Do stuff

  return new WebsocketResponse({ foo: 'bar' });
}

Sending messages

export class WebSocketController extends SocketIOController {
  @dependency
  websocket: WebsocketService;

  @EventName('create product')
  createProduct(ctx: WebsocketContext, payload: any) {
    ctx.socket.emit('foo', 'bar');
    ctx.socket.broadcast.emit('foo', 'bar');
  }
}

Joining rooms

export class WebSocketController extends SocketIOController {
  @dependency
  websocket: WebsocketService;

  @EventName('create product')
  createProduct(ctx: WebsocketContext, payload: any) {
    ctx.socket.join('room1');
    ctx.socket.to('room1').emit('foo', 'bar');
  }
}

Using multiple server instances

import { createAdapter } from 'socket.io-redis';

export class WebSocketController extends SocketIOController {
  adapters = createAdapter('redis://localhost:6379');
}

Error handling

A controller method can either return a WebsocketResponse or a WebsocketErrorResponse or it returns nothing.

export class WebSocketController extends SocketIOController {

  // Optional
  handleError() {
    // This method works exactly like `AppController.handleError` but for websockets.
  }

  @EventName('create')
  createUser(ctx, payload) {
    if (1 === 1) {
      return new WebsocketErrorResponse({
        message: 'something went wrong'
      });
    }

    return new WebSocketResponse();
  }
}

The client always receives an object of this shape (if the method sends a response):

interface Response {
  status: 'ok'|'error';
  error?: any;
  data?: any;
}

Examples

// Server
return new WebSocketResponse();
// Client
{
  status: 'ok'
}

// Server
return new WebSocketResponse('foo');
// Client
{
  status: 'ok',
  data: 'foo'
}

// Server
return new WebsocketErrorResponse('foo');
// Client
{
  status: 'error',
  error: 'foo'
}

Re-using HTTP hooks

@EventName('create product')
@HttpToWebsocketHook(UserRequired())
createProduct() {

}

Keeping users logged in

export class WebSocketController extends SocketIOController {

  @EventName('create product')
  @HttpToWebsocketHook(UseSessions())
  @HttpToWebsocketHook(UserRequired())
  createProduct() {
    // ...
  }
}

[pospile]

Any update on this one?

[LoicPoullain]

@pospile still in progress. I just need to find some time to finish this PR.

[LoicPoullain]

Update : the code, the tests and the documentation are now ready.

What remains to be done: write the release notes.

Websockets support will be published with version 2.8.

[LoicPoullain]

Feature added in v2.8 🎉