Websocket Plugin

[cymen]

Did this get tossed in favor of hooking up NATS and leaving the rest up to the user? Just curious as websockets are mentioned in a couple of places but I don't see a plugin.

[jmealo]

@cymen: I didn't think a websocket plugin was a good idea as it'd need to be pretty opinionated to implement per-user authentication and filtering (or pub sub) properly. I could provide one that's a "dumb pipe" for example purposes.

If you're looking to use socket.io pairing it with NATS with some custom routing (mapping row-to-user) should be around 200 LoC (I've been running something like this in production for years).

If you're working on a greenfield project and are using NGINX, I'd recommend checking out nchan: https://nchan.io/

I would be interested in adding an nchan example possibly if that's the route you decide on going. If you let me know a little more about your use case I can point you in the right direction.

[cymen]

@jmealo That makes sense. I was just curious as it's still in the README graphic (I get it's annoying to update a graphic file).

I did find nchan and I'm going to go with that. My use case is a SaaS application and I want to keep the client state in sync with the database. The other problem I have is multi-tenancy. I'm considering using logical decoding messages to pass the tenant (so using JSONCDC or something like it directly). I'm curious if you have any thoughts in that area.

[cymen]

I did an audit of my database schema and realized I only have a couple of tables that are not including the tenant id. I think it would be much simpler to just add the tenant id to those tables.

I think the only remaining thing I'm curious about is if you handle ensuring the clients know if they missed a message or not. Not so much ensuring in order delivery but more ensuring the clients know if they are receiving out of order (and they can decide what to do at that point).

[cymen]

I didn't realize nchan did in-order delivery too -- so I think all my issues are addressed after all.

[jmealo]

@cymen: yep, it's a great find. I'm thinking about separating the PostgreSQL specific bits into a module separate from Lapidus. I have some LuaJIT* code based on RapidJSON that's blazing fast and uses 12MB of ram at peak. I may have time this weekend to hack on that and get it pumping into nchan. How important is it that the solution uses Node.js?

*My ideal solution would run inside of NGINX/OpenResty alongside nchan. I stopped prototyping while I evaluated the best way to run workers. I was evaluating Tarantool as well.

[cymen]

To me, it's not too important if it's Node.js or not. Right now, my use case is small project that is growing so it's nice being able to run it on a relatively basic VPS. So as long as the resource requirements are not excessive and it run ons Linux. I'd be curious what you came up with.

[jmealo]

@cymen: I don't use Docker in production but the number of moving parts here is not for the faint of heart to setup so in order to collaborate I'm going to provide a docker image.

If I published it correctly, it should be published as jmealo/topsy.

I tried to keep the image small, it's 376MB (the base Ubuntu image is 123MB, the Node.js image in the official documentation is: 676MB).

I didn't compose anything with multiple images because I want these all to run in one container for development purposes. Depending on how this goes I'll probably make a helm plan for Kubernetes.

The docker file is bootstrapped using a bash script which I will adapt to run on Ubuntu (should be about 15 minutes of work).

Included software:

• OpenResty - nginx fork with lua goodness
• Lua Rocks - this is deprecated in favor of OPM (see below)
• OPM - this supercedes luarocks... but not all packages are available on it
• nchan - as discussed
• gnatsd - used to plugin to lapidus (for now)
• confd - to pull config from consul or etcd
• redis 4 - also scriptable with lua, required for persistent nchan and ledge caching

Included lua libraries

(check out the bold entries, they're pretty great but unrelated to what we've discussed, this is really powerful to have at the edge)

• lua-resty-http
• lua-resty-openidc
• lua-resty-string
• lua-resty-session
• lua-resty-template
• lua-resty-validation
• lua-resty-qless
• lua-resty-qless-web
• ledge
• lua-struct
• rapidjson

You may notice PostgreSQL is missing. Once you check out ledge, you'll see this should sit at the edge. I don't want the edge to have direct access to PostgreSQL.

Begrudgingly, I may package up lapidus as a docker file with all supported databases configured for testing purposes.

Let me know what you think so far.

If you combine this with postgrest and/or postgraphql and nchan hooked up to Lapidus you can do SSR or SPA with real-time and API out of the box (I'll be using vuetify/vue with nuxt). Just get your data in PostgreSQL and rock out. The working name of this project is Topsy.

[cymen]

Ah! Nice, I'll check it out. In my reply above, I was going to mention I was mulling over the redis 4 streams feature. I guess we thought alike -- it's really exciting the potential opportunities it opens up (particularly for smaller scale/built-in functionality).

I'll take a look. Thank you. It might take me a couple of days (I'm heads down on something else for a bit).

[jmealo]

@cymen: In that case I'll try to stick to my some examples and use cases. Let's discuss Redis together when you have time. I'm going to do some experiments with Go/NATS tonight and compare that to Luajit.