Realtime Benchmark

[kiwicopple]

Chore

As part of our move from Alpha to Beta, we will want to be clear about the limitations and performance of each of the components in Supabase. We need to do 3 things:

• [ ] Design a benchmark (if it isn't already designed)
• [ ] Perform the benchmark
• [ ] Create a blog post or documentation detailing the results

[soedirgo]

Weird, looks like there's no performance numbers for Debezium as late as 2018. I was thinking about trying their approach for benchmarking CDC. The gitter chat link from the bottom has some interesting ideas though.

[kiwicopple]

I think @inian actually started some work on this one?

[inian]

I have some basic code with k6 and websockets but have not managed to get it working fully - test never seems to finish (might be related to https://github.com/supabase/supabase-js/issues/44). I can push it a separate branch if needed so that you can work on top of that if it helps.

[kiwicopple]

Note that the changes aren't in the main release of supabase-js yet, so you could consider using realtime-js directly for this one: https://github.com/supabase/realtime-js

Let me know if you spot any problems - it's our new interface so I want to make sure it's working well

[inian]

Pushed my old attempt here. It is very much a work in progress.

I have documented my challenges in getting k6 to work with our realtime server here

I have modified the webpack config to get the script to play nice with k6's custom runtime. These hacks are very inelegant but at least doesn't throw any error when running the test.

Even though the test doesn't error out, it doesn't seem to terminate (or start?)

[soedirgo]

Preliminary results

Setup

• Postgres: AWS supabase-postgres-0.13.0 image (t2.micro)
• Realtime: AWS supabase-realtime-0.7.5 image (t2.micro)
• Client: My laptop

One small annoyance: need to uncap the no. of open file descriptors for the Realtime server and possibly the client. (More info here). There's also a section for this in Phoenix's 2m WebSockets article, might want to follow it as well.

• /etc/sysctl.conf

  ...
  fs.nr_open=2147483584
  fs.file-max=2147483584

• /etc/systemd/system/realtime.service

  [Service]
  ...
  LimitNOFILE=214783584

• Restart
• Run ulimit -n 214783584

Postgres schema

ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 5;
CREATE PUBLICATION supabase_realtime FOR ALL TABLES;

CREATE TABLE foo (id SERIAL);
ALTER TABLE foo REPLICA IDENTITY FULL;

Load

• 1 INSERT per second
• 1000 subscriptions (code is slightly modified from supabase/realtime/examples/node-js)

Result

CPU utilization at 7%. If we ignore Amdahl's Law, a ~10x machine would get us to 100k :)

[inian]

Great work on setting this up Bobbie. Have you pushed the benchmarking code somewhere?

Can you test on the following instances as well?

• t3a.micro
• t4g.micro (you will need to use supabase-postgres-arm-0.13.0 image for this in the tokyo region)

For next steps, it would be interesting to know what load would push the CPU to 100% (or is memory the bottleneck?) on these instances..

[soedirgo]

👌

The only relevant code I guess is the node-js realtime example code, I'll try working on a reproducible bench suite with the modified code and AWS CLI/SDK (or if you have something like this, I'll use that).

Other next steps:

• Use a dedicated instance for the client (maybe use Tsung? Easy to automate increasing WS connections)
• Use a more realistic Postgres schema
• Bench throughput (max. events per second)

[inian]

Couple of other points which may help

• Make sure that the client instance is in the same subnet as the instance you are benchmarking. That made a huge difference in another benchmark I did
• For a more realistic schema, consider using Chinook. Our postgrest benchmark uses it.