API Rate limiting

[abitmore]

We need to design a rate limiting mechanism for APIs. It's reported by some API service providers that their API nodes are likely being abused.

[abitmore]

If can do this with Nginx or another load-balancing or reverse-proxy software, we don't need to do it in core.

[clockworkgr]

nginx/rev proxy only handles the intital connection attempt no? actual api calls traffic will not be visible to those...

Hence, correct me if i'm wrong but any API rate limiting would have to be done on the core node itself.

This is why I mentioned the other day a standalone "proxy" API layer might be a good idea to provide all those features without bolting it all on core.

[abitmore]

According to https://www.nginx.com/blog/tuning-nginx/:

You can set various limits that help prevent clients from consuming too many resources, which can adversely the performance of your system as well as user experience and security. The following are some of the relevant directives:

• limit_conn and limit_conn_zone – Limit the number of client connections NGINX accepts, for example from a single IP address. Setting them can help prevent individual clients from opening too many connections and consuming more than their share of resources.
• limit_rate – Limits the rate at which responses are transmitted to a client, per connection (so clients that open multiple connections can consume this amount of bandwidth for each connection). Setting a limit can prevent the system from being overloaded by certain clients, ensuring more even quality of service for all clients.

More info: https://docs.nginx.com/nginx/admin-guide/security-controls/controlling-access-proxied-http/

I'm not sure if these would help.

Testing is appreciated.

[clockworkgr]

limit_conn and limit_conn_zone -> WS is one connection...I understand we are concerned with the rate of API calls made through it no? limit_rate -> refers to proxied HTTP requests...again, we are using an open WS stream...no knowledge of the traffic through it as far as nginx is concerned

[abitmore]

@clockworkgr perhaps we can use something like this:

location /ws {
    limit_conn addr 1;
    limit_rate 50k;
    limit_rate_after 500k;
}

If you have time you can do some testing. Thanks.

Yes, normal use case may be one connection, but we're talking about rate limiting, which is to avoid abuse. Normal use case may be 10 requests per second per connection, which may use up to e.g. 50k bandwidth, but an abnormal user may be doing 1000 requests per second, etc.

[clockworkgr]

I was talking to DL from apasia and he mentioned how on one of his nodes that was having issues, most client IPs had 1 or 2 ws connections open to the node except for a few (possibly bots) having between 50 -200+

With that in mind,

limit_conn addr 

is probably a good thing to set (or the equivalent in whatever proxy you use).

[dls-cipher]

I'm confirming what @clockworkgr said above. We had issue with England node that in April was hitting 3TB Bandwidth usage before end of the month. Before nginx was edited for conn per ip limit, this month started wild even after complete re-deploy.

By doing netstat for 8090 on that time, I've noticed 50+ connections per IP which would be impossible from a standard/client user, and that node was not even listed in wallet. After limit was applied node calmed down with connection/bandwidth abuse.

Simple example between Japan (listed and not being abused) and England (not listed and being abused before nginx conf)

I'm currently doing all 28 nodes from scratch 50/50 (Debian/Ubuntu) with 2 completely different setups to test against each other. I'll have detailed report and stats within few days( also howto nginx part - re limits per zone/ip including mix ssl deployment from @clockworkgr & @xeroc ).

[clockworkgr]

Spent some time trying to find documentation on shaping websocket traffic but no luck.

As soon as a connection is upgraded to websocket, it might as well be a black box to nginx/haproxy/etc.

The only way to do more traffic shaping except for connections per IP is on the API level in core or actually crafting a custom websocket request proxy that implements the traffic-shaping strategies we need.

[abitmore]

@clockworkgr what does it mean?

[clockworkgr]

That we can only limit connections per IP...no way to limit rate of API calls, payload size and other potential attack or overuse vectors unless:

a) Craft our own intermediate proxying layer (instead of haproxy/nginx) that implements these

or

b) Implement such traffic shaping strategies inside core

[abitmore]

Perhaps can take a look at this https://github.com/EOSIO/eos/issues/3497

[clockworkgr]

Meh. That's exactly the kind of thing we can enforce on the proxy level (connections per IP).

Ideally we want to be able to rate limit API requests too. Such as sending 1000x get_object() in 1s through the single websocket connection.

[tbone-bts]

Instead of rate limiting API requests, what about giving API node operators a share of the fees associated with the transactions flowing through their nodes? In addition to making it economically viable to run an API node, this would also incentivize more market makers and other high frequency traders to run their own private API nodes.

Another beneficial outcome would be to make it economically viable for non-gateway businesses to develop custom UIs (since they could earn by operating their own nodes and have their users' traffic flow through them).

[ryanRfox]

Is it possible for the API node operator to provide an authenticated pay-for-service connection to the end user? Here the API node operator accepts value from the end user to pre-fund an account which allows authenticated API access at a specified rate.

[clockworkgr]

@ryanRfox that kind of the monetization is one (of many) reasonings behind my push for a standalone API proxy server (ideally REST based)

[ryanRfox]

To the point of the OP (rate limit API connections): What can this project implement to directly mitigate those impacts, or must the solution be done implemented outside the scope of our protocol?

[xeroc]

We use haproxy as loadbalancer and that allows to also limit by bytes_in/out per connection. So a rate limitation would be possible. Still, a request count is not possible with a websocket connection (to my knowledge).

[abitmore]

@xeroc or @clockworkgr would you like to create a document about using HAProxy as load balancer? Thanks a lot.

[xeroc]

@abitmore done: https://github.com/bitshares/bitshares-core/wiki/HaProxy-and-BitShares

[abitmore]

@xeroc thank you very much. Do we need to link it in the main wiki page? Also I guess @cedar-book will have interest in this.

[cedar-book]

@abitmore, Yes. Thank you for the information!!