Consider increasing fees for writable accounts

[aeyakovenko]

Problem

Bots spam the network, often with failed transactions. Well behaving senders are able to avoid failing txs if the senders are signing a transactions that are simulated against recent state. Malicious senders simply flood the chain with pre-formed transactions as fast as possible.

The flood of transactions can occur if there is a known opportunity that is scheduled, like Raydium IDO, or NFT mint. Or opportunistically during high volatility in markets. Programs can't just charge a large flat fee for small trades or all transactions, because attacker can write a custom program to check if there is liquidity and only then execute, but send the TX flood anyways. The flood will take write locks on state, and all other users will be starved. The program can't defend itself against being simulated.

Proposed Solution

This proposal is in addition to the market driven additional_fee signed by users to prioritize access to state. #23211

• get_active_account_write_lock_fee system call: return the current activated write lock fee for this account

• update_account_write_lock_fee system call: A system call that allows a program to set a write lock fee for accounts that it owns. The fee is applied at the start of TX processing to the original write locked accounts such that the write lock account data remains immutable, but the additional lamports are committed even when the TX fails. Reducing the fee can be activated immediately, increasing the fee has to wait for at least 240 blockhashes, or the current system wide blockhash timeout.

It is up to the program to distribute the lamports back to successful callers at the end of the call. Program would need to guard that it's not being called twice within the same transaction, and only refund on the first call. A re-entry safe helper function should be provided to the program so it refund the current fee to the transaction fee payer.

1. total_failed_lamports += max(0, current_lamports - (rent_exempt_lamports + write_lock_fee))
2. refund = current_lamports - total_failed_lamports
   • total_failed_lamports would need to be tracked by the program's state.

Program can implement an eth eip1559 like mechanism by periodically setting the write_lock_fee with the MIN additional_fee paid by callers. MIN would be the min price a caller had to be included in the block to take this write lock.

Implementation needs to have an activation slots longer than the 240 slot timeout for a blockhash. so users know the expected fee they are signing. Transactions using durable nonces may need to specify the maximum fee they are willing to pay including write lock fees.

Why it’s better then only additional_fee

The huge advantage to this, is that it’s possible for a program to build its own congestion control and capture fees and punish misbehaving senders and refund well behaving senders.

Example NFT drop

1. Candy machine is deployed
2. Artist sets the write_lock_fee for the candy machine mint to 0.1 sol
3. candy machine is configured to refund the caller on a successful mint
4. any bots that try to spam the network prior to the mint fail and get charged 0.1 sol

with fewer bots bidding, there is less load on the leaders and UX improves.

Example defi market

1. high volatility causes prices to move and creates tons of arbs
2. Market continuously sets the write_lock_fee to the MIN additional_fee over the last 10 slots
3. Every epoch, market does a buy/burn of its marketplace token with the collected fees

Market created the demand for state, and market is now capturing value that otherwise would go to the L1 only.

tag @taozhu-chicago @sakridge @jackcmay

[nikhayes]

I wrote this in a thread elsewhere and I'm copying over here: Right now you can just access all the most popular writable accounts in one tx and essentially make the network single threaded. That should be very expensive to do. Solana should charge tx fees based on how often an account is read from or written to. There should be a multiplier for accessing a popular account as writable over readonly. The price for each account can be adjusted dynamically with some pricing formula (maybe some EMA of usage per slot? pricer should be exponential and not linear i think). Then compute used in the tx can be multiplied by the sum of the base charges for each account to give a full charge for the tx.

e.g. I send a tx using accounts A, B as read only and C as writable. The pricer says A = 1, B = 4 and C = 2 and writable has a 5x multplier. And my tx costs 10k compute. So total cost = (1 + 4 + 10) * 10k = 150k lamports. Then you can update the pricer for each account.

I don't know how feasible it is to implement my idea but I like it for a few reasons:

1. It is still somewhat deterministic on fee pricing. Nobody likes choosing how much gas they want to pay. The mental complexity there really sucks
2. It comes from the principle that you should be charged the larger your externality to the network. The more popular a resource and the longer you hold it, the more you're charged. The intuition is that someone taking all Serum books, raydium, orca, mango and drift accoutns into one tx and doing complex calculations for 1m compute is doing a lot more damage than someone doing 1m compute on one writable account. This also protects against the problem @mschneider was talking about where 20 liquidators all go for a liquidation and the first guy gets it. The other 19 liqors won't pay much because they'll exit before using much compute.
3. It incentivizes devs to develop more parallelizable and less spammy mechanisms. Right now, devs are handing out economic value to people who spam a lot and are first. This leads to a lot of congestion that affects other network participants but is still profitable to the spammer. I have a theory that what you really need is distribution of writable accounts to be varied enough that entries created by leader are large.

To add to this discussion, I think what @ruuda mentions here (https://github.com/solana-labs/solana/issues/24827#issuecomment-1125247440) about the dynamic of having "stale" transactions charged makes a lot of sense. At the beginning of an NFT drop, deterministic congestion fees will be low and start to ramp up, so presumably most transaction senders will try to reference blockhashes that are as old as possible (it doesn't matter if they just time out soon, they can send new ones), and so it might take a while for fees to ramp up enough to lower demand. In the meantime, you'll have bots still try and flood the network with transactions. If you have fee charges on "stale" transactions though, using this strategy of referencing older blockhashes would mean that your transaction has higher risk of being charged a "stale" fee, and so that would put pressure on transaction senders to find a sweet spot between longer transaction life and not getting charged a higher congestion fee. You can base the stale fee on the same deterministic compute charge that the transaction would have used, but just charge the fee payer (lower compute than transfers and highly parallelizable). There are some other ideas in the thread I liked as to how these fees could be distributed among validators. Ultimately, the higher guarantee of getting charged for transactions sent (either normal or "stale") could help disincentivize spammy behavior quite a bit I think, and would also provide validators another revenue source. They can re-share these stale fees captured from spamming with their own stakers (who might be annoyed their transactions sometimes get charged for going stale).

As a side note, priority fees seem like a lot of guesswork and seem like they'll still be kinda spammy with people trying to guess proper gas?

*Just added this which describes this possible system https://github.com/solana-labs/solana/issues/25211

[ruuda]

A note about priorities: the discussion so far has focused on fine-grained fees that can differ per program/account, so congestion on Candy Machine would not make regular transfers more expensive. This is valuable and useful, but not the most important problem to solve right now.

Right now, the network degrades or even goes down ~daily, because bots have no way to get their transaction prioritized, except to spam. As one of the larger validators, for the past month we have been struggling with null-routing issues on the one hand and overly aggressive DoS protection on the other hand. Technical work like QUIC support is valuable, but only moves the problem. Volume will just increase until it reaches the new limit. Variable transaction fees are the only realistic way to deal with variable demand. I think we need to focus on getting variable fees out as soon as possible.

If that means that regular transfers become more expensive when there is congestion for Candy Machine, that’s a shame, but it is something that can be improved upon later, by making the fees more fine-grained as discussed above. I would argue that being able to do transfers at all, is better than not being able to transact because the network is degraded to the point of being unusable. I would introduce global variable fees first, and iterate on it later.

I wrote some thoughts in #24827 for adaptive fees (set by the network, not a priority fee that users can choose), which I think is a less invasive change compared to adding a priority fee or the more fine-grained fees discussed in this issue.

[nikhayes]

I think they're prety close to having priority fees done but would be nice to hear your input here (https://github.com/solana-labs/solana/pull/25178#issuecomment-1126963687) @ruuda. There was a comment further below that talking about having queues/batches/threads dedicated to transactions of certain compute ranges, which I think you could apply a more localized EIP1559 mechanism to (congestion pricing specific to transactions falling within certain compute ranges).

[thesoftwarejedi]

Simplest way to implement refunds for all successful callers would be to refund current lamports - rent exempt lamports for account.

Is it really this simple? current lamports on the account would include fees collected from failing txs, so this proposed simplest way would actually give the first successful caller after a failure all collected fees from preceding failures.

A re-entry safe helper function should be provided to the program so it refund the current fee to the transaction fee payer.

This becomes crucial, otherwise the program would have to use the instructions sysvar on every ix to properly detect reentry.

[wkshare]

Consider using account balance to prioritize?

[t-nelson]

this has been totally rewritten since the time i made any comments above. they no longer apply and i rescind my support for the current proposal

[nikhayes]

this has been totally rewritten since the time i made any comments above. they no longer apply and i rescind my support for the current proposal

What are your main objections btw?

[mschneider]

As recent changes to tx submission seem to have resolved a lot of the bottlenecks in feeding transactions, this issue now becomes more relevant. I recently looked at the cost_tracker_stats.costlies_account distribution and noticed that all accounts related to Mango's most active perp markets (bids/asks/event queue/market) now regularly hit the 12M cost limit.

A bunch of users started to use the increased compute limit to execute custom programs before placing orders with up to 7-800k CU per TX. We need an efficient way to penalize these actors as they are reducing QOS for all traders due to their lack of incentive to optimize their own code. Their transactions rarely fail, so solutions discussed before that only apply to failed transactions would not be enough.

An ideal solution would allows us to add a larger fee to traders that: a) trade on highly congested markets (identified by a set of 4 accounts in our case) b) use over-proportional amounts of compute (2x compute could cause 10x fees) c) have no strong bias for failing or successful transactions (failing <20%)

[nikhayes]

Adding a base fee per cu cost across all transactions would help too and is pretty simple -- there are probably periods where they don't need to use priority so are only paying for a signature or two on those massive transactions. Adding a base fee per cu would help incentive more efficiency (and pay validators :) ). If bots/rpcs collocate with a leader validator they could also get better latency and get in cheap transactions before more highly prioritized transactions land I'd imagine? If so, they can starve out some compute limits before priority is kicking in (I guess there's transaction forwarding though). I haven't seen anyone post any in depth analysis into how effective priority is.

[tao-stones]

An ideal solution would allows us to add a larger fee to traders that: a) trade on highly congested markets (identified by a set of 4 accounts in our case) b) use over-proportional amounts of compute (2x compute could cause 10x fees) c) have no strong bias for failing or successful transactions (failing <20%)

Furnishing FeeStructure can help, especially an exponential compute_fee_bins, and lamports_per_write_lock. We need help on the actual numbers tho.

[mschneider]

Spent a few hours with @jstarry mapping out the problem & solution space for this kind of fee today. We double checked behavior of both Market Makers as well as Liquidators on Mango v3:

• Only 1 MM uses fee priority, strong indication that traders don't experience QoS issues (yet)
• 6 of the top 10 liquidators use fee priority, see below statistic for impact on 30d liquidation volume

We concluded that building more tools to improve priority fee adoption and usage measurement would help to decide in which situations the current priority fee model does not sufficiently serve users and needs a more granular fee model.

Possible next steps could be: Integrate fee priority into leading wallets or dapp uis Gather more statistics on priority fee usage during large liquidation cascades

[godmodegalactus]

Hello everyone,

I have an idea which could be interesting in future roadmap. It involves changes both in solana and dapps with writable account which have heavy traffic. I went to through the code and I found that in file invoke_context.cs we have a method verify_and_update. This method essentially has access to pre_account state and post_account state. This method essentially verifies the difference and updates the accounts.

I suggest we can create a trait called Mergeable accounts which have some methods or helpers to help us accumulate the difference. Something like

fn merge(current: Account, pre_account: Account, post_account: Account) -> Result<Account> The above method will essentially apply difference in pre_account and post_account to the current account. Once we have that, then for a small context or batch we can run transactions on mutable accounts implementing mergeable in parallel and then merge the difference at the end. Instructions which generate non mergeable states will be failed. The differences of lamports can also be accumulated easily.

This trait then can be used by dapp developers to convert accounts that are locked continuously to mergeable accounts. I guess this solution will enable to be partially multithreaded instead of being single threaded on writable accounts effectively increasing the TPS of solana.

[ruuda]

Before going there, it would be worth sorting transactions into parallelizable batches. Right now they end up in arbitrary batches that lock whatever they need, so there’s a good probability that all batches need to execute serially.

[godmodegalactus]

I guess we can reduce the complexity of sorting these transaction into parallelizable batches by creating a short list of accounts with most write requests for last n slots. This list can be managed by the validator iteself no need to propogate to the network.

[nikhayes]

I would follow the discussion on the transaction scheduler channel of the Solana Tech discord.. there are a lot of changes planned. If you have any ideas you might get more ideas by posting them in the core research channel as well.

[mschneider]

Started to gather a bit more qualitative data:

1. example: arb bot that we might want to punish for spamming, basically write locking super popular spot markets pairwise (serum & orca), simulating profitability on-chain with minimal priority fee: https://explorer.solana.com/tx/Uvqd35FYYER47iGFGL4thWm4iZQRkC7BRr9vTBM1rbLH7HxGZUGV9vn8crGPfCLXdGALMKzD72qdkzpLbF3EHg8

2. a LP that we would punish for spamming, but it's actually because he's measuring tx forwarding delay and failing the transaction to prevent placing orders with stale prices: https://explorer.solana.com/tx/4wTpT38qDPRw1PMuYfVfmrkUK8NVkwj2fpcU7G8LffA4QsFXVdED8kC9TvYE1xT7AdZfnvu31aErtDGGQQBVsNuw

3. another LP that we would punish for spamming, but this time it's using a custom program to drop transactions that are executed out of order: https://explorer.solana.com/tx/21kuas9vTg5dBwhoxMoyEBozUCh6FtPLUs4QuURzt6StMErMQ5XpPBvx1Hz5zE2im3cTni7dxKftRhffEBcevf6P

It would probably save us a ton of compute if we could drop the latter 2 transactions way earlier in the pipeline, even before the validator or mango need to charge fees. 
 (2) should be possible if the user would use older blockhashes to send his transactions, but practically that involves a lot of infrastructure on the client side, so improving protocol UX could make it “easier” and hence improve adoption.

 (3) requires changes to the protocol afaik, but maybe @buffalu / jito can help.

i'm surfacing this, because I would like to get the users who are actually trying to circumvent short-comings of the transaction submission protocol out of the firing line, before we ramp up fees. even just 2xing fees would be really bad for order book LPs.

[mschneider]

I would follow the discussion on the transaction scheduler channel of the Solana Tech discord.. there are a lot of changes planned. If you have any ideas you might get more ideas by posting them in the core research channel as well.

don't have access to write there, so will keep it to github

[godmodegalactus]

Started to gather a bit more qualitative data:

1. example: arb bot that we might want to punish for spamming, basically write locking super popular spot markets pairwise (serum & orca), simulating profitability on-chain with minimal priority fee: https://explorer.solana.com/tx/Uvqd35FYYER47iGFGL4thWm4iZQRkC7BRr9vTBM1rbLH7HxGZUGV9vn8crGPfCLXdGALMKzD72qdkzpLbF3EHg8

2. a LP that we would punish for spamming, but it's actually because he's measuring tx forwarding delay and failing the transaction to prevent placing orders with stale prices:
   https://explorer.solana.com/tx/4wTpT38qDPRw1PMuYfVfmrkUK8NVkwj2fpcU7G8LffA4QsFXVdED8kC9TvYE1xT7AdZfnvu31aErtDGGQQBVsNuw

3. another LP that we would punish for spamming, but this time it's using a custom program to drop transactions that are executed out of order:
   https://explorer.solana.com/tx/21kuas9vTg5dBwhoxMoyEBozUCh6FtPLUs4QuURzt6StMErMQ5XpPBvx1Hz5zE2im3cTni7dxKftRhffEBcevf6P

It would probably save us a ton of compute if we could drop the latter 2 transactions way earlier in the pipeline, even before the validator or mango need to charge fees. 
 (2) should be possible if the user would use older blockhashes to send his transactions, but practically that involves a lot of infrastructure on the client side, so improving protocol UX could make it “easier” and hence improve adoption.

 (3) requires changes to the protocol afaik, but maybe @buffalu / jito can help.

i'm surfacing this, because I would like to get the users who are actually trying to circumvent short-comings of the transaction submission protocol out of the firing line, before we ramp up fees. even just 2xing fees would be really bad for order book LPs.

I had also noticed the point 3. There are lot of mm that use the sequencing program to order their orders. I think this issue can be solved if can lock the mutable accounts lazily like instead of locking them for whole transaction we can just lock it when an instruction needs the account. This way all the transactions which are out of sequence will be dropped before the mutable accounts are locked.

[aeyakovenko]

Won't order book LPs get a rebate? Basically the steady state here should be close to a maker rebate and a taker fee.

[mschneider]

most of the blockspace goes to cancel replace, fills are rarely relevant. all lps will both send maker & taker orders when refreshing prices

[hydrogenbond007]

Having the flexibility to have an application/writeable fee is one of the main features of app-specific rollup approach, would be really to have it on solana. As developers are not able to capture much incentives from users in other forms application fee can incentivize them to onboard more individual users rather than just activity.