Program serialization is inefficient due to copies of accounts into and out of the program

[jackcmay]

Problem

BPF program parameters (accounts, instruction data, etc...) are copy-serialized into an input buffer on program entry and back out on program exit. The overhead of copying has become significant as use cases require larger and larger accounts.

For context: https://github.com/solana-labs/solana/issues/14523

Proposed Solution

• Build a custom allocator for the Account struct (probably somewhere around here) which places all accounts loaded in a TransactionBatch in a consecutive memory range.
• Let each transaction create a BPF VM MemoryRegion from a slice of this accounts array into the BPF VM, so that only the accounts belonging to that transaction are inside the slice.
• Replace the account data copy steps by using the MemoryRegion, but keep the serialization interface.

[jackcmay]

@Lichtso The current infrastructure allows a program to create a new account with a data size up to 10k via the SystemProgram::CreateAccount instruction. This is essentially seen as a realloc of the new account's data size. When loading all the accounts into a consecutive memory range, what did you have in mind for account allocation? We will want to put in place a mechanism that allows for both the currently allowed alloc up to 10k but also for future general re-allocations which will include both increasing and decreasing the account size up to the max account size (1GB iirc).

[Lichtso]

@jackcmay Good question. Do we know the number and size of SystemProgram::CreateAccount before the transaction is processed? If we at least know the maximum number of accounts created, then we can leave that amount of padding in the consecutive memory range for creations and reallocations. For future bigger reallocations it might actually make sense to reserve continuous memory address space using mmap.

[jackcmay]

Another good question, the current transaction format does not specify inner-instructions so we can't know all of the SystemProgram instructions that might be called by a particular transaction.

[Lichtso]

I think we are close to finally fixing this issue with: https://github.com/solana-labs/solana/pull/28053