stellar / stellar-core

Reference implementation for the peer-to-peer agent that manages the Stellar network.
https://www.stellar.org
Other
3.12k stars 970 forks source link

Improve "apply-load" to cover more network settings and scenarios #4520

Open MonsieurNicolas opened 2 weeks ago

MonsieurNicolas commented 2 weeks ago

Tagging as a discussion for now.

Right now it's fairly difficult to validate proposed ledger wide limits, yet having a good understanding of their impact can really help with

Here is a high level way we could iterate on bench-marking capability in this context:

dmkozh commented 2 weeks ago

we should document how to build a "reference min-spec node" that can be used to run those benchmarks. I

That's a good idea, but I'm not sure which spec this should be. I'm not sure we should hinder ourselves too much, given that validators should have much more reasonable specs.

should get a new method that performs more interesting work

There are some issues with this contract, but I'm not sure if they're about not 'interesting' enough work. I don't think randomization on the contract side is necessary. I'm also not even sure if we should modify any entries on the contract side at all, it might be better to just pass them through (at least for now; eventually we might start filtering out no-op writes). Basically all the IO happens based on the footprint and it's not very useful to do anything about the ledger entries on the host side. A big issue about this benchmark so far has been variance, I would work on eliminating it as much as possible before trying anything fancy.

ledger state is generated by adding contract data that will correspond to all IDs reached during a run

Yeah, I've been thinking about BL emulation as a next step. Appending directly to the buckets is an interesting idea, I didn't know that's possible. I've no clue as to how much that will change the results though.

basically, ApplyLoad::benchmark() should always apply transactions over the same state (so same ledger number), by resetting its state after applying a ledger

That's a great idea, thanks.

MonsieurNicolas commented 2 weeks ago

That's a good idea, but I'm not sure which spec this should be. I'm not sure we should hinder ourselves too much, given that validators should have much more reasonable specs.

I think the issue is that people's laptops are likely much faster at a lot of things than the typical validator in AWS and in general we need a way for everyone to normalize numbers so that we can have sane conversations.

The actual docker args cannot be hardcoded (at least CPU seems to be relative to the host CPU), but we can and should document the absolute numbers in terms of IOPS and core performance (probably look at some raw number easy to benchmark like X sha256/second so that people can easily tweak their setup to get close enough to that performance) . Note that we may already have some instructions/second "basic validator" number as we need something like that in the context of calibration. IOPS -> at some point we'll shoot for 1M IOPS, but for now conservative number should be more like 100k IOPS.

Basically all the IO happens based on the footprint and it's not very useful to do anything about the ledger entries on the host side. A big issue about this benchmark so far has been variance, I would work on eliminating it as much as possible before trying anything fancy.

yeah I don't think you actually need to modify entries within the contract, but I am pretty sure you need to call the put host function (how entries end up in the "modified" list) -- otherwise we'd allow contracts to rewrite entries they don't own.

dmkozh commented 2 weeks ago

but I am pretty sure you need to call the put host function (how entries end up in the "modified" list)

Nope, currently we actually overwrite all the RW entries unconditionally, the security guarantee comes from the host not allowing contracts to modify entries they don't own. We might start filtering out no-op changes eventually, but I'm not sure if that's a useful optimization for the real traffic.