Dedup fork requests - Githubissues

kariy commented 5 months ago

When running katana in forking mode, there's a potential inefficiency that occurs when the forked backend processes identical requests simultaneously. This leads to unnecessary duplication in data fetching, as the same data is retrieved multiple times. To optimize this, we should implement a mechanism to deduplicate incoming requests, allowing the reuse of data from the first identical request.

Example scenario for context:

Consider a situation where a specific storage slot is not present in the local storage. katana then attempts to fetch it from the forked network. If, while this request is still processing, another request for the same storage slot is made, katana will again try to fetch from the forked provider. As a result, two identical requests are sent to the forked network, which is inefficient.

Proposed solution:

Implement a request deduplication process. When a request is made, katana should first check if an identical request is already in progress. If so, it should wait for the initial request to complete and then use that data, rather than initiating a new request to the forked network.

We can use the requests params (eg. contract address, storage key) as the request id, and perform the dedup process on an incoming request that has identical id.