DipDup pre-release playground

[droserasprout]

Hi! 👋

This branch is an early preview of a new DipDup version introducing advanced config section with tunables like reindex action and early realtime, rewritten websocket library and lots of performance optimizations.

You can track development progress:

• pysignalr: https://github.com/baking-bad/pysignalr/pull/1
• dipdup: https://github.com/dipdup-net/dipdup-py/pull/170
• indexer: here

These numbers are for indexing from scratch, without first/last level sharding (actually idk how do you achieve that 👀). My main problem now is REST sync stage. There's also a couple of blocking issues:

• Processing of realtime messages is very slow
• Origination received during sync causes resync 👀

upd: removed outdated benchmarks

[sbihel]

Hiya, thank you for the PR! Is it OK with you if I edit your PR to rebase on top of #166 ?

[droserasprout]

Sure!

[droserasprout]

Hi, @sbihel! I've brought some code for you to play with.

1. DipDup updated to 4.0.2.
2. Kepler resolver branch merged.
3. Resolver moved to a separate process. Run it with dipdup -c ... resolver.
4. Some mypy and CI.

Give it a try. I've also enabled a WIP Prometheus integration in this branch. Visit 0.0.0.0:8000 to get performance metrics and indexing progress.

[sbihel]

As discussed on Discord, I'm having trouble running the resolver in Docker.

Also a couple of comments:

• there could be concurrency issues between the indexer and the resolver, I think it's very unlikely but it could happen; and
• most of the Prometheus metrics should be counters or histograms instead of gauges.