search

XLNT / gnarly

🤙 A blockchain ETL tool to make your life better. Now that’s fuckin’ gnarly.
https://xlnt.co
Apache License 2.0
119 stars 11 forks source link

blockstream state persist #3

Closed shrugs closed 6 years ago

shrugs commented 6 years ago

Problem

If we are following the chain but then stop processing during a short-lived fork, once we restart the process the blockstream queue will be empty so it will blindly trust that the next block it sees is the main chain because it can't do any resolution since it doesn't know the blocks before that to calculate longest chain.

Solution

When resuming from a block number n, actually import all blocks from n-10 to n and then start triggering any listeners that want info from blocks n+

nionis commented 6 years ago

Could we, take our latest block in the database, follow up until parent is not within the fork and continue from there? This makes it future proof in case there are longer chains than 10 (extremely unlikely right?)

shrugs commented 6 years ago

@nionis can you explain "follow up until parent is not within the fork and continue from there?" more deeply?

nionis commented 6 years ago

@shrugs I am not sure if I am technically correct, but when the system boots again after restart:

  1. query eth node by block hash to check if block was invalidated (is that possible somehow?)
  2. if not invalidated continue normally | if invalided trace back the chain using parentHash until parent block is the root of the split and thus its valid, then we start from there

But I am not sure if there is an easy way to check if block was invalidated cause then blockstream would have implemented it already

edit: as long as there is a guarantee that the short lived fork is below 10 blocks then we are way better of using initial method stated of course

shrugs commented 6 years ago

It may be difficult to check if a specific block has been invalidated, I haven't tried that though.

Hmm, we may not be able to just plug-and-play the blockstream; we'll have to consolidate our own database with whatever it sees, so we'd have to be able to replace its internal blockchain with the one in the database and then run it. Hmm, this requires a bit more thought from me, code-wise, sorry.

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 360.0 DAI (360.0 USD @ $1.0/DAI) attached to it.

shrugs commented 6 years ago

@nionis that's yours to claim 😉

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 11 months, 3 weeks from now. Please review their action plans below:

  1. nionis has been approved to start work.

    I suppose what is left right now it to test the state, since the erc20 issue is unrelated with block-persist branch

Learn more on the Gitcoin Issue Details page.

nionis commented 6 years ago

@shrugs sweet! thanks :)

shrugs commented 6 years ago

closed via #47, thanks @nionis!

gitcoinbot commented 6 years ago

@nionis Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

nionis commented 6 years ago

@shrugs I guess this is done?

shrugs commented 6 years ago

@nionis yup; I'm not the funder, though, so I gotta let vivek get around to approving it. I've dm'd him :)

shrugs commented 6 years ago

@nionis actually, can you submit a PR on gitcoin or something? I haven't gone through the flow as a claimer, so idk what it's like.

vs77bb commented 6 years ago

Hi @nionis can you 'Submit Work' on Gitcoin here? If you need a bit of ETH to make your first transaction, the Gitcoin Faucet can help.

From there, I'll pay out the bounty! Great work here 🙂

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 360.0 DAI (360.0 USD @ $1.0/DAI) has been submitted by:

  1. @nionis

@vs77bb please take a look at the submitted work:

gitcoinbot commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 360.0 DAI (360.0 USD @ $1.0/DAI) attached to this issue has been approved & issued to @nionis.