add support to import archives

[friedger]

Hiro provides archives for the chain states. It would be nice if there are instructions or commands how to import or use an archive.

Currently, the persistent-data folder has root permissions and it is not straight forward to import both the chain state and the postgres archive.

[wileyj]

100% agree. I've already been speaking to them about utilizing this before i just do it 😄

On Thu, Mar 9, 2023 at 14:41 Friedger Müffke @.***> wrote:

Hiro provides archives for the chain states. It would be nice if there are instructions or commands how to import or use an archive.

Currently, the persistent-data folder has root permissions and it is not straight forward to import both the chain state and the postgres archive.

— Reply to this email directly, view it on GitHub https://github.com/stacks-network/stacks-blockchain-docker/issues/100, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVXIHBE5O4UQEDJTILLQ6LW3JMBJANCNFSM6AAAAAAVVWROPM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[rafaelcr]

+1 I was just going to create this issue, would be super useful! It would also be a good time to upgrade to Postgres 15 @wileyj given that the latest archives we have are for pg15 👍

[wileyj]

+1 I was just going to create this issue, would be super useful! It would also be a good time to upgrade to Postgres 15 @wileyj given that the latest archives we have are for pg15 👍

https://github.com/stacks-network/stacks-blockchain-docker/pull/104

i have to document how to use it, but the gist is:

1. clone repo
2. sudo ./scripts/seed-chainstate.sh
3. wait a really long time (didn't time my tests, but around 60 minutes is what i recall the entire process taking)
4. start the services normally

the next stage now that the script is working reasonably well is to add a flag to do this on first startup, i.e. ./manage.sh -n mainnet -a seed or something like that, but that'll take some time to figure out.

[wileyj]

there are 2 places in the script that require user input, the first to delete any existing data, and the second to start the download process. simple "y" keystroke at the prompt

[friedger]

Nice

[friedger]

After running through the flow I now receive a missing parent error. Do I need to do it again? Is something else I can do? @wileyj

stacks-blockchain-api  | {"level":"error","message":"error processing core-node /new_block: Error: DB does not contain a parent block at height 98596 with index_hash 0x37fafb8b53639d59be2873586aeceb0847bff50c623365528fe062267149adcb DB does not contain a parent block at height 98596 with index_hash 0x37fafb8b53639d59be2873586aeceb0847bff50c623365528fe062267149adcb","stack":"Error: DB does not contain a parent block at height 98596 with index_hash 0x37fafb8b53639d59be2873586aeceb0847bff50c623365528fe062267149adcb\n    at PgWriteStore.handleReorg (/app/src/datastore/pg-write-store.ts:2655:15)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at /app/src/datastore/pg-write-store.ts:223:7\n    at scope (/app/node_modules/postgres/cjs/src/index.js:222:18)\n    at Function.begin (/app/node_modules/postgres/cjs/src/index.js:209:14)\n    at PgWriteStore.update (/app/src/datastore/pg-write-store.ts:221:5)\n    at handleBlockMessage (/app/src/event-stream/event-server.ts:320:3)\n    at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2023-03-30T12:32:14.866Z"}

[wileyj]

something is definitely off, the block height in your log message should be higher. if you can run it again but save the output this time that would be helpful. if you don't want to share here you can email it to me.

i think I'll adjust the process a bit as well today

On Thu, Mar 30, 2023 at 05:33 Friedger Müffke @.***> wrote:

After running through the flow I now receive a missing parent error. Do I need to do it again? Is something else I can do? @wileyj https://github.com/wileyj

stacks-blockchain-api | {"level":"error","message":"error processing core-node /new_block: Error: DB does not contain a parent block at height 98596 with index_hash 0x37fafb8b53639d59be2873586aeceb0847bff50c623365528fe062267149adcb DB does not contain a parent block at height 98596 with index_hash 0x37fafb8b53639d59be2873586aeceb0847bff50c623365528fe062267149adcb","stack":"Error: DB does not contain a parent block at height 98596 with index_hash 0x37fafb8b53639d59be2873586aeceb0847bff50c623365528fe062267149adcb\n at PgWriteStore.handleReorg (/app/src/datastore/pg-write-store.ts:2655:15)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at /app/src/datastore/pg-write-store.ts:223:7\n at scope (/app/node_modules/postgres/cjs/src/index.js:222:18)\n at Function.begin (/app/node_modules/postgres/cjs/src/index.js:209:14)\n at PgWriteStore.update (/app/src/datastore/pg-write-store.ts:221:5)\n at handleBlockMessage (/app/src/event-stream/event-server.ts:320:3)\n at run (/app/node_modules/p-queue/dist/index.js:163:29)","timestamp":"2023-03-30T12:32:14.866Z"}

— Reply to this email directly, view it on GitHub https://github.com/stacks-network/stacks-blockchain-docker/issues/100#issuecomment-1490224387, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVXIHE3DWMPSZXUCZSFEVLW6V4S5ANCNFSM6AAAAAAVVWROPM . You are receiving this because you were mentioned.Message ID: @.***>

[pleaseshutup]

@wileyj i see you added updates to seed-chainstate.sh as you said you would. If all good would you mind adding to the README? I didn't even know this option was available until i was lucky to catch a comment on it on the stacks discord.

[wileyj]

simple reason: the script isn't fully ready, but will be documented. it should work now but needs more dry and better error handling.

On Sat, Apr 1, 2023 at 08:57 pleaseshutup @.***> wrote:

@wileyj https://github.com/wileyj i see you added updates to seed-chainstate.sh as you said you would. If all good would you mind adding to the README? I didn't even know this option was available until i was lucky to catch a comment on it on the stacks discord.

— Reply to this email directly, view it on GitHub https://github.com/stacks-network/stacks-blockchain-docker/issues/100#issuecomment-1493016653, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVXIHGC5JUXTXUY5YFQVZ3W7BF73ANCNFSM6AAAAAAVVWROPM . You are receiving this because you were mentioned.Message ID: @.***>

[pleaseshutup]

OK FWIW I recently upgrade a node to 2.1 Waited 3 days to see if I could get it to re-sync but i only got to block ~47000 before I figured F-it i'll try this script.

It ran successfully just now, and my node is at 100498 and catching up. Should be fully caught up soon!

Thanks :-)

[wileyj]

nice, glad to hear it! i will get to the docs soon but appreciate the feedback

On Sat, Apr 1, 2023 at 10:06 pleaseshutup @.***> wrote:

OK FWIW I recently upgrade a node to 2.1 Waited 3 days to see if I could get it to re-sync but i only got to block ~47000 before I figured F-it i'll try this script.

It ran successfully just now, and my node is at 100498 and catching up. Should be fully caught up soon!

Thanks :-)

— Reply to this email directly, view it on GitHub https://github.com/stacks-network/stacks-blockchain-docker/issues/100#issuecomment-1493047964, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVXIHEIXTRK3WCPKTOMAZLW7BOB7ANCNFSM6AAAAAAVVWROPM . You are receiving this because you were mentioned.Message ID: @.***>

[wileyj]

@pleaseshutup @friedger added some documentation here after simplifying the script a bit, but note that now sudo is required (if you run as a normal user, it will error out that you need to run with sudo). still need to lint it and possibly add an option to run it as part of start but it's getting close to being done.

https://github.com/stacks-network/stacks-blockchain-docker/pull/106

[pleaseshutup]

@wileyj looks great to me!

[wileyj]

@wileyj looks great to me!

excellent! i'll keep this issue open since there's still some additional logic i'd like to add.

[wileyj]

@s4ndxyz can you open a new issue for this? this sounds like more an API issue i want to clarify, but don't want to spam here.