BitcoinDesign / Bitcoin-Core-App

Tracking the design process for the Bitcoin Core App.
https://bitcoincore.app
MIT License
55 stars 11 forks source link

Design assumeUTXO user flow #79

Open GBKS opened 10 months ago

GBKS commented 10 months ago

AssumeUTXO is a bootstrapping technique that will allow a user to see some data about their wallet, and possible make transactions, before having gone through the full initial block download. Let's figure out how this works and what the best user experience for this feature is.

There seem to be a few steps:

Some detail to sort through here. Let's figure it out.

--

List of related issues and PRs (update as we go):

yashrajd commented 10 months ago

Based on what I read, it seems the feature is opt-in and disabled by default. I've created a doc to track user flow/spec.

Also a few questions from me too, hope they're relevant:

yashrajd commented 9 months ago

Answers to questions above would be very helpful to get more clarity, but following might be one way assumeutxo could potentially be accommodated within current user flows. Looking for feedback and thoughts on this.

(edit: added following headings) During onboarding:

assumeutxo setup

Post-onboarding:

manage utxo setup

D33r-Gee commented 9 months ago

Hello @GBKS and @yashrajd

It’s great to see that you are both figuring out the best UX to make use of this new powerful feature!

I have been working on an early implementation of AssumeUTXO (AU) into qml for the last few months.

The current wip branch can be found here

Also here are some screenshots of the progress so far (Built on WSL Ubuntu 22.04 on signet)

Please keep in mind that this only place holder stuff to test the functionality and should not be considered anywhere near deployment ready UX ;)

BCA_onboarding_connection_page BCA_onboarding_snapshot_page BCA_node_settings_page BCA_load_utxo_page

And here are some pictures of the mainnet snapshot loaded onto a Samsung Galaxy S8+ with 64GB of storage:

BCA_BlockClock_07dec2023

BCA_Storage_07dec2023

This is the total storage use after full snapshot validation:

BCA_Size_07dec2023

D33r-Gee commented 9 months ago

@GBKS

There seem to be a few steps:

  • User opts-in to this feature, or it is enabled by default

This could be OS dependent i.e. on Android AU is the default and on others it’s opt-in and IBD is the default

  • The application downloads the startup data (just going to call it this for sake of a better term) automatically, or the user uploads it from their hard drive

Currently the user has to have a snapshot already generated or downloaded separatly

  • Initial block download starts for full validation

    • Do we communicate this?

Yes that a great idea

  • The wallet can show some data

    • Wallet balance
    • Can it also show the payment history?
    • Can the user spend?

Haven’t tested with wallet yet will do soon…

  • Initial block download finds a discrepancy to the startup data

    • How to resolve this?

Currently it stops the loading of the snapshot and keeps going with regular IBD

  • Initial block download finishes

    • Do we communicate this? What changes for the user?

Here I’d say that when the snapshot is fully validated perhaps a simple notification would be a nice to have. Currently one has to check the debug log to see if the background validation has completed

  • The user can create and download their own startup data to use elsewhere Yes
D33r-Gee commented 9 months ago

@yashrajd

Thanks for your questions, hope my answers help…

Based on what I read, it seems the feature is opt-in and disabled by default. I've created a doc to track user flow/spec.

Also a few questions from me too, hope they're relevant:

  • Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?

AU can be initiated anytime before full IDB has finished synchronizing. Once launched, it will check and wait till IBD headers have hit the snapshot block height then it activates the snapshot (see ActivateSnapshot()), for more info please see the mark down doc here

  • Is downloading UTXO snapshot from peers possible in 26.0?

Not that I know of in the doc it says: “There is currently no canonical source for snapshots,...”

  • What’s the block height at which the snapshot hash is hardcoded in 26.0?

The block heights that were used to test and validate the AU feature were on:

mainnet 780,000 and 800,000

signet 160,000

testnet 2,500,000

*Would it be updated in every Core version?

Anyone with a full IDB can generate their own snapshot at whatever height they prefer using the following dev tool: ./contrib/devtools/utxo_snapshot.sh More information can be found in the original PR

Would it be able to partially validate snapshots taken at higher block heights?

I haven’t tested this but my guess is that no it would return an error.

  • Do we allow snapshots that fail hash validation to be loaded?

No if the hash validation fails the snapshot loading process stops.

  • Should the user have to wait for the UTXO set to be updated till network tip before they can use wallet functions? (Does code prioritize processing blocks above the snapshot height to make things faster?)

Great question, I currently don’t know haven’t tested with a wallet yet (Definitely a TODO)

  • Should the block clock (IBD status metrics/display) change in any way (if yes, what) once a snapshot is successfully loaded?

With the test I’ve been doing as soon as the snapshot is loading, the block clock displays 56% completion

  • Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that?

Until the back background validation of the snapshot has completed on signet it remains around 1 GB

On mainnet it can get up 30 GB while both chainstates are present then it drops to 21 GB after full validation (These number may differ depending on the OS)

  • How much extra in disk space requirements would be required & needed to be communicated?

Well it doesn’t require more space than the raw IBD however it is more than just the prune state, so something to play with most definitely

  • Does this look different on whether the snapshot is downloaded from peers or loaded from a file?

At this moment there is no peers download option, so no

  • Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

No, it still has to go through the full validation to ensure chainstate fidelity

yashrajd commented 9 months ago

edit: deleting some irrelevant quoted text, plus minor edit in storage question

Thanks for getting the ball rolling on the dev side and even the UI @D33r-Gee great stuff! Your answers were very helpful, my designs (not included here) hopefully reflect that. I still have a few questions:

  • Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?

AU can be initiated anytime before full IDB has finished synchronizing. Once launched, it will check and wait till IBD headers have hit the snapshot block height then it activates the snapshot (see ActivateSnapshot()), for more info please see the mark down doc here

My understanding from the doc is that "activate" basically happens automatically after loading (which is the user-initiated activity) and validation of the snapshot. Is that correct? I ask coz I noticed that one of your screen mockups has an activate toggle in addition to the Load snapshot toggle. Is it required and the user expected to decide to toggle it at some point?

Another question here: what's the time scale of loading & validating processes expected to take? If it's several seconds or longer, we might need a progress/transition screen etc.

  • What’s the block height at which the snapshot hash is hardcoded in 26.0?

The block heights that were used to test and validate the AU feature were on:

mainnet 780,000 and 800,000

signet 160,000

testnet 2,500,000

*Would it be updated in every Core version?

Anyone with a full IDB can generate their own snapshot at whatever height they prefer using the following dev tool: ./contrib/devtools/utxo_snapshot.sh More information can be found in the original PR

Would it be able to partially validate snapshots taken at higher block heights?

I haven’t tested this but my guess is that no it would return an error.

Assuming the snapshot generation takes place at the current block height but the hardcoded hashes are at the block heights you mentioned, wouldn't it always result in a mismatch during validation?

  • Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that?

Until the back background validation of the snapshot has completed on signet it remains around 1 GB

On mainnet it can get up 30 GB while both chainstates are present then it drops to 21 GB after full validation (These number may differ depending on the OS)

  • How much extra in disk space requirements would be required & needed to be communicated?

Well it doesn’t require more space than the raw IBD however it is more than just the prune state, so something to play with most definitely

I take this to mean, yes, if a snapshot needs to be loaded the space requirement would increase but I'm unsure if this is right and by how much? If it's right we should specify and communicate the same on the Storage screen during onboarding.

Also, if it's a lot, and the user tries to load a snapshot AFTER the onboarding UX is done and IBD in progress, do we just disallow loading a screenshot?

  • Does this look different on whether the snapshot is downloaded from peers or loaded from a file?

At this moment there is no peers download option, so no

  • Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

No, it still has to go through the full validation to ensure chainstate fidelity

yashrajd commented 9 months ago

Adding a few mockups based on the scheme, would love to know what others think.

Loading a snapshot (before IBD)

Storage

                                                         |
                                                         |
                                                         |
                                                         v

onboarding-assumeUTXO-load

                                                         |
                                                         |
                                                         |
                                                         v     

assumeUTXO-confirmation

The above flow could sit in the onboarding below as detailed below:

assumeutxo setup

Post-onboarding:

Once onboarding user flow is complete, the user can go to Settings > Wallet > AssumeUTXO to load (and later, generate) the snapshot. The load snapshot user-flow should look pretty much the same from here as well.

The Generate snapshot user-flow might look like below:

assumeUTXO settings

                                                         |
                                                         |
                                                         |
                                                         v

assumeUTXO-generation

                                                         |
                                                         |
                                                         |
                                                         v

assumeUTXO-generation-confirmation

The above flow could sit in the application interface as detailed below:

manage utxo setup

yashrajd commented 9 months ago

There's also a much simpler UI that can essentially be captured on a single screen (shown below), but might not be intuitive enough or allow for good feedback or information. What do folks think?

Send the user to this screen (see below) whether before (i.e- during onboarding screens, detailed in previous comment) or after IBD has started, and require one-step click generate/load buttons to get process started.

assumeUTXO settings-simplistic

D33r-Gee commented 9 months ago

edit: deleting some irrelevant quoted text, plus minor edit in storage question

Thanks for getting the ball rolling on the dev side and even the UI @D33r-Gee great stuff! Your answers were very helpful, my designs (not included here) hopefully reflect that. I still have a few questions:

with pleasure @yashrajd , thanks for the mock ups and questions

  • Can assumeutxo be enabled anytime during IBD (presumably before IBD reaches the snapshot block height) or only before onboarding starts?

AU can be initiated anytime before full IDB has finished synchronizing. Once launched, it will check and wait till IBD headers have hit the snapshot block height then it activates the snapshot (see ActivateSnapshot()), for more info please see the mark down doc here

My understanding from the doc is that "activate" basically happens automatically after loading (which is the user-initiated activity) and validation of the snapshot. Is that correct? I ask coz I noticed that one of your screen mockups has an activate toggle in addition to the Load snapshot toggle. Is it required and the user expected to decide to toggle it at some point?

Your understanding of loading/activating the snapshot is correct, the reason I added an activate toggle in the Load UTXO options is because I have yet to figure out how to load/activate the snapshot during node init.

Another question here: what's the time scale of loading & validating processes expected to take?

That’s device dependent, i.e. on the WSL device it took a few hours (way less than IBD) to complete the background validation. On the Galaxy S8+ it took a couple of weeks to fully validate the snapshot. If it's several seconds or longer, we might need a progress/transition screen etc.

yes that would be great to have since I had to manually look into the debug log, definitely not the most user friendly way.

  • What’s the block height at which the snapshot hash is hardcoded in 26.0?

The block heights that were used to test and validate the AU feature were on: mainnet 780,000 and 800,000 signet 160,000 testnet 2,500,000

*Would it be updated in every Core version?

Anyone with a full IDB can generate their own snapshot at whatever height they prefer using the following dev tool: ./contrib/devtools/utxo_snapshot.sh More information can be found in the original PR Would it be able to partially validate snapshots taken at higher block heights? I haven’t tested this but my guess is that no it would return an error.

Assuming the snapshot generation takes place at the current block height but the hardcoded hashes are at the block heights you mentioned, wouldn't it always result in a mismatch during validation?

If the snapshot generation is at the current block (i.e. 824,361) then it needs to be updated in the chainparams from 800,000 to 824,361 like it was done here

  • Disk space requirement: It seems 2 UTXO sets will be maintained until a certain point in IBD. What point is that?

Until the back background validation of the snapshot has completed on signet it remains around 1 GB On mainnet it can get up 30 GB while both chainstates are present then it drops to 21 GB after full validation (These number may differ depending on the OS)

  • How much extra in disk space requirements would be required & needed to be communicated?

Well it doesn’t require more space than the raw IBD however it is more than just the prune state, so something to play with most definitely

I take this to mean, yes, if a snapshot needs to be loaded the space requirement would increase but I'm unsure if this is right and by how much? If it's right we should specify and communicate the same on the Storage screen during onboarding.

To communicate this there’s definitely some math that can be extracted depending which chain is used (mainnet, signet and/or testnet). Will dig around and see how that could be done… Also, if it's a lot, and the user tries to load a snapshot AFTER the onboarding UX is done and IBD in progress, do we just disallow loading a screenshot?

Well it’s definitely much less than IBD so allowing loading a snapshot after onboarding can still be an option as long as IBD hasn’t finished and blocks haven’t been pruned.

  • Does this look different on whether the snapshot is downloaded from peers or loaded from a file?

At this moment there is no peers download option, so no

  • Can snapshot validation be skipped if the user themselves generated the snapshot (or any other scenarios)?

No, it still has to go through the full validation to ensure chainstate fidelity

D33r-Gee commented 9 months ago

@yashrajd The links to the following images seems to be broken:

The above flow could sit in the onboarding below as detailed below:

assumeutxo setup

Post-onboarding:

The above flow could sit in the application interface as detailed below:

manage utxo setup

D33r-Gee commented 9 months ago

Adding a few mockups based on the scheme, would love to know what others think.

FWIW it like both flows, the detailed one could be useful to start with since AssumeUTXO (AU) is a new feature which I'm assuming most users haven't been expose to so extra steps and communication are helpful.

[Alternative] *

The simple one, I imagine would be catering to more advanced user since they are familiar with the AU functionality

GBKS commented 9 months ago

Awesome work getting this going. I have a few thoughts:

I think this makes for a great discussion on our weekly calls.

D33r-Gee commented 8 months ago

I think this makes for a great discussion on our weekly calls.

Yeah that sounds great... Will join the discussion :)

D33r-Gee commented 8 months ago

@GBKS,

I've been pondering your insightful comment, during the weekly call, about snapshot file availability. You raise a crucial point about trust and accessibility.

  1. Currently, users have two options:

  2. Generating snapshots: This necessitates a fully synced, unpruned blockchain, which might be resource-intensive for some.

Downloading snapshots: This introduces trust concerns, as users need assurance of snapshot integrity.

To address these challenges:

To your point those steps will have to be defined and refined so the process can be effortless and secure for users to leverage snapshots confidently.

D33r-Gee commented 8 months ago

Just updated the working branch with this commit https://github.com/D33r-Gee/gui-qml/commit/959753b4f1410bc7c85aef48d80fa391299ecc8f

If the user chose to load a snapshot during onboarding it now loads automatically after node init

In terms of next steps, as far as UX is concerned, right now the screen freezes for a bit when activating the snapshot (this happens whether loading after onboarding or activating it manually from the options page)

@GBKS and @yashrajd have you encountered behavior like that and if so how was it addressed?

@johnny9 @jarolrod , I'm thinking of modifying the BlockClock code to display the snapshot loading information to the user, would that be the best way to go?

D33r-Gee commented 8 months ago

@yashrajd what's your preferred testing environment (Mac, Linux, Android)?

johnny9 commented 8 months ago

@johnny9 @jarolrod , I'm thinking of modifying the BlockClock code to display the snapshot loading information to the user, would that be the best way to go?

If you have hooks available for progress % that might be an easy way to get going. The current IBD just shows % progress and is generic enough that it might not be too confusing to the user. Might also have to see what the timing is generally like to go from 0 to 100%

GBKS commented 8 months ago

I'm thinking of modifying the BlockClock code to display the snapshot loading information to the user, would that be the best way to go?

Would it be OK to go through the logic and design changes first before touching code?

yashrajd commented 8 months ago

@yashrajd what's your preferred testing environment (Mac, Linux, Android)?

One order of macOS please

yashrajd commented 8 months ago

I'm thinking of modifying the BlockClock code to display the snapshot loading information to the user, would that be the best way to go?

Would it be OK to go through the logic and design changes first before touching code?

Created #91 for discussion around the block clock.

yashrajd commented 8 months ago

@GBKS,

I've been pondering your insightful comment, during the weekly call, about snapshot file availability. You raise a crucial point about trust and accessibility.

  1. Currently, users have two options:
  2. Generating snapshots: This necessitates a fully synced, unpruned blockchain, which might be resource-intensive for some.

Downloading snapshots: This introduces trust concerns, as users need assurance of snapshot integrity.

To address these challenges:

  • PGP/GPG Signing: I'm exploring digitally signing snapshots to enable independent verification by users, ensuring their authenticity and integrity.
  • Hash Verification: As outlined in the original AssumeUTXO PR's "UTXO Snapshots" section, users can verify snapshot hashes against trusted sources for added confidence. This could be done behind the scenes in the app once the snapshot file is picked.
  • Clear User Guidance: Comprehensive documentation and an intuitive in-app interface will be paramount to guide users seamlessly through the snapshot selection and verification process.

To your point those steps will have to be defined and refined so the process can be effortless and secure for users to leverage snapshots confidently.

Absolutely. Actually in the screens I designed I thought I was describing this "verification" process. It seems this hash verification process is different from "validation" that takes places after the snapshot has already been loaded. Perhaps this validation phase can be hidden from the users altogether for simplicity.

I support hosting the snapshot file on say bitcoincore.app and some devs signing it with PGP/GPG. Then after the file is downloaded and actually specified during IBD, we can do a quick hash verification and display result along with the hash (like I did in my design) which can be verified from the bitcoin repo.

GBKS commented 8 months ago

Here's the simplest addition to the block clock that I could think of to indicate that the snapshot is being verified. Maybe hovering/clicking could reveal more details (like we want to do with the peers dots)? Users should understand what this refers to because they went through the snapshots upload process and we told them that the snapshot will be verified.

Note that I did not consider technical details here, I just assume we can abstract them and show a simple progress bar. But let me know if that doesn't work.

image

yashrajd commented 8 months ago

I was able to Load a SIGNET snapshot (obtained from https://github.com/bitcoin/bitcoin/pull/27596 where they distribute the snapshots as torrents, something we could do as well) into the Bitcoin Core App created by @D33r-Gee. Took less than 5 mins to load. Feel free to try this and report your experience & the time it takes for you.

Also discussed the idea of leaving the Load functionality as it is for now, and building a basic Generate snapshot UI so users could start trying & testing the feature without needing RPCs or torrents. What do folks think about that?

Screenshot 2024-01-19 at 14 31 38
yashrajd commented 8 months ago

Updated my design file to accommodate for:

GBKS commented 8 months ago

Helpful comment here.

Currently, AssumeUTXO uses hardcoded block and snapshot hashes so you can only load specific snapshots. It does not work generically with any snapshot. If you want to use different snapshot blocks, then you will need to modify the source code and recompile. There is no snapshot data set for mainnet yet, so you cannot use AssumeUTXO on mainnet without modifying the code. The snapshot info for testnet can be found at https://github.com/bitcoin/bitcoin/blob/3695ecbf680a66b718f97d504308578d001eec49/src/kernel/chainparams.cpp#L269 and for signet at https://github.com/bitcoin/bitcoin/blob/3695ecbf680a66b718f97d504308578d001eec49/src/kernel/chainparams.cpp#L378

D33r-Gee commented 8 months ago

There is no snapshot data set for mainnet yet, so you cannot use AssumeUTXO on mainnet without modifying the code.

Yep Bitcoin Core v26 only has the hardcoded hashes for signet and testnet.

For mainnet functionality (and this would be for testing purposes only) there's a separate fork here by @Sjorns.

I used that fork to make a mainnet branch of the Bitcoin Core App with assumeutxo functionality it can be found here with the understanding that it's for testing purposes only.

GBKS commented 8 months ago

I wonder how long until this works by default on mainnet and with other snapshot hashes. Until then, this is more of a dev feature and we should present it that way.

GBKS commented 7 months ago

I just reviewed the four prototypes that were linked in the notes from the call on Wednesday. The prototype format is great for getting a feel close to the end-user experience.

Overall it feels like we can still remove half the UI elements in the prototype. It feels overly explicit and complicated compared to the other screens in onboarding. Simplifying a bit, we're just loading a file.

I also see a lot of potential to simplify and streamline the copy (especially using one term for this feature and not "Coin state snapshot", "UTXO snapshot", and "assumeUTXO"). Button labels could also be simpler ("Proceed to file verification" -> "Verify"). The "assumeUTXO" settings screen contains the term "UTXO" 5 times.

I would still place this feature in the secondary option screen rather than the happy path. It is brand-new and there are no canonical sources for snapshots, so users either have to dig around online to find one or generate their own via some other node. That may change in the future, but for now it seems like the more appropriate path.

GBKS commented 7 months ago

@D33r-Gee we are trying to wrap up a first set of designs for implementation so you're not blocked. Just have a few practical questions.

GBKS commented 7 months ago

As I just posted in Discord, I tried to gather all the assumeUTXO stuff and create a refined set of mocks. For a guided tour, here's a brief video that goes over the experience and design considerations.

With @yashrajd also working on designs, let's see what we can piece together from our efforts into a final design. Hope this helps move things forward.

For the question about feature naming, here are some options to maybe consider or vote on or so.

image

I think these are the two 2 keys screens where users encounter this feature for the first time and need to form an understanding of what it is.

image

D33r-Gee commented 6 months ago

@GBKS thanks for creating the the guide tour video, it's truly helpful.

RE: naming the feature the first one "Import snapshot" resonates the most although it might be due to working with the code and having it be repeated ;)

I'm curious how other people feel about the choices, not knowing what a snapshot is?

  • What is the scope for your first implementation? Does it also include snapshot creation or only import?

Currently it doesn't include snapshot generation since it requires an archival node. The early use cases, I was envisioning, was running a node on an android device without an external hard drive. However the functionality is there and can be exposed, perhaps this could be a feature that appears only to users that have opted to download the entire blockchain, meaning without the prune option.

  • Are you planning on adding some type of import/create progress bar indicator, for example on the block clock screen?

I like what you show in the video the little progress bar in the blockclock and in the navigation bar look great, and they would be helpful so the user can know where the snapshot verification is at.

  • Generally, it seems like the user will never have the option to both import and create snapshots, right? Seems like it should be one or the other.

That is correct and they can only generate the snapshot if they have an unpruned copy of the entire blockchain.

D33r-Gee commented 6 months ago

I wonder how long until this works by default on mainnet and with other snapshot hashes. Until then, this is more of a dev feature and we should present it that way.

Would love to know that as well, my limited understanding is that there could be attack vectors opening due to improper use of a utxo snapshot, that's why they only merged the testnet and signet options into Bitcoin Core 26. So yes I agree with that for now treating it like an experimental feature is the way to go and communicating that to the user is important.

yashrajd commented 6 months ago

I wonder how long until this works by default on mainnet and with other snapshot hashes. Until then, this is more of a dev feature and we should present it that way.

Would love to know that as well, my limited understanding is that there could be attack vectors opening due to improper use of a utxo snapshot, that's why they only merged the testnet and signet options into Bitcoin Core 26. So yes I agree with that for now treating it like an experimental feature is the way to go and communicating that to the user is important.

If it is merged (even for testnet and signet), I wouldn't consider it any more experimental than Bitcoin Core itself is experimental. The UI allows for more opportunities to make the feature full-proof with the constraints it operates under, with clear communication including error messages. For eg: we can show Load/Generate snapshot features disabled in Settings if chain=mainnet.

Another thing to consider is timelines: by the time of the next release of the Bitcoin Core App, it might not even be disallowed on mainnet, so in keeping with the philosophy of design being way ahead of dev, we should design (and build) the feature in its full glory which is where design is with wallet stuff like multisig...

I've tried to reflect this in my designs...

yashrajd commented 6 months ago

I stand corrected on mainnet readiness 💀

https://github.com/bitcoin/bitcoin/issues/29616