BUG: Latest PR #145, Blocks folder exist, but 0%

[ChristopherA]

Built latest #147, started Gordian Server:

It told me there was no bitcoin.conf, but there should have been. But I told it to add:

It is stuck at 0% and no blocks.

The data dirs are default:

And the blocks director exists with the old data:

Tail of my log is:

2021-10-05T23:32:51Z Synchronizing blockheaders, height: 543997 (~77.45%)
2021-10-05T23:32:52Z Synchronizing blockheaders, height: 545997 (~77.74%)
2021-10-05T23:32:53Z Synchronizing blockheaders, height: 547997 (~78.03%)
2021-10-05T23:32:53Z Synchronizing blockheaders, height: 549997 (~78.33%)
2021-10-05T23:32:54Z Synchronizing blockheaders, height: 551997 (~78.65%)

But GS window still shows 0%

[ChristopherA]

After 20 minutes or so:

I don't understand why it isn't using the old blocks, they are in the right place.

[shannona]

The 0% is expected behavior. The whole blockchain has to synchronize before it starts verifying, and they're both slow processes. It was under 80% synchronized, so the verification wouldn't have started yet.

(The fact that the blockchain had to resynchronize even though there was a copy of the blocks on disk is the actual issue here.)

[Fonta1n3]

This is actually a bug from the original GS version which included the escaping \ in the dataDir path. We had stopped using the custom dataDir (so the bug went away) and only just added that ability back again (so the bug reappeared), GS is still remembering the old path. You can see that by looking at the two paths in @ChristopherA image, the blocksDir does not include the escaping \. I am adding a check that will automatically convert the dataDir path to the correct one (e.g. no escaping \), to fix this issue you can just reselect the correct dataDir now. I have also added the ability for the kill switch the reset all UserDefaults so any settings like this will be forgotten by the app.

https://github.com/BlockchainCommons/GordianServer-macOS/pull/147/commits/130b1f1426a068404636878bebdc81a6f891461e should prevent this bug from effecting users of the original app.

[Fonta1n3]

And just so it is clear why this resulted in Christopher not seeing his blocks is because the blockDir is specified in the bitcoin.conf which itself is located in the specified dataDir.

One further item of customization would be to add the ability to specify a custom location for the bitcoin.conf so the blocksDir is not dependent on the dataDir.

This also makes me think we ought to use something like ~/.gordian/BitcoinCore/DataDir directory as the default instead of the standard macOS location as it does confuse people. Or perhaps just a prompt on installation where the user is forced to select it so there is no confusion.

[artk42]

And just so it is clear why this resulted in Christopher not seeing his blocks is because the blockDir is specified in the bitcoin.conf which itself is located in the specified dataDir.

One further item of customization would be to add the ability to specify a custom location for the bitcoin.conf so the blocksDir is not dependent on the dataDir.

This also makes me think we ought to use something like ~/.gordian/BitcoinCore/DataDir directory as the default instead of the standard macOS location as it does confuse people. Or perhaps just a prompt on installation where the user is forced to select it so there is no confusion.

+1 for fixing that default BitcoinCore path during GordianServer default installation behaviour - would be a good deal:

1) For newbies trying BitcoinCore it creates unnecessary confusion: "Library" folder is not visible by default from macOS Finder, this creates obscurity and uncertainty, might look scary, datadir falls out of newbie control. If blocks synced but user tired of "being his own bank" and BitcoinCore deleted (no GS KillAll switch used) the sizy datadir with blocks stored somewhere will probably stay there till the end of user's macOS days. The learning curve lengthens and the cost of mistake raises, not critical but not worths of supporting this way of UX.

2) I believe there should be a Developer's concern too: spaces in path is a pain for scripting, linking and other cli operations. This exact issue w/ GS is an example, though vice versa, the bug might blow in the wild when ppl will start adding already synced blocks at BitcoinCore default datadir and there wouldn't exist this proper "spaces" check).

Both issues above are very popular questions/concerns on google and stackoverflow.

I'm afraid ~/.gordian/BitcoinCore/DataDir also will turn working folder into obscure space and out of newbie controls.

May be default structure like bellow might be more straightforward and transparent for entry level users:

Create ~/Node/ for everything in future to keep control on: blocksdir = ~/Node/BitcoinCore/BlocksDir datadir = ~/Node/BitcoinCore/DataDir Keep BitcoinCore and Tor binaries in ~/Node/BitcoinCore/Executables (instead of ~/.gordian) Keep Gordian configs in: ~/Node/GordianServer Create if necessary symlinks in ~/.gordian and ~./torrc Suggest using ~/Node for installing further sovereignty related services and apps, like CLightning/LND/Wallets, etc.

Also would be great to represent it the same way both during installation or killall switch pushed:

All GordianServer services and data to installed/wiped as follows:

• [ ] All installed and created files
  • [ ] Bitcoin Blockchain block data: ~/Node/BitcoinCore/BlocksDir
  • [ ] BitcoinCore data and configs: ~/Node/BitcoinCore/DataDir
  • [ ] BitcoinCore and Tor executables: ~/Node/BitcoinCore/Executables
  • [ ] GordianServer configs: ~/Node/GordianServer
  • [ ] Tor configs:

  • [ ] etc.

    I believe such an approach also clears some burden of those explanation texts hidden under (?) buttons. The helicopter view are much easier to read now and needs less efforts to understand what happens.

[ChristopherA]

@artk47 I agree with you that we need to explore some improvements where Bitcoin data and configuration is stored, however…

• post this 1.0.0 release
• we need to consider a number of issues
  • brew bitcoin currently installs bitcoind to different places on intel & m1
  • bitcoin-core, either downloaded as .dmg or installed with brew, uses ~/Library/Bitcoin by default.
  • having now 4 networks (mainnet, testnet, signet and regtest) also complicates things.
  • we'd like to support the installation of optional plugins like c-lightning, electrum server, fibre (for satellite), cyperpunkpay, spotbit, etc. each who expect bitcoin to be in a consistent place as $path is often incorrect, as well as separate local apps like Wasabi who expect to get RPC values to connect to core.
• AND we need to puzzle through how to lower the barriers of entry and admin for non-power Mac users
  • including ways for people to selectively & non-destructively fix when things go wrong (i.e. delete X first. try again. if that doesn't work, delete X & Y, and try again, where things like tor keys, wallets, and blocks are Z+.

I appreciate your thoughts here, and welcome your participation in making some decisions here (post 1.0.0), but probably not using this issue, and maybe we should have it first in a discussion?

Could you repost your thoughts here, either as concrete new proposals as separate issues for each "feature", or as a broader discussion for further design thoughts?

Thank you!

[shannona]

I've tested for this issue by installing the current rc (from the main repo), not removing the Bitcoin data, and then installing this PR over it. It seems to work correctly now, so I'm going to verify Fonta1n3's fix as of a week ago.

@artk47, as Christopher requests, please repeat your suggestions to discussion or as issues for each new feature. Thank you very much for the feedback!