search

BlockchainCommons / Community

Discussions & shared documents for stakeholders in Blockchain Commons
Other
68 stars 10 forks source link

2020 Summer Internship Project: Bitcoin Standup Linux/VPS/Windows Script Improvements #8

Closed ChristopherA closed 3 years ago

ChristopherA commented 4 years ago

Currently the linux scripts for Bitcoin Standup Scripts are fairly simple, and establish just a single node.

This script was originally based on the script in Learning Bitcoin from the Command Line as I found it very inexpensive and easy for a novice to install a Bitcoin testnet full-node with tx-index on Linode by simply pasting a single stackscript and waiting a half-day. This node would be free for the first month (and Blockchain Commons gets a $10 credit if they use the referral code), and only $5 - $10 a month thereafter. When I had taught it before with people laptops we had all kinds of problems, and this solved it.

However, for purposes of a Bitcoin mainnet full node, especially one that is not pruned, hosting a VPS using this script is expensive. On Linode a single VPS that can hold a full node is $160 a month, but that includes 32GB RAM and SSD which is overkill. So we should research the cheapest way to host the bitcoin data on various VPS service separate from the VPS itself.

This will also help us in the future as hosting a Lightning Network requires a full node. See Issue #7.

There are other improvements that can be made to the Bitcoin Standup scripts, including:

We also should release scripts to work standalone for Mac (for those who don't want to use the Bitcoin Standup Mac app) and Windows (a big challenge).

Eventually, after thorough testing, these scripts should audited formally by people in the bitcoin-core community.

What other features am I missing?

-- Christopher Allen

chiselinc commented 4 years ago

As per our discussion @ChristopherA , I'm "splitting out" the task of "release scripts to work standalone for Windows" into its own designated issue, and claiming "point person" for that deliverable.

I feel confident enough poking around existing scripts, and whether I like it or not, some of my grad work has required me to commit in-depth to Windows literacy. I'll still always fall back to Linux when I have a choice, but the wrangling experience will likely prove valuable!

gg2001 commented 4 years ago

However, for purposes of a Bitcoin mainnet full node, especially one that is not pruned, hosting a VPS using this script is expensive. On Linode a single VPS that can hold a full node is $160 a month, but that includes 32GB RAM and SSD which is overkill. So we should research the cheapest way to host the bitcoin data on various VPS service separate from the VPS itself.

Linode has block storage which can be attached to a linode. 320GiB costs $32/month and 640GiB costs $64/month ($0.10/GiB per month). Using 640GiB attached to a 4GB RAM VM would cost $84 a month. DigitalOcean also has block storage at a similar price.

OVH seems to have cheaper options: VPS/block storage, but I haven't looked into the specifics of the pricing. Also AWS has HDD storage, which may turn out to be cheaper than the previous options, but the unpredictable pricing would be a problem.

jodobear commented 4 years ago

Proposal

Tasks

  1. Improve Standup Script with best practices inspired by BTCPay Server.
  2. Add pruned node setup options.
  3. Add options to get deployment environment to make it deployable on VPS/Desktop/RPi with one script.
  4. Add LND script with RTL(for node management) and then add option for c-lightning (It would be better to add them both together but, haven't looked at c-lighting. Need to assess.).
  5. Break up the script for better management of code.
  6. Add option for luke-jr's bitcoin knots and kallewoof's signet
  7. Add option to use pre-signed blockchain snapshot for faster sync.

Enhancements and Other Services

Questions

Hosting Data

However, for purposes of a Bitcoin mainnet full node, especially one that is not pruned, hosting a VPS using this script is expensive. On Linode a single VPS that can hold a full node is $160 a month, but that includes 32GB RAM and SSD which is overkill. So we should research the cheapest way to host the bitcoin data on various VPS service separate from the VPS itself.

AWS S3 upto 50 TB:

jodobear commented 4 years ago

Additionally, scope how to incorporate BitcoinFiber for faster sync between nodes and Blockstream Sattelite for hosting multiple networks.

ChristopherA commented 4 years ago

Hosting Data

BTW, to be clear, Blockchain Commons is willing to pay for full hosting. In fact, I'd like to eventually also offer full Esplora (aka Blockstream.info). I just want to allow others to be able to follow us by offering redundant services more cheaply.

Another point of interest. Every time someone uses my Bitcoin course and sets up a Linode to give it a try, we get a $10 credit. So I have a bunch of unused credit on Linode. That, and that is is very easy for a new developer to set up a StackScript (as opposed to the much more complicated deployment systems), is why we currently use Linode. However, we don't have to.

Ultimately I'd like Blockchain Commons to be able to have 3 redundant hosts around the world.

-- Christopher Allen

ChristopherA commented 4 years ago

What other services are desirable?

Good question.

I know that we already use Esplora via Blockstream.info, and it is probably good for us to host an Esplora instance of our own.

Maybe a survey of each of these requirements? Also how stable are they? So far our scripts for self-hosting and our devops are an all volunteer group, so we have to consider ongoing support costs.

-- Christopher Allen

jodobear commented 4 years ago

Maybe a survey of each of these requirements? Also how stable are they?

Esplora needs 440GB after compaction (~1TB while indexing) and 270GB at 620k block height in --lightmode,

jodobear commented 4 years ago

I know that we already use Esplora via Blockstream.info

@ChristopherA I can get on top of this and would like to volunteer to lead on the Standup scripts for Linux.

Currently both Linode & Linux Standup scripts are pretty much the same which can be merged and tested.

Review @ NicolasDorier scripts in the @btcpayserver community, ... for best practices for our use as well as submit changes back.

We need better documentation of our environment variables for better configurability, relevant additions, add flags especially --help

Supporting some form of pre-signed download of pruned full-node data like BTCPay offers as an option.

Until we figure out hosting pre-signed data, I propose to use BTCPay Server's copy. That way we can at least test out the setup.

Proposed Milestones

  1. Add Help and merge Linode & Linux scripts and document the environment variables with additional prune mode and FastSync (if we agree on using BTCPay data).
  2. Add Esplora, HWI and BTCPay to the Stackscript.
  3. Add update.sh for the node.
  4. Incorporate LN integration (LND & c-lightning). (Dependency #7 )
  5. Add the Price History API service once it is complete. (Dependency #6)
  6. Add option for luke-jr's bitcoin knots and kallewoof's signet.

it is probably good for us to host an Esplora instance of our own.

Once, milestone 2 is achieved we can test the script on Linode and host a full node if we have the credits?

Tasks

Scope

Question: Do we want to add BTCPay Server to this script?

@Fonta1n3 what do you think?

Fonta1n3 commented 4 years ago

@jodobear

Proposed Milestones

  1. Add Help and merge Linode & Linux scripts and document the environment variables with additional prune mode and FastSync (if we agree on using BTCPay data).

This should certainly be number 1 priority. Number two priority should be:

  1. Add Esplora, HWI and BTCPay to the Stackscript.

I feel Esplora is too cumbersome for a VPS, can we make it optional?

  1. Add update.sh for the node.
  2. Incorporate LN integration (LND & c-lightning). (Dependency #7 )

I personally prefer c-lightning so would like to see that take priority, but beggars can't be choosers ;-)

  1. Add the Price History API service once it is complete. (Dependency #6)
  2. Add option for luke-jr's bitcoin knots and kallewoof's signet.

I am really not sure about adding knots or signet as it would probably just confuse people and I am sure we can add other more useful features instead?

Question: Do we want to add BTCPay Server to this script?

I think that would be great.

jodobear commented 4 years ago

align the scripts with Standup.app MacOS. As of the latest update there was a major change to how we create hidden services, we now create a seperate HS for all three rpcports so a user can run all three simultaneously and connect to all three remotely via independent HS. See the macos linux script here line 70 - 108. Currently the scripts for linux/vps create one HS that controls all three rpcports.

Meaning modify how we create HS for linux accordingly? If so, sure!

I feel Esplora is too cumbersome for a VPS, can we make it optional?

Agreed and yes it will be optional. Infact, the way I plan to implement it is only bitcoind will be the base installation, everything else will be optional based on user preference.

I personally prefer c-lightning so would like to see that take priority, but beggars can't be choosers ;-)

I have never worked with c-lightning, and this will be a great motivation for me to learn how to c-lightning. Since, LND is very easy, i will start with that and then also integrate c-lightning option or the other way round. I will look into it.

I am really not sure about adding knots or signet as it would probably just confuse people and I am sure we can add other more useful features instead?

I only added those since @ChristopherA mentioned it. But, i feel these are low priority and there's a lot to do before that. We can take a call on them at a later date.

What other services would you like to be added to the scripts?

ChristopherA commented 4 years ago

I feel Esplora is too cumbersome for a VPS, can we make it optional?

Yes, it is unlikely our typical BitcoinStandup VPS users will want to do Esplora, but I know we will want to do our own as it is needed for some future things like BTCR.

I am really not sure about adding knots or signet as it would probably just confuse people and I am sure we can add other more useful features instead?

Knots has the lowest priority, but I'd love to be able demonstrate that we can, and get @Luke-jr's endorsement.

Signet is potential much more useful for us, as I have concerns about FN2 performing well under adversarial conditions or if full nodes are tied up in forks and such. Signet can simulate those, making me feel much more confident about FN2 on mainnet.

Fibre could potentially also be very useful once we have more people doing BitcoinStandup, as we can lower latency between associated nodes. It also is a prerequisite for Blockstream Satellite.

--Christopher Allen

jodobear commented 4 years ago

Knots has the lowest priority, but I'd love to be able demonstrate that we can, and get @luke-jr's endorsement.

I like this idea, will be done! :grin:

javiervargas commented 4 years ago

When i installed standup.sh for using FullyNoded2 with my own mainnet node I don't needed perform IBD, so in addition to the first option to install all stuff (including two steps for c-lightning), 

sudo apt-get install -y software-properties-common
sudo add-apt-repository -u ppa:lightningnetwork/ppa
sudo apt-get install lightningd

I think it is a good idea to give the user different options depending on their situation. Perhaps creating a menu that calls different functions and executes the pre-checks is an ideal approach.

Something like this:

~~~~~~~~~~~~~~~~~~~~~
  BitcoinStandup.sh  
~~~~~~~~~~~~~~~~~~~~~
0. Install all
1. Install Tor
2. Install tools
3. Debian updates
4. Install ufw
5. Install StandUp User
6. Install Bitcoin core
7. Create and show QR Code
8. Install c-lightning
9. Exit
Enter choice [ 1 - 9]

Script standup.sh

show_menus() {
 clear
 echo "~~~~~~~~~~~~~~~~~~~~~"
 echo "  BitcoinStandup.sh  "
 echo "~~~~~~~~~~~~~~~~~~~~~"
 echo "0. Install all"
 echo "1. Install Tor"
 echo "2. Install tools"
 echo "3. Debian updates"
 echo "4. Install ufw"
 echo "5. Install StandUp User"
 echo "6. Install Bitcoin core"
 echo "7. Create and show QR Code"
 echo "8. Install c-lightning"
 echo "9. Exit"
}

read_options(){
 local choice
 read -p "Enter choice [ 1 - 9] " choice
 case $choice in
  1) one ;;
  2) user_info "who" ;;
  3) debian_updates ;;
  9) exit 0;;
  *) echo -e "${RED}Error...${STD}" && sleep 2
 esac
}

debian_updates(){
 echo "debian"
        pause
}

Waiting your comments. J.

jodobear commented 4 years ago

@javiervargas Would this require displaying the main menu repeatedly to install each service/item?

Here's how i'm thinking to implement the structure:

UX

  1. Display default installation info and give option for that or configure install
  2. If default then install.
  3. Else give one by one options for:
    1. Tor
    2. SSH
    3. Bitcoin - version (core, knots, signet), chain, prune, FastSync, txindex
    4. LN - LND, c-lightning
    5. HWI
    6. Esplora
    7. BTCPay Server
  4. Core components no matter the config:
    1. UFW
    2. QuickConnect

All of these options can be given as arguments to the script for a one liner.

Default Installation

All communication defaults through Tor. Services:

  1. UFW
  2. Tor
  3. SSH
  4. Bitcoin- bitcoin-core, full node, mainnet, FastSync=False, txindex=1
  5. QuickConnect
  6. LN - c-lightning
  7. HWI
  8. BTCPay Server
javiervargas commented 4 years ago

@jodobear Not exactly, the menu is in a do while cycle that you can interrupt at any time with ctrl-c.

jodobear commented 4 years ago

Ok, I'm currently deconstructing it, laying down best practices and putting the variables in place. I'll probably push a basic structure by this weekend and we can take it from there

jodobear commented 4 years ago

I've merged the scripts, added help, documentation, checks for each step and installing bitcoin from onion site:

ss.sh

It's tested on Debian 10 VM. Everything works and is setup properly.

Still need to test on VPS, don't have access at the moment.

Please provide feedback if i can do it better or making any mistakes.

shannona commented 3 years ago

Closing out summer 2020 issues. Thanks to @jodobear for work on the Standup Scripts!