Raspiblitz installation on a VM

BlockHackTk commented 4 years ago

Hello, I would like to ask for some help that I didn't get on the Raspiblitz Telegram.

I'm trying to run raspiblitz on a virtual machine. I just bought a Freebox V6 internet box on which you can run a VM in ARM! Here is a link to a tutorial to install a Bitcoin Core Node

So I tried on a Debian Buster VM to run the wget command https://raw.githubusercontent.com/rootzoll/raspiblitz/master/build_sdcard.sh && sudo bash build_sdcard.sh

Everything goes well and I'm at the end of the installation, but I have a problem with the HDD. I have a message that it is not connected (the HDD). I have to connect a HDD in Usb for it to be recognized.

The next problem is that the Network line of the config file is not filled and I have a problem. If I enter Bitcoin and Main in the file, I manage to move forward, but when I'm asked to download the Blockchain, I get a count from 0 to 200 and then it comes back to the start message to download the Blockchain.

I have some screenshots that I can share with you. I'm going to start again and make a recording of what happens to share it with you.

Maybe with your help I could make a tutorial to share with Freebox Delta S users.

An internet box is always running, H24 7/7 I think it's a good opportunity to run a bitcoin node and get interested in bitcoin. Free is a french FAI (in french) or ISP (Internet Service Provider)

I hope I wouldn't be the only owner of this box to want to try the adventure? I have the basic knowledge of Linux that I've been using for more than 20 years.

adevoss commented 4 years ago

@BlockHackTk Please try #1297 Possible data partitions are /dev/sdXn and /dev/vdXn (X a-z and n=1..9)

What is the data partition on your VM? Do you use a 'real' usb drive or a virtual drive (file act as drive) I can test the installation on my VM if that's of any help. My VM isn't arm but I can rename (link) the data partition to what your data partition is called.

BlockHackTk commented 4 years ago

Next

Thank you for your answers. I managed to move forward a little but now I don't stumble over the same mistake anymore. This freebox can be used as a NAS, there is the possibility to put 4 3.5" hard disks inside. For the moment, there are 2 disks, one of 160Gb and one of 750Gb, I put a 500Gb hard disk on the USB-A port, there is also a USB-C port but I don't have the hardware to use it for the moment. I'm going to invest in two SSDs, especially for the blockchain which will be in USB, it's the only way I can get the disk detected during the installation and I would respect the @Openoms instructions. The operation is as follows: we can install distributions, there is the choice and they are .qcow2 files that are downloaded. We determine a size for the VM system partition. Sorry I'm new to VM. I'll try to give you the partition table. @ChuckNorrison I didn't understand very well what you're asking for

BlockHackTk commented 4 years ago

Now I get to the password request C then I stop here Generating new Wallet .... Traceback (most recent call last): File "/home/admin/config.scripts/lnd.initwallet.py", line 7, in <module> import grpc ModuleNotFoundError: No module named 'grpc' FAIL!! -> MISSING seedwords data - but also no err data ?!? CHECK output data above - PRESS ENTER to restart 70initLND.sh

BlockHackTk commented 4 years ago

and now ` Copy LND Macaroons to user admin checking LND wallet ... (can take some time) manual input calling: lncli unlock unkown error

############################ Calling: lncli unlock Please re-enter Password C: Input wallet password: [lncli] rpc error: code = Unknown desc = wallet not found --> OK LND wallet unlocked ls: cannot access '/home/bitcoin/.lnd/data/chain/bitcoin/mainnet/admin.macaroon': No such file or directory

FAIL - LND Macaroons not created Please check the following LND issue: https://github.com/lightningnetwork/lnd/issues/890 You may want try again with starting ./70initLND.sh`

I think I'll come back when I've invested in SSDs. and I will give you the information you asked me for.

BlockHackTk commented 4 years ago

`admin@ttets:~$ sudo fdisk -l Disk /dev/vda: 100 GiB, 107374186496 bytes, 209715208 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 344B7EF9-4A32-4BB5-9356-40E028F26688

Device Start End Sectors Size Type /dev/vda1 2048 206848 204801 100M EFI System /dev/vda2 208896 209715174 209506279 99.9G Linux filesystem

Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors Disk model: External
Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 1ADBD983-FFD9-4F56-BF44-0568713C381F

Device Start End Sectors Size Type /dev/sda1 2048 976773119 976771072 465.8G Linux filesystem admin@ttets:~$ `

BlockHackTk commented 4 years ago

> Do you use a 'real' usb drive or a virtual drive (file act as drive) I'm using the : Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors Disk model: External Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 1ADBD983-FFD9-4F56-BF44-0568713C381F

Device Start End Sectors Size Type /dev/sda1 2048 976773119 976771072 465.8G Linux filesystem

BlockHackTk commented 4 years ago

My debug output https://termbin.com/cjisz

adevoss commented 4 years ago

@BlockHackTk What release did you install? Did you use my repository (adevoss/adevoss-more-datadrive-options-ext4)? My repository is v1.6 with my pull request. With rootzoll/v1.6 your datadrive MUST be /dev/sda1. From your debug info:

*** MOUNTED DRIVES ***
/dev/vda2      ext4     103154728 3936848  94995728   4% /
/dev/sda1      ext4     479669928 4213856 451020412   1% /mnt/hdd

Data partition is OK (/dev/sda1) using ext4. What did you gave as root partition when you first installed? /dev/vda2 should work when you use my repository. The root partition of a Raspberry Pi is /dev/mmcblk0p2. If you leave that default that could give unexpected results.

adevoss commented 4 years ago

Tested again with both repos. The trick with linking /dev/sda1 did not work. @BlockHackTk Please test it yourself with my repo.

rootzoll/v1.6: Had to add this line to /etc/fstab after executing install script. /dev/vdb2 /mnt/hdd ext4 noexec,defaults 0 2 Where vdb2 is my data partition

adevoss/adevoss-more-datadrive-options-ext4: Installation went fine.

With both repos LND won't start. Similar errors as https://github.com/rootzoll/raspiblitz/issues/1500#issuecomment-683468323 Can't help you with that. Sorry.

ChuckNorrison commented 4 years ago

In the related issue of your log, LND was not working cause of a small swap partition. Maybe you should try some suggestions on your LND error from this thread: https://github.com/lightningnetwork/lnd/issues/890

BlockHackTk commented 4 years ago

Thank you for all these ideas. I order RAM and SSD to match the recommended specifications. In the meantime I'll explore what you've explained to me and get back to you. @adevoss Your repo is interesting, I'll take a closer look at it.

ChuckNorrison commented 4 years ago

Iam interested in a RaspiBlitz environment without usage of raspi hardware too :+1: Maybe you can debug the setup steps to the point of first failure and analyze the dependencies on it.

Sounds the /mnt/hdd part worked so far but the lnd macaroons are not created as expected. This should be done while setup lnd from 70initLND.sh

rootzoll commented 4 years ago

Adding this to the v1.7 release .. still a bit as a research topic. But I also like to run RaspiBlitz for Development and with v1.7 I will refactor the setup process anyway - so to look into VM install makes also sense at this point.

But there seems very different "ideas" of running a VM needs to look like. @BlockHackTk you talk aboz a VM running on a Freebox, I would target a VirtualBox VM and @ChuckNorrison whats your target?

Also the question is where to store the blockchain? Would it be still on an external HDD that gets connetced to the VM?

Or @frennkie you liked the option to use an external bitcoind .. can we proxy and external bitcoind locally with all needed interfaces (bitcoin-cli & RPC) so that it looks for the rest of the RaspiBlitz as the bitcoind is running locally. And all the other scripts would need to work that there is no blockchain storage locally. Is all that work, worth of also doing this option?

nolith commented 4 years ago

I was working a on a vagrant setup. My main goal is testing new features on VM before the final test on the real HW.

I have a packer script that builds a vagrant bootable RaspiBlitz VirtualBox VM based on the mentioned PR.

I don;t have my computer with me at the moment, but IIRC one of the base config script was failing and I have to debug why.

It's a bit hard to make use of vagrant development capabilities because RaspiBlitz is designed to copy stuff in the user home, but the home contains also other things and it's not trivial to mount it from the developer machine.

I plan to write a vagrant provisioning script that will install a boot script that simply links files in /vagrant/admin.home/ into /home/admin.

Consider the amount of time I'm investing in this, it could have been cheaper to simply buy an extra rpi4 for development 😂

ChuckNorrison commented 4 years ago

@rootzoll I would like to run the RaspiBlitz in Xen or ESXi virtualization environment. I would like to use Ubuntu 20 LTS for that RaspiBlitz installation. Maybe a docker container setup could be fine to make it more portable.

rootzoll commented 4 years ago

@ChuckNorrison and @nolith how would you do the blockchain storage with the VM?

ChuckNorrison commented 4 years ago

I did a setup of a xen based citrix hypervisor 8.2. I would add a second partition or better a second vhd disk. Currently it is hard-coded to /mnt/hdd. A vhd disk would be configured within the partition table.

Path could be in example /blockchain rather than /mnt/hdd.

Alternatively /mnt/hdd is fine if mapped to NFS share or something.

// mount vhd for blockchain data manually to /mnt/hdd could be a possibility too

// we got some dependencies on the hostname raspberrypi? This host will have a different name.

nolith commented 4 years ago

I have a "working" vagrant VM (virtualbox) for development purpose running on testnet. You can see it https://github.com/rootzoll/raspiblitz/pull/1523

I've declared a secondary disk in vagrant and build my stuff on top of https://github.com/rootzoll/raspiblitz/pull/1297

In my experience there is a bug for my setup on that MR, but I managed to overcome it running the installation scripts manually as I wrote in https://github.com/rootzoll/raspiblitz/pull/1297#issuecomment-689433009

rootzoll commented 3 years ago

The setup process refactor is moved to v1.8 and also I need to wait a bit more until my Apple M1 is hopefully able to work with virtualbox again.

openoms commented 3 years ago

See my notes on installing the current dev branch (v1.7.1RC1+) to an Ubuntu v20.04 VM on TrueNAS: https://github.com/rootzoll/raspiblitz/issues/2104#issuecomment-917444238

Restoring and old virtual disk ran into permission issues with Tor: https://github.com/rootzoll/raspiblitz/issues/2531

Disk config at start:

$ lsblk
vda    252:0    0   32G  0 disk 
├─vda1 252:1    0  512M  0 part /boot/efi
└─vda2 252:2    0 31.5G  0 part /
vdb    252:16   0 1000G  0 disk

DrBlackross commented 3 years ago

hmm, is there a way to "roll" a install script (arm and x86) and compile and/with install? like the old gentoo days.

i have ton of parts sitting here to stuff in a 1ru space, with more overhead than a RPI4, i just like the raspiblitz setup and roll out. plus depending on the mobo headers, you can wire a serial epaper display to the front of the rack.

(i hate docker though lol)

nyxnor commented 3 years ago

https://github.com/SL-RU/ranger_udisk_menu This script draws menu to choose, mount and unmount drives using udisksctl and ncurses for ranger file manager

openoms commented 2 years ago

The VM installation is functional again. For testing just starting off with the virtual box image to be loaded (does not need installation): https://www.osboxes.org/ubuntu/ Password: osboxes.org can carry on straight to build the SDcard and attach another disk (can be even small if you prune or stop bitcoind ).

The latest SDcard build script:

# download
wget https://raw.githubusercontent.com/rootzoll/raspiblitz/dev/build_sdcard.sh
# build
sudo bash build_sdcard.sh false false rootzoll dev headless

to manage VMs I use this: https://virt-manager.org/ apt-get install virt-manager

nyxnor commented 2 years ago

does it work with the debian image or just ubuntu? also, not necessary to build from openoms/vmbuild?

On the virt-manager: Which network source should I choose, stuck here, the other options need to fill device name.

Edit: https://askubuntu.com/questions/1036297/cant-start-kvm-guest-network-default-is-not-active

sudo virsh net-list --all
sudo virsh net-start default

openoms commented 2 years ago

@nyxnor I leave everything default running the virt-manager on Debian Bullseye.

The debian image for the VM should work also and testing is welcome. I use Ubuntu because Debian didn't run on my TrueNAS bhyve VM before, but should be compatible with both.

SBCs do more often receive an Ubuntu image also.

My vmbuild branch is merged to rootzoll dev and that is aimed to be kept the most up-to-date.

nyxnor commented 2 years ago

thank you, it booted now

openoms commented 2 years ago

The quickest to get a running system on signet to use minimal space:

sudo systemctl stop bitcoind
sudo systemctl disable bitcoind
sudo systemctl stop tbitcoind
sudo systemctl disable tbitcoind

nano /mnt/hdd/raspiblitz.conf change to: chain=sig reboot: sudo reboot

The LND wallet password for signet and testnet is: raspiblitz slncli newaddress p2wkh coins: https://signetfaucet.bublina.eu.org/

Warning: pretty much only the lightning services are implemented for signet and it is known that LND and CL can't open channels to each other on signet (unless it is fixed already).

nyxnor commented 2 years ago

Openoms, does networking work out of the box with you? can get internet to start.

openoms commented 2 years ago

yes, it creates another LAN on my computer = NAT.

try to give root acces to the virt-manager

can also try another software like virtualbox

nyxnor commented 2 years ago

With virtualbox the network worked out of the box. Also, as I don't know how to scroll up on it, ssh to virtual box guide Also, to download virtualbox (whonix guide):

sudo apt update -y
sudo apt install fasttrack-archive-keyring  -y
echo 'deb https://fasttrack.debian.net/debian/ bullseye-fasttrack main contrib non-free' | sudo tee /etc/apt/sources.list.d/fasttrack.list
sudo apt update  -y
sudo apt install virtualbox linux-headers-$(dpkg --print-architecture)  -y
sudo adduser $(whoami) vboxusers

nyxnor commented 2 years ago

Failed to mount drive Using osboxes debian 64bit image Using virtualbox, NAT and ssh with portfowarding See https://github.com/rootzoll/raspiblitz/pull/2711#issuecomment-983047128

openoms commented 2 years ago

Add a second virtual disk to be used as the BLOCKCHAIN drive. This makes that data portable and independent from the OS similar with the combination of the SDcard and separate SSD.

adevoss commented 2 years ago

The debian image for the VM should work also and testing is welcome.

Used Debian as host OS and qemu/kvm from Debian repo as VM software. Didn't get it to work. Had to make some obvious changes to the sd card build script. At the end the bitcoind service did not start.

I will make a separate issue for this configuration.

openoms commented 2 years ago

@adevoss build from the dev branch. See these instructions: https://github.com/openoms/raspiblitz/tree/VM-instructions/alternative.platforms#virtual-machine

can comment here: https://github.com/rootzoll/raspiblitz/pull/2768

but yes a separate issue for your config detail would be useful for continuity

openoms commented 2 years ago

the VM install is functional, see: https://github.com/rootzoll/raspiblitz/tree/dev/alternative.platforms

raspiblitz / raspiblitz

Raspiblitz installation on a VM #1500