Vagrant build environment

TD-er commented 4 years ago

A warning to start with:

DO NOT try to run HyperV as a test.

See: https://github.com/letscontrolit/ESPEasy/issues/2594#issuecomment-528810624 to help fully disabling HyperV again. For now, HyperV is not supported as build platform since we have no easy way to share the build files.

As shortly introduced here: https://github.com/letscontrolit/ESPEasy/issues/2590#issuecomment-527070868

This is the first initial test to see if using Vagrant is a viable option for quickly building your own binaries.

To get started:

checkout the latest from the mega branch and go to tools/vagrant.
Read the (very short) readme about what to install
Go to that folder with a terminal/shell.

In Windows it can be CMD or Windows PowerShell and in Linux it can be any terminal. N.B. in Windows you can also quickly open a command shell by holding the shift and right click on the folder and select the Windows Power Shell.

In the opened shell:

Type vagant up

Please let me know if anything fails. If it is finished building something in the newly created build folder, then you can type vagrant halt in the same window you typed vagrant up.

Ideas for improving on this concept:

Parse a TXT (or YAML) file describing a build, or set of builds
Install Apache in the VM and run a web page where you can select your own set of builds
Accept JSON in the same Apache to let a node request its own new build (this requires the VM to have its own IP on the network)
Select a Git tag or PR to build. (the build script in tools already has a -p flag for building a PR)

For debugging: When the Vagrant box is running, you can connect to it using Putty (or any other ssh client).

User: vagrant
Pass: vagrant
Hostname: localhost
Port: 2222 (may change if you run multiple instances, see output on terminal)

The Github repository is cloned into:

Dir: ~/GitHub/letscontrolit/ESPEasy
User: vagrant

At boot of the Vagrant VM, the file bootstrap.sh is copied to the home directory of the vagrant user and stripped of typical Windows line endings (if present) (CRLF => Unix line endings) This is then executed, so for now, you can test with it to see if it needs to build some other environment.

Please note there is a shared folder between the Vagrant VM and your own OS. This is the same folder as where the Vagrantfile (no extension) and bootstrap.sh are located. From within the Vagrant VM these are available as a mount on /vagrant

The build is zipped like any nightly build and then this zip file is moved to the shared folder in a separate directory called build. (will be created by the script) So from within Vagrant this is /vargant/build/

One last note on the Vagrantfile. Do not use tabs in there, only spaces. On every call of the vagrant command, this file will be checked for errors.

To start over with a clean instance, call:

vagrant destroy

So in short, the commands:

vagrant up
vagrant up --provision (to run the apt-get install commands again)
vagrant halt
vagrant destroy

The build script will not (yet) update the Git Repository if the directory is already present. So either start over, or remove the Git directory or run from within ~/GitHub/letscontrolit/ESPEasy :

git fetch
git pull

LeeNX commented 4 years ago

@TD-er I have an ascii screen recording, about 275KB ... What is easiest to get this too you?

From what I can see, I think the problem is an missing apt-get update before doing apt-get install, as I remember seeing errors with missing debs.

uzi18 commented 4 years ago

On linux tools/build_espeasy.sh should still work easily without vargant

virtualbox has got recording feature

TD-er commented 4 years ago

There is a call to apt-get update in the Vagrant file:

  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y python-minimal virtualenv build-essential zip binutils
  SHELL

Maybe you could run vagrant up --provision on that one? It should be done the first time you call a vagrant up.

LeeNX commented 4 years ago

The error I saw might have been my VB plugin to upgrade the VB Tools.

But later, I have the following red/errors

    default: Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 virtualenv all 15.1.0+ds-1.1 [4,476 B]
    default: Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 zip amd64 3.0-11build1 [167 kB]

The following in red default: dpkg-preconfigure: unable to re-open stdin: No such file or directory

    default: Fetched 2,253 kB in 8s (297 kB/s)
    default: Selecting previously unselected package python-pip-whl.

Later, but I think it's just a warning ...

    default: Cloning into '/home/vagrant/GitHub/letscontrolit/ESPEasy'...
    default: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

then alot of red ...

    default: Platform Manager
    default: ================
    default:
    default: /home/vagrant/.platformio/packages/framework-arduinoespressif8266*//cores/esp8266/Esp.cpp
    default: : No such file or directory
    default: Patching /home/vagrant/.platformio/packages/framework-arduinoespressif8266*/
    default: patch: **** Can't change to directory '/home/vagrant/.platformio/packages/framework-arduinoespressif8266*/'
    default:  : No such file or directory
    default: Error: Unknown environment names 'custom_ESP8266_4M'. Valid names are 'memanalyze_ESP8266, normal_core_241_ESP8266_4M1M, dev_ESP8266_4M1M, normal_core_260_sdk222_alpha_ESP8266_4M1M, hard_SONOFF_POW_4M1M, normal_WROOM02_2M, normal_ESP8266_1M, test_core_260_sdk222_alpha_ESP8266_16M, esp32test_1M8_partition, test_core_260_sdk222_alpha_ESP8266_4M1M, normal_core_260_sdk222_alpha_ESP8266_16M, hard_other_POW_ESP8285_1M, minimal_IRext_ESP8266_4M2M, normal_core_241_ESP8266_1M, test_core_252_ESP8266_16M, minimal_core_252_ESP8285_1M_OTA, normal_core_242_ESP8266_1M, test_core_252_ESP8266_4M1M, esp-wrover-kit_test_1M8_partition, test_ESP8266_4M_VCC, hard_core_252_Shelly_1_2M256, normal_ESP8266_1M_VCC, custom_ESP8266_4M1M, test_core_242_ESP8266_4M1M, normal_core_252_ESP8266_16M, normal_ESP8285_1M, test_core_260_sdk3_alpha_ESP8266_4M1M, hard_Ventus_W266, normal_core_252_ESP8266_4M1M, minimal_IRext_ESP8266_4M1M, hard_core_252_SONOFF_POW_4M1M, normal_ESP8266_4M1M, normal_core_252_ESP8266_1M, minimal_core_242_ESP8285_1M_OTA, minimal_core_242_ESP8266_1M_OTA, normal_core_252_WROOM02_2M256, minimal_core_252_ESP8266_1M_OTA, hard_core_252_other_POW_ESP8285_1M, test_core_260_sdk3_alpha_ESP8266_16M, minimal_IRext_ESP8266_1M'
    default: /home/vagrant/bootstrap.sh: line 56: DESCRIPTION: command not found
    default: stat:
    default: cannot stat '.pio/build/custom_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment custom_ESP8266_4M1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/custom_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/custom_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_custom_ESP8266_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/custom_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_custom_ESP8266_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/dev_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment dev_ESP8266_4M1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/dev_ESP8266_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/dev_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_dev_ESP8266_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/dev_ESP8266_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_dev_ESP8266_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/esp-wrover-kit_test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment esp-wrover-kit_test_1M8_partition for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/esp-wrover-kit_test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/esp-wrover-kit_test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp-wrover-kit_test_1M8_partition.bin'
    default: mv:
    default: cannot stat '.pio/build/esp-wrover-kit_test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp-wrover-kit_test_1M8_partition.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/esp32test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment esp32test_1M8_partition for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/esp32test_1M8_partition/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/esp32test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp32test_1M8_partition.bin'
    default: mv:
    default: cannot stat '.pio/build/esp32test_1M8_partition/ESP_Easy_mega-20190830-16-ga68dc43_esp32test_1M8_partition.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_SONOFF_POW_4M1M/firmware.bin'
    default:
    default: ### Deploying environment hard_SONOFF_POW_4M1M for version mega-20190830-16-ga68dc43
    default: : No such file or directory
    default: cp:
    default: cannot stat '.pio/build/hard_SONOFF_POW_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:     with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError: [Errno 2] No such file or directory: '.pio/build/hard_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_SONOFF_POW_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_SONOFF_POW_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_Ventus_W266/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_Ventus_W266 for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_Ventus_W266/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_Ventus_W266/ESP_Easy_mega-20190830-16-ga68dc43_hard_Ventus_W266.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_Ventus_W266/ESP_Easy_mega-20190830-16-ga68dc43_hard_Ventus_W266.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_core_252_SONOFF_POW_4M1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_core_252_SONOFF_POW_4M1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_core_252_SONOFF_POW_4M1M/firmware.bin': No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_core_252_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_SONOFF_POW_4M1M.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_core_252_SONOFF_POW_4M1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_SONOFF_POW_4M1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_core_252_Shelly_1_2M256/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_core_252_Shelly_1_2M256 for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_core_252_Shelly_1_2M256/firmware.bin'
    default: : No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_core_252_Shelly_1_2M256/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_Shelly_1_2M256.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_core_252_Shelly_1_2M256/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_Shelly_1_2M256.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_core_252_other_POW_ESP8285_1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_core_252_other_POW_ESP8285_1M for version mega-20190830-16-ga68dc43
    default: cp:
    default: cannot stat '.pio/build/hard_core_252_other_POW_ESP8285_1M/firmware.bin': No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default: Traceback (most recent call last):
    default:   File "crc2.py", line 58, in <module>
    default:
    default: with open(FileName, mode='rb') as file: # b is important -> binary
    default: IOError
    default: :
    default: [Errno 2] No such file or directory: '.pio/build/hard_core_252_other_POW_ESP8285_1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_other_POW_ESP8285_1M.bin'
    default: mv:
    default: cannot stat '.pio/build/hard_core_252_other_POW_ESP8285_1M/ESP_Easy_mega-20190830-16-ga68dc43_hard_core_252_other_POW_ESP8285_1M.bin'
    default: : No such file or directory
    default: stat:
    default: cannot stat '.pio/build/hard_other_POW_ESP8285_1M/firmware.bin'
    default: : No such file or directory
    default:
    default: ### Deploying environment hard_other_POW_ESP8285_1M for version mega-20190830-16-ga68dc43
    default: cp: cannot stat '.pio/build/hard_other_POW_ESP8285_1M/firmware.bin': No such file or directory
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file

Might be just easier with the screen recording?

TD-er commented 4 years ago

Just make sure you are running the current state of the Github. So I guess perform a vagrant destroy on the VM, make sure you are up to date with the current HEAD of the Git branch and do a new vagrant up I think you may have tried the first time just between 2 commits.

LeeNX commented 4 years ago

Clean up the test, and did a git pull and looks much better. Testing again, just to make sure ... But I think this is working as expected!!! Thanks @TD-er

LeeNX commented 4 years ago

Clean re-run works as expected

    default: hard_Ventus_W266                           IGNORED
    default: ========================= 1 succeeded in 00:01:30.255 =========================
    default:
    default: ### Deploying environment custom_ESP8266_4M1M for version 20190902_vagrant
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default:
    default:
    default: BINARY PART
    default: Segments: 0x1
    default: SEGMENT 0: memory position: 0x4010f000 to 0x4010f568 length: 0x568
    default:
    default:
    default: BINARY PART
    default: Segments: 0x5
    default: SEGMENT 1: memory position: 0x40201010 to 0x402c2984 length: 0xc1974
    default: SEGMENT 2: memory position: 0x40100000 to 0x401000b0 length: 0xb0
    default: SEGMENT 3: memory position: 0x401000b0 to 0x40107684 length: 0x75d4
    default: SEGMENT 4: memory position: 0x3ffe8000 to 0x3ffe86d0 length: 0x6d0 <-- CRC is here. <-- filename is here.
    default: SEGMENT 5: memory position: 0x3ffe86e0 to 0x3ffe9ac0 length: 0x13e0
    default: hash includes segments: 1 2
    default: MD5 hash: a39c54476156b92412b2f174855b7eb9
    default:
    default: writing output file:
    default: .pio/build/custom_ESP8266_4M1M/ESP_Easy_20190902_vagrant_custom_ESP8266_4M1M.bin
    default:
    default: ### Creating zip archive

LeeNX commented 4 years ago

@TD-er ping me if you need anything else tested, else I think the base of the vagrant build system seems to work.

TD-er commented 4 years ago

Maybe @micropet (see https://github.com/letscontrolit/ESPEasy/issues/2590#issuecomment-527101151 ) can test by changing the version of the Vagrant image to see if it also works on HyperV. Described here

workgroupengineering commented 4 years ago

Hi @TD-er , Why didn't you use Docker?

Grovkillen commented 4 years ago

@workgroupengineering Why should we? Please give pros and cons.

TD-er commented 4 years ago

Just as a reply to @micropet on his post here: https://github.com/letscontrolit/ESPEasy/issues/2590#issuecomment-527336483

You do need any of the supported visualization platforms. If you plan to use HyperV, then you don't need to install Virtualbox (I would say you should even not install it) I don't have any experience with HyperV, but what I've read from it, you need administrator rights to create a new instance, so I guess you should start the "vagrant up" from a shell with administrator rights. Right now, we don't have the click-and-build site yet, so this is just a test to see if this is the right way to go. It will just build a predefined build. But later on it is planned to make this a very simple build env. to make your own build. Let's call it a pre-alpha stage :)

@workgroupengineering The reason I chose Vagrant, over Docker is very simple. I do know Vagrant and I have 0 experience with Docker. And also some of the already existing scripts we use for the nightly builds and for making a quick test build is based on running in Ubuntu. So for a quick-and-dirty test using Vagrant is by far the easiest for me to get some environment setup.

Also this is very easy to explain to the inexperienced and took me about 3 - 4 hours yesterday to setup. I guess it would take me more to read into Docker, create and test images and find a way to make running Docker images easy on all platforms people use. I know Tasmota does have a Docker image to build and I know you can run Docker images even on your NAS (the ones running x86 CPUs), so maybe for a future build env. it could be a good choice. For now just let's focus on one and see if people actually can run it.

LeeNX commented 4 years ago

So I can't demand a DOS build env? Hahaha ... Just adding my +1, because this will/does make it easier for me to build custom firmware for the ESPEasy eco system, thanks @TD-er

micropet commented 4 years ago

Thanks for the information. I can not get any further. I also have the same error message with Admin right:

d:\Soft\ESPEasy-mega\tools\vagrant>vagrant up Vagrant failed to initialize at a very early stage:

There was an error while executing VBoxManage, a CLI used by Vagrant for controlling VirtualBox. The command and stderr is shown below.

Command: ["--version"]

Stderr: Der Befehl ""c:\Program Files\Oracle\VirtualBox\VBoxManage.exe"" ist entweder falsch geschrieben oder konnte nicht gefunden werden.

TD-er commented 4 years ago

OK, I think you'll also need to patch the Vagrantfile then to set a provider. I can make a test Vagrantfile tomorrow and if that does work, we can have 2 definitions in the Vagrantfile so you just have to add an extra parameter to the vagrant up command For example vagrant up hyperv (or whatever name we will give the other definition)

micropet commented 4 years ago

Great.

TD-er commented 4 years ago

Have a look at Vagrant providers Especially the "Default Provider" section.

It seems like we can add a set of providers here. But as a first test, so we don't have to change anything, you could try: vagrant up --provider=hyperv

Edit: You still need to change the Vagrantfile like described here: https://github.com/letscontrolit/ESPEasy/issues/2590#issuecomment-527149932 Or maybe this will also work:

vagrant up --provider=hyperv --box-version 201812.27.0

micropet commented 4 years ago

With the original Vagrant File I get the following message:

vagrant up --provider=hyperv Bringing machine 'default' up with 'hyperv' provider... ==> default: Verifying Hyper-V is enabled... ==> default: Verifying Hyper-V is accessible... ==> default: Importing a Hyper-V instance default: Creating and registering the VM... default: Successfully imported VM default: Please choose a switch to attach to your Hyper-V instance. default: If none of these are appropriate, please open the Hyper-V manager default: to create a new virtual switch. default: default: 1) Default Switch default: 2) V-Switch default: default: What switch would you like to use? 2 default: Configuring the VM... ==> default: Starting the machine... ==> default: Waiting for the machine to report its IP address... default: Timeout: 120 seconds default: IP: 192.168.0.236 ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 192.168.0.236:22 default: SSH username: vagrant default: SSH auth method: private key default: default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... default: Removing insecure key from the guest if it's present... default: Key inserted! Disconnecting and reconnecting using new SSH key... ==> default: Machine booted and ready!

Vagrant requires administrator access for pruning SMB shares and may request access to complete removal of stale shares. ==> default: Preparing SMB shared folders... default: You will be asked for the username and password to use for the SMB default: folders shortly. Please use the proper username/password of your default: account. default: default: Username: xxxxxx default: Password (will be hidden):

Vagrant requires administrator access to create SMB shares and may request access to complete setup of configured shares. ==> default: Mounting SMB shared folders... We couldn't detect an IP address that was routable to this machine from the guest machine! Please verify networking is properly setup in the guest machine and that it is able to access this host.

As another option, you can manually specify an IP for the machine to mount from using the smb_host option to the synced folder.

workgroupengineering commented 4 years ago

Hi @Grovkillen ,

Why docker?

runs on Raspberry Pi !!!!
has become a world standard
there is so much documentation on the net
works with both virtualbox and hyper-v without any configuration

Why not docker?

high learning curve

TD-er commented 4 years ago

runs on Raspberry Pi !!!!

Compiling on a R'pi is not really fun. But it is one of the most available "servers" among users. I think the build script we use, may also run on a Pi, so I guess we can check where things need to be adapted to work on a Pi too.

Like I said, we can later look into it, but for now the 'con' outweighs the 'pros'. First let's see what is a setup with a high UX and is easy to use and then we can focus on other container techniques.

TD-er commented 4 years ago

Vagrant requires administrator access to create SMB shares and may request access to complete setup of configured shares. ==> default: Mounting SMB shared folders... We couldn't detect an IP address that was routable to this machine from the guest machine! Please verify networking is properly setup in the guest machine and that it is able to access this host.

As another option, you can manually specify an IP for the machine to mount from using the smb_host option to the synced folder.

Right now I did not add the option for routable IPs, since it is for now a test setup to see where people run into. Also making a VM available on the entire network does require some extra precautions. What if we 'depend' on it and there is no DHCP running? How about security? (default no problem, but if we add web services later it might) If it is routable, then people will make it run 24/7 on their network.

So for the first test I explicitly did not add the routed IP option.

Did it build a version for you?

uzi18 commented 4 years ago

on linux (rpi) you not need this, just use buildESPEasy.sh script it use virtualenv so will no mess with your system/profile dir

TD-er commented 4 years ago

Only thing may be that the reported packages you should install using apt-get may have different names on the Pi, since it is Debian based.

micropet commented 4 years ago

Good question. Not in my ESPEasy folder.

Where should the version be found?

TD-er commented 4 years ago

Good question. Not in my ESPEasy folder.

Where should the version be found?

In a newly created build folder at the same location where the Vagrantfile is located.

N.B. you should also have the bootstrap.sh file in the same directory as the Vagrantfile or else it will do nothing.

Maybe you have run it multiple times already, so it is not considering it a provision step.

vagrant halt
vagrant destroy
vagrant up --provider=hyperv --box-version 201812.27.0

You can also trigger a provision with adding --provision

uzi18 commented 4 years ago

Only thing may be that the reported packages you should install using apt-get may have different names on the Pi, since it is Debian based.

it should only need python,pip,virtualenv and git :) this is for pro users they will find all in their distros we should document types of build envs and pros/cons

micropet commented 4 years ago

A biuld Folder was not created. The Vagrant commands are not executed:

vagrant halt

Vagrant failed to initialize at a very early stage:

There was an error while executing VBoxManage, a CLI used by Vagrant for controlling VirtualBox. The command and stderr is shown below.

Command: ["--version"]

Stderr: Der Befehl ""c:\Program Files\Oracle\VirtualBox\VBoxManage.exe"" ist entweder falsch geschrieben oder konnte nicht gefunden werden.

TD-er commented 4 years ago

Hmm, it is really trying to do stuff with Virtualbox here. You can force the destroy of the VM instance with a -f. Maybe you really should test with the added box-version parameter. That one has HyperV mentioned and the later ones don't.

uzi18 commented 4 years ago

@micropet is your Virtualbox installed on this path c:\Program Files\Oracle\VirtualBox\VBoxManage.exe?

TD-er commented 4 years ago

@uzi18 He should have none installed since he is using HyperV.

micropet commented 4 years ago

No c:\Program Files\Oracle\VirtualBox\VBoxManage.exe does not exist.

I also have no virtualbox installed. I do not know where the path is coming from.

vagrant up --provider=hyperv is the only command that works. All other commands will give the error message.

I have just uninstalled and reinstalled vagrant Is the same - no build

Budman1758 commented 4 years ago

Installed both Vagrant and Virtualbox without issues. Ran the vagrant file and everything updated and then created the zipfile without any errors at all. Worked great.

I did have to upgrade Powershell first as I'm on Win7 and had v2 installed. Upgraded to v4 and all went well from there.

Zip file contained 4 blank bins, ESPEasy_2step_UploaderMega_1024 and ESP_Easy_20190904_vagrant_custom_ESP8266_4M1M

Where to from here?

TD-er commented 4 years ago

@Budman1758 I want to add the option to parse a TXT/INI/YAML file if present next to the other files. Maybe at first it is the simplest option to simply state the plugins to include. I think the ".ini" syntax is then the most forgiving.

Not sure yet if I should add a Python script for it, or let it include in the existing platformio.ini.

uzi18 commented 4 years ago

@micropet for hyperV try set box to generic/ubuntu1804 in Vagrantfile @TD-er it is also available for all types of providers: https://app.vagrantup.com/generic/boxes/ubuntu1804/versions/1.9.26 but it is 1GB of data (almost 2x more to fetch)

LeeNX commented 4 years ago

Just updated to VirtualBox v6.0.12, Vagrant v2.2.5 on OSX as host and was able to successfully build custom firmware

    default: ========================= 1 succeeded in 00:02:36.155 =========================
    default:
    default: ### Deploying environment custom_ESP8266_4M1M for version 20190905_vagrant
    default:
    default:
    default: Replacing dummy MD5 checksum in .bin file
    default:
    default:
    default: BINARY PART
    default: Segments: 0x1
    default: SEGMENT 0: memory position: 0x4010f000 to 0x4010f568 length: 0x568
    default:
    default:
    default: BINARY PART
    default: Segments: 0x5
    default: SEGMENT 1: memory position: 0x40201010 to 0x402c27e4 length: 0xc17d4
    default: SEGMENT 2: memory position: 0x40100000 to 0x401000b0 length: 0xb0
    default: SEGMENT 3: memory position: 0x401000b0 to 0x40107684 length: 0x75d4
    default: SEGMENT 4: memory position: 0x3ffe8000 to 0x3ffe86d0 length: 0x6d0 <-- CRC is here. <-- filename is here.
    default: SEGMENT 5: memory position: 0x3ffe86e0 to 0x3ffe9ac0 length: 0x13e0
    default: hash includes segments: 1 2
    default: MD5 hash: fc9b8f9188be527d9886f16c5ec9c30c
    default:
    default: writing output file:
    default: .pio/build/custom_ESP8266_4M1M/ESP_Easy_20190905_vagrant_custom_ESP8266_4M1M.bin
    default:
    default: ### Creating zip archive

TD-er commented 4 years ago

@uzi18 When I used Vagrant in the past, the "Bento" images always were my favorite, since they had the most sensible defaults and clean selection. So when preparing .deb files, you simply knew the ones that would run on that image would have all the needed dependencies mentioned in the .deb files. Also no hassle with pre-configured default firewalls etc that may be present on some other images.

I think listing a number of supported providers in the Vagrantfile is also a good test. If providers (multiple) are mentioned, you also set a pre-defined order of providers to test. It looks like @micropet does experience an inconsistent or undesired order of checks here.

If no provider is given in the Vagrantfile, you should give it on the command line for every (!!) vagrant command. Not just for the vagrant up. It is telling Vagrant how to communicate with the VM instance to execute the command you're giving.

uzi18 commented 4 years ago

If I understand you need to call vargant up in tools/vagrant folder

TD-er commented 4 years ago

If I understand you need to call vargant up in tools/vagrant folder

Yep, but if you need to specify anything else to make it work, you should add it to that command line, or add it in the Vagrantfile.

For example: vagrant up --provider=hyperv --box-version 201812.27.0 The box-version is probably only needed when calling up. But the --provider option may be needed for all commands interacting with the VM. Meaning:

vagrant up --provider=hyperv 
vagrant halt --provider=hyperv 
vagrant destroy --provider=hyperv

micropet commented 4 years ago

Thank you. I did that. Now no virtual machine is created. and it looks like this:

d:\Soft\ESPEasy-mega\tools\vagrant>vagrant up --provider=hyperv Bringing machine 'default' up with 'hyperv' provider... ==> default: Verifying Hyper-V is enabled... ==> default: Verifying Hyper-V is accessible... ==> default: Box 'generic/ubuntu1804' could not be found. Attempting to find and install... default: Box Provider: hyperv default: Box Version: 1.9.26 ==> default: Loading metadata for box 'generic/ubuntu1804' default: URL: https://vagrantcloud.com/generic/ubuntu1804 ==> default: Adding box 'generic/ubuntu1804' (v1.9.26) for provider: hyperv default: Downloading: https://vagrantcloud.com/generic/boxes/ubuntu1804/versions/1.9.26/providers/hyperv.box default: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com default: The box failed to unpackage properly. Please verify that the box file you're trying to add is not corrupted and that enough disk space is available and then try again. The output from attempting to unpackage (if any):

x Vagrantfile x Virtual Hard Disks/generic-ubuntu1804-hyperv.vhdx: Write failed x Virtual Machines/F662D1AB-1038-4459-89D6-AF4327E5C7DF.VMRS: Write failed x Virtual Machines/F662D1AB-1038-4459-89D6-AF4327E5C7DF.vmcx: Write failed x Virtual Machines/F662D1AB-1038-4459-89D6-AF4327E5C7DF.vmgs: Write failed x Virtual Machines/box.xml: Write failed x info.json x metadata.json bsdtar.EXE: Error exit delayed from previous errors.

TD-er commented 4 years ago

The obvious questions:

Enough free drive space (not sure where the image is downloaded temporary)
Permission to write/execute?
No HyperV instance running from previous attempts?

Maybe I should also enable HyperV here, to see what's happening

uzi18 commented 4 years ago

if you enable hyperv virtualbox will prabably not run anymore till disable it, there are some hacks with bcd boot menu

micropet commented 4 years ago

I came a bit further. I only had 5 GB left on C: free, so he could not unpack the file. I now use the following Vagrant File:

Vagrant.configure("2") do |config| config.vm.box = "generic/ubuntu1804" config.vm.box_version = "1.9.26" config.vm.provider "hyperv" config.vm.network "public_network" config.vm.synced_folder ".", "/vagrant", disabled: true config.vm.provider "hyperv" do |h| h.enable_virtualization_extensions = true h.linked_clone = true end end

messages:

d:\Soft\ESPEasy-mega\tools\vagrant>vagrant up --provider=hyperv Bringing machine 'default' up with 'hyperv' provider... ==> default: Verifying Hyper-V is enabled... ==> default: Verifying Hyper-V is accessible... default: Configuring the VM... ==> default: Starting the machine... ==> default: Waiting for the machine to report its IP address... default: Timeout: 120 seconds default: IP: 192.168.0.228 ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 192.168.0.228:22 default: SSH username: vagrant default: SSH auth method: private key ==> default: Machine booted and ready! ==> default: Machine already provisioned. Run vagrant provision or use the --provision ==> default: flag to force provisioning. Provisioners marked to run always will still run.

The .bin file is not created.

TD-er commented 4 years ago

Have you tried to destroy it and created again? I guess it could have been an impartial run due to the previous issues. It will still use the already downloaded images, so it will not take long to redo.

micropet commented 4 years ago

Now it looks better. I had forgotten the bootstrap.sh part. ;)

....... default: Setting up python (2.7.15~rc1-1) ... default: Setting up binutils-x86-64-linux-gnu (2.30-21ubuntu1~18.04.2) ... default: Setting up cpp (4:7.4.0-1ubuntu2.3) ... default: Setting up binutils (2.30-21ubuntu1~18.04.2) ... default: Setting up gcc-7 (7.4.0-1ubuntu1~18.04.1) ... default: Setting up g++-7 (7.4.0-1ubuntu1~18.04.1) ... default: Setting up gcc (4:7.4.0-1ubuntu2.3) ... default: Setting up dpkg-dev (1.19.0.5ubuntu2.2) ... default: Setting up g++ (4:7.4.0-1ubuntu2.3) ... default: update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode default: Setting up build-essential (12.4ubuntu1) ... default: Processing triggers for libc-bin (2.27-3ubuntu1) ... ==> default: Running provisioner: shell... default: Running: inline script

But no .bin File

I do an destroy ..

TD-er commented 4 years ago

Also double check you have the latest bootstrap.sh file

micropet commented 4 years ago

The apt-get update and apt-get install work.

Then the bootstrap.sh is not found. It's in the Vagrant directory.

In the Vagrantfile stands: config.vm.provision "shell", run: "always", privileged: false do | s | s.inline = "sed 's / \ r $ // g' /vagrant/bootstrap.sh> ~ / bootstrap.sh; / bin / bash --login ~ / bootstrap.sh"

Error message:

==> default: Running provisioner: shell ... default: Running: inline script default: sed: can not read /vagrant/bootstrap.sh: No such file or directory

TD-er commented 4 years ago

Hmm, what image do you use, the bento one, or another? Can you ssh to the VM (running on port 2222 I assume) and see what mount points are active?

For example df -h

vagrant@vagrant:~$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          463M     0  463M   0% /dev
tmpfs                          99M  4.9M   94M   5% /run
/dev/mapper/vagrant--vg-root   62G  2.2G   57G   4% /
tmpfs                         493M     0  493M   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         493M     0  493M   0% /sys/fs/cgroup
tmpfs                          99M     0   99M   0% /run/user/1000
vagrant                       476G  183G  294G  39% /vagrant
vagrant@vagrant:~$

As you can see, the last line (doesn't have to be the last one) lists one mount point on /vagrant That should be the shared dir between the Windows PC and the VM.

TD-er commented 4 years ago

Hmm your output doesn't look like mine, so you must be using some other image than I defined in the Vagrantfile. This is my output:

PS C:\GitHub\TD-er\ESPEasy\tools\vagrant> vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'bento/ubuntu-18.04' version '201906.18.0' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => C:/GitHub/TD-er/ESPEasy/tools/vagrant
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
==> default: Running provisioner: shell...
    default: Running: inline script
....

micropet commented 4 years ago

i have config.vm.box = "generic/ubuntu1804"

if i only use: s.inline = "sed 's/\r$//g' bootstrap.sh > ~/bootstrap.sh; /bin/bash --login ~/bootstrap.sh"

(without path to bootstrap.sh) i get:

vagrant>vagrant provision ==> default: Running provisioner: shell... default: Running: inline script default: Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease default: Hit:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease default: Hit:3 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease default: Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease default: Reading package lists... default: Reading package lists... default: Building dependency tree... default: default: Reading state information... default: build-essential is already the newest version (12.4ubuntu1). default: python-minimal is already the newest version (2.7.15~rc1-1). default: zip is already the newest version (3.0-11build1). default: virtualenv is already the newest version (15.1.0+ds-1.1). default: binutils is already the newest version (2.30-21ubuntu1~18.04.2). default: 0 upgraded, 0 newly installed, 0 to remove and 150 not upgraded. ==> default: Running provisioner: shell... default: Running: inline script

/etc$ df -h Filesystem Size Used Avail Use% Mounted on udev 923M 0 923M 0% /dev tmpfs 191M 6.6M 185M 4% /run /dev/sda3 30G 2.5G 26G 9% / tmpfs 953M 0 953M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 953M 0 953M 0% /sys/fs/cgroup /dev/sda1 464M 141M 295M 33% /boot tmpfs 191M 0 191M 0% /run/user/1000

i test now with bento/ubuntu-18.04

TD-er commented 4 years ago

Apparently this generic image doesn't supply a shared mount between the guest and host and that's what I'm using :)

letscontrolit / ESPEasy

Vagrant build environment #2594

DO NOT try to run HyperV as a test.