search

Linutronix / elbe

Embedded Linux Build Environment
https://elbe-rfs.org
GNU General Public License v3.0
168 stars 59 forks source link

Elbe failed to run the hello world example from tutorial on a fresh installed debian #409

Closed cnnblike closed 3 months ago

cnnblike commented 3 months ago

Hi, I'm now running a fresh installed debian 12.6 with latest elbe from your official repository. However it keep failing with strictly following the tutorial on https://elbe-rfs.org/docs/sphinx/v15.0/article-quickstart.html on the step "Submitting an XML file".

Minor tweaking around SubmitAction(InitVMAction) by adjusting the following line:

ps = run_elbe(['initvm', 'ensure', *additional_params], capture_output=False,

Gives me some more information:

cnnblike@kl-desktop:~/workplace/elbe$ elbe initvm submit examples/x86_64-pc-rescue-busybox-dyn-cpio.xml
*
***********Waited for 2.0 minutes and the daemon is still not active.
Starting the initvm Failed
Giving up

I guess this shall be related to something going wrong on the previous "setting up initvm" step. I've attached log output of that step, could you please take a look into it?

installer.log

t-8ch commented 3 months ago

The installer.log you sent looks fine.

What is the output of virsh --connect qemu:///system dominfo initvm?

If the state is running, log into the initvm with virsh --connect qemu:///system console initvm (root:root) and post the output of systemctl status python3-elbe-daemon.

cnnblike commented 3 months ago

Hi, t-8ch@, I've erase the full image on my machine, another fresh install stuck in the first step with log at /var/log/syslog ends like below, it seems like some magic source from http://ftp.de.debian.org/debian/pr is refusing my connection. 

Jul 30 17:27:57 in-target: Get:179 http://ftp.de.debian.org/debian bullseye-backports/main amd64 reprepro amd64 5.3.1-1~bpo11+1 [453 kB]
Jul 30 17:27:57 in-target: Get:180 http://ftp.de.debian.org/debian bullseye/main amd64 squashfs-tools amd64 1:4.4-2+deb11u2 [135 kB]
Jul 30 17:27:57 in-target: Get:181 http://ftp.de.debian.org/debian bullseye/main amd64 sudo amd64 1.9.5p2-3+deb11u1 [1,061 kB]
Jul 30 17:27:58 in-target: Get:182 http://ftp.de.debian.org/debian bullseye/main amd64 subversion amd64 1.14.1-3+deb11u1 [997 kB]
Jul 30 17:28:00 in-target: Get:183 http://ftp.de.debian.org/debian bullseye/main amd64 python3-tz all 2021.1-1 [34.8 kB]
Jul 30 17:28:01 in-target: Get:184 http://ftp.de.debian.org/debian bullseye/main amd64 python3-pyparsing all 2.4.7-1 [109 kB]
Jul 30 17:28:01 in-target: Get:185 http://ftp.de.debian.org/debian bullseye/main amd64 python3-spyne all 2.13.16-1 [278 kB]
Jul 30 17:28:03 in-target: Fetched 118 MB in 7min 27s (264 kB/s)
Jul 30 17:28:03 in-target: E: Failed to fetch http://ftp.de.debian.org/debian/pr. Remote end closed connection [IP: 141.76.2.4 80]deb  Error reading from r
Jul 30 17:28:03 in-target: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Jul 30 17:28:03 main-menu[2749]: (process:26786): dpkg-divert: warning: divertirous, use --no-renamestop-daemon' from an Essential package with rename is r
Jul 30 17:28:03 main-menu[2749]: (process:26786): dpkg-divert: warning: divertirous, use --no-renamestop-daemon' from an Essential package with rename is r
Jul 30 17:28:03 main-menu[2749]: (process:26786): dpkg-divert: warning: divertirous, use --no-renamestop-daemon' from an Essential package with rename is r
Jul 30 17:28:03 main-menu[2749]: (process:26786): dpkg-divert: warning: divertirous, use --no-renamestop-daemon' from an Essential package with rename is r
Jul 30 17:28:03 main-menu[2749]: (process:26786): dpkg-divert: warning: divertirous, use --no-renamestop-daemon' from an Essential package with rename is r
Jul 30 17:28:03 main-menu[2749]: WARNING **: Configuring 'pkgsel' failed with error code 100
Jul 30 17:28:03 main-menu[2749]: WARNING **: Menu item 'pkgsel' failed.
Jul 30 17:32:22 main-menu[2749]: INFO: Modifying debconf priority limit from 'high' to 'medium'
Jul 30 17:32:22 debconf: Setting debconf/priority to medium
Jul 30 17:32:23 main-menu[2749]: INFO: Falling back to the package description for brltty-udeb
Jul 30 17:32:29 main-menu[2749]: INFO: Falling back to the package description for brltty-udeb
Jul 30 17:32:29 main-menu[2749]: INFO: Menu item 'di-utils-shell' selected

after re-enable from the installation sequence, it's now like this:

20: Execute a shell,    
 21: Abort the installation,    
Prompt: '?' for help, default=14> 

Select and install software  ... 1%... 10%
Select and install software
---------------------------

Applying updates on a frequent basis is an important part of keeping the system
secure.

By default, security updates are not automatically installed, as security 
advisories should be reviewed before manual installation of the updates using 
standard package management tools.

Alternatively the unattended-upgrades package can be installed, which will 
install security updates automatically. Note however that automatic 
installation of updates may occasionally cause unexpected downtime of services 
provided by this machine in the rare cases where the update is not fully 
backward-compatible, or where the security advisory requires the administrator 
to perform some other manual operation.
The system is ghaveged: haveged: Stopping due to signal 15                      71%... 82%
Installing GRUB boot loader  ... 16%... 3... 21%... 32%... 42%... 50%... 60%... 
Sent SIGKILL to all processes.. 3%... 10%
Requesting system reboot
[17083.464278] reboot: Restarting system
qemu-img create -f qcow2 -F qcow2 -b initvm-base.img initvm.img
Formatting 'initvm.img', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=85899345920 backing_file=initvm-base.img backing_fmt=qcow2 lazy_refcounts=of6
libvirt: Storage Driver error : Cannot access storage file '/home/cnnblike/workplace/elbe-base/initvm/initvm.img' (as uid:64055, gid:64055): Permission denied
Traceback (most recent call last):
  File "/usr/bin/elbe", line 56, in <module>
    cmdmod.run_command(sys.argv[2:])
  File "/usr/lib/python3/dist-packages/elbepack/commands/initvm.py", line 94, in run_command
    action.execute(directory, opt, args[1:])
  File "/usr/lib/python3/dist-packages/elbepack/initvmaction.py", line 164, in execute
    self.initvm.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1373, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: Cannot access storage file '/home/cnnblike/workplace/elbe-base/initvm/initvm.img' (as uid:64055, gid:64055): Permission denied
Starting the initvm Failed
Giving up
t-8ch commented 3 months ago

I'm a bit confused be the two logs.

The presence of installer.log in the first one indicates v15.0 while the usage of ftp.de.debian.org bullseye and the permission error at the end indicate an older version.

Could you clarify which version of elbe you are using and try to get back to v15.0?

cnnblike commented 3 months ago

hi, both of them are on fresh installed debian 12 (running on different flash drive but I don't think it's the root cause).
The second one log was from this:

cnnblike@kl-desktop:~/workplace/elbe-base$ elbe --version
elbe v14.9.3
cnnblike@kl-desktop:~/workplace/elbe-base$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:    12
Codename:   bookworm

Maybe it's caused by different debian apt source list? would you mind share which mirror do you use?

t-8ch commented 3 months ago

The official mirror is at http://debian.linutronix.de. See: https://elbe-rfs.org/download.html

This should give the following source line:

deb [signed-by=/usr/share/keyrings/elbe-archive-keyring.gpg] http://debian.linutronix.de/elbe bookworm main

Could you share the output of apt policy elbe?

You can also enable the elbe repo through extrepo:

apt install extrepo
extrepo enable elbe
apt update
apt install elbe
cnnblike commented 3 months ago

Aahh, I think I spot the blind point. I just copied from the source setting up line from tutorial and use the "buster" instead of "bookworm" here without much think about this. I'm now setting up with elbe 15.0, let me check if things goes correct.

cnnblike commented 3 months ago

Hmm, it looks strange now. It seems like everything is setting up correctly, which is expected, but still can't submit xml to initvm tho.

Configuring apt  ... 7%... 14%... 14%... 21%... 35%... 40%... 50%... 64%... 71%... 85%... 92%... 100%
Select and install software  ... 1%... 10%... 13%... 20%... 30%... 40%... 50%... 61%... 71%... 81%... 78%... 80%... 90%... 92%... 100%
Installing GRUB boot loader  ... 16%... 33%... 50%... 66%... 83%... 100%
The system is going down NOW!.. 3%... 11%... 22%... 33%... 40%... 44%... 51%... 62%... 70%... 81%... 92%
Sent SIGTERM to all processes
haveged: haveged: Stopping due to signal 15

Sent SIGKILL to all processes
Requesting system reboot
[ 1145.723851] reboot: Restarting system
qemu-img create -f qcow2 -F qcow2 -b initvm-base.img initvm.img
Formatting 'initvm.img', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=85899345920 backing_file=initvm-base.img backing_fmt=qcow2 lazy_refcou6
*****
cnnblike@kl-desktop:~/workplace/elbe-base$ ls
elbe  initvm
cnnblike@kl-desktop:~/workplace/elbe-base$ cd elbe/
cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ ls
AUTHORS          conftest.py  CONTRIBUTE.adoc  debian  elbe      elbe.spec     examples       pyproject.toml  setup.cfg  test   THANKS
bash-completion  contrib      COPYING          docs    elbepack  elbevalidate  newsfragments  README.adoc     setup.py   tests  website
cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ elbe initvm submit examples/x86_64-pc-rescue-busybox-dyn-cpio.xml
Starting the initvm Failed
Giving up
cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ virsh --connect qemu:///system dominfo initvm
Id:             1
Name:           initvm
UUID:           019108b4-dd27-7437-b7d2-ae9396e0ec9b
OS Type:        hvm
State:          running
CPU(s):         8
CPU time:       15.3s
Max memory:     1048576 KiB
Used memory:    1048576 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: apparmor
Security DOI:   0
Security label: libvirt-019108b4-dd27-7437-b7d2-ae9396e0ec9b (enforcing)
Messages:       tainted: custom configuration parameters specified

cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ virsh --connect qemu:///system console initvm
Connected to domain 'initvm'
Escape character is ^] (Ctrl + ])

elbe-daemon login: root
Password: 
Linux elbe-daemon 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@elbe-daemon:~# systemctl status python3-elbe-daemon
● python3-elbe-daemon.service - ELBE Daemon
     Loaded: loaded (/lib/systemd/system/python3-elbe-daemon.service; enabled; >
     Active: active (running) since Wed 2024-07-31 12:31:57 UTC; 1min 2s ago
       Docs: man:elbe-daemon(1)
   Main PID: 518 (elbe)
     Status: "Serving requests"
      Tasks: 2 (limit: 1104)
     Memory: 92.5M
        CPU: 431ms
t-8ch commented 3 months ago

Does curl localhost:7587/soap?wsdl return output?

cnnblike commented 3 months ago

on host machine, it's 

cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ curl localhost:7587/soap?wsdl
not found

in initvm, it's like 

root@elbe-daemon:~# curl localhost:7587/soap?wsdl
connect to localhost port 7587 after 0 ms: Couldn't connect to server

I also checked journalctl -xe in vm, also nothing special

t-8ch commented 3 months ago

This is only supposed to work outside the initvm. What about curl localhost:7587/?wsdl

cnnblike commented 3 months ago

also not found, I just checked occupied port

cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      900/cupsd           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      921/sshd: /usr/sbin 
tcp        0      0 0.0.0.0:7587            0.0.0.0:*               LISTEN      13266/qemu-system-x

the request should have been routed to qemu machine from my understanding.

t-8ch commented 3 months ago

Can you apply c105bf6fc42a1369cb621b3ebeded040578e9237 inside the initvm and then systemctl restart python3-elbe-daemon. I'm wondering why this now(?) also affects the soap server, will investigate.

t-8ch commented 3 months ago

FYI for the specific example you are using you also need to apply 6b90d50cfdcdf3414bf162dc4c1c80390665d533 (at least the one line to elbepack/finetuning.py)

I'll try to get a release out with these fixes as fast as possible.

cnnblike commented 3 months ago

em, maybe that's why, the elbepacks inside that vm is the old version I guess

import elbepack.daemons

def _not_found(start_response):
    start_response('404 Not Found', [('Content-Type', 'text/plain')])
    return [b'not found']

class _WsgiDispatcher:
    def __init__(self, mapping):
        self.mapping = mapping

    def __call__(self, environ, start_response):
        path_info = environ['PATH_INFO']
        parts = path_info.split('/', maxsplit=2)
        if len(parts) != 3:
            return _not_found(start_response)
        _, app_name, _ = parts

        app = self.mapping.get(app_name)
cnnblike commented 3 months ago

ok, I solved the issue by directly copy whatever we have under in the master branch fro elbepack. It looks like the latest one should be good now.

root@elbe-daemon:~# git clone https://github.com/Linutronix/elbe/
Cloning into 'elbe'...
remote: Enumerating objects: 22854, done.
remote: Counting objects: 100% (5313/5313), done.
remote: Compressing objects: 100% (1447/1447), done.
remote: Total 22854 (delta 4064), reused 5015 (delta 3811), pack-reused 17541
Receiving objects: 100% (22854/22854), 9.46 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (17126/17126), done.
root@elbe-daemon:~# cp elbe/
AUTHORS                 elbepack/               pyproject.toml
bash-completion         elbe.spec               README.adoc
conftest.py             elbevalidate/           setup.cfg
contrib/                examples/               setup.py
CONTRIBUTE.adoc         .git/                   test/
COPYING                 .git-blame-ignore-revs  tests/
debian/                 .gitignore              THANKS
docs/                   newsfragments/          website/
elbe                    .pylintrc                                                                         elbe/elbepack/* /usr/lib/python3/dist-packages/elbepackn3/dist-packages/elbepack 
root@elbe-daemon:~# reboot

after that:

cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ curl localhost:7587/soap?wsdl
<?xml version='1.0' encoding='UTF-8'?>
<wsdl:definitions xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-
instance" xmlns:plink="http://schemas.xmlsoap.org/ws/2003/05/partner-link/" xmlns:wsdlsoap11="http://schem
cnnblike commented 3 months ago

but it seems like it still can't build correctly if I submit xml file to the endpoint:

cnnblike@kl-desktop:~/workplace/elbe-base/elbe$ elbe initvm submit examples/x86_64-pc-rescue-busybox-dyn-cpio.xml
Starting the initvm Failed
Giving up
t-8ch commented 3 months ago

I'll try to reproduce this.

But if you have a git checkout anyways, you can use elbe directly from there.

./elbe initvm destroy
./elbe initvm create
./elbe initvm submit examples/x86_64-pc-rescue-busybox-dyn-cpio.xml
cnnblike commented 3 months ago

Hi, sorry for the loong troubleshooting thread, but it still don't work on my side. output.log What version of Debian would you recommend use?

t-8ch commented 3 months ago

Please do a git pull. The file examples/x86_64-pc-rescue-busybox-dyn-cpio.xml you are using was temporarily in another location. (See 149efadc6da6dbf3b26acc2979f24e173a7bbf2c)

The second error on initvm destroy is also fixed.

bookworm should be fine to use.

cnnblike commented 3 months ago

awesome! It's working on my machine now! Thanks a lot for your time!

t-8ch commented 3 months ago

You're welcome. I hope to release a fixed version soon.