search

ilbers / isar

Integration System for Automated Root filesystem generation
Other
177 stars 72 forks source link

Calling a shell script with build-time dependencies from a .bb recipe #66

Closed mihaiolteanu closed 3 years ago

mihaiolteanu commented 3 years ago

What is the proper way to call a shell script that has build-time dependencies from a .bb recipe file?

I've tried calling the script in the do_install task with no luck. I believe the reason is that do_install is called before the dependencies are installed. As a result, I've added a new task that gets called after the do_install_builddeps` task,

inherit dpkg-raw
DEBIAN_DEPENDS += "build-essential, dkms, git, libelf-dev "
SRCREV = "0c6a77c94d29a3907b39eb8ab15958bb0b646ab6"
SRC_URI = "git://github.com/morrownr/8821cu;nobranch=1;protocol=https"

do_install_driver(){
    cd ${WORKDIR}/git
    sudo ./install-driver.sh
}

addtask do_install_driver after do_install_builddeps before do_dpkg_build

But I'm getting the same error as I did with do_install, namely, that one of the dependencies, the dkms is not found,

install-driver.sh version 20210401
| Copying source files to: /usr/src/rtl8821cu-5.8.1.7
| Copying 8821cu.conf to: /etc/modprobe.d
| All required files have been copied to the proper places.
| /work/build/tmp/work/debian-buster-amd64/rtl8821cu/1.0-r0/git/install-driver.sh: line 38: dkms: command not found
| An error occurred while running: dkms add : 127
| Please report errors.
| WARNING: exit code 127 from a shell command.

I've also used DEBIAN_BUILD_DEPENDS instead of DEBIAN_DEPENDS with the same result.

I can install the driver on a running debian system following the installation steps. This system is also built with isar with the only difference that it does not contain the above recipe. And now I'm trying to bake the same steps into a .bb recipe.

jan-kiszka commented 3 years ago

You likely rather want to use https://github.com/ilbers/isar/blob/master/meta/recipes-kernel/linux-module/module.inc on this, see https://github.com/siemens/jailhouse-images/tree/master/recipes-kernel/wilc and https://github.com/siemens/jailhouse-images/tree/master/recipes-kernel/xradio for similarly ugly pattern (horrible out-of-tree drivers...).

If this particular driver is even worse: DEBIAN_BUILD_DEPENDS, see manual. Or check the dkms patches floating on the mailing list.

mihaiolteanu commented 3 years ago

Now I'm even more confused.

First, I've tried the wilc recipe that you've linked above, just as a test, adding the KERNEL_NAME="4.19.0-16-amd64" on top of it and PROVIDES = "wilc_15.2-avnet" at the end (isar errors with nothing provides wilc_15.2-avnet, otherwise), but I get errors,

| Removing wilc-4.19.0-16-amd64-build-deps (15.2-avnet) ...
| mk-build-deps: Unable to install wilc-4.19.0-16-amd64-build-deps at /usr/bin/mk-build-deps line 416.
| mk-build-deps: Unable to install all build-dep packages
| WARNING: exit code 1 from a shell command.

Secondly, I've tried using module.inc in a simplified version of the driver recipe,

KERNEL_NAME="4.19.0-16-amd64"
require recipes-kernel/linux-module/module.inc

SRCREV = "0c6a77c94d29a3907b39eb8ab15958bb0b646ab6"
SRC_URI = "git://github.com/morrownr/8821cu;nobranch=1;protocol=https"

PROVIDES = "rtl8821cu"

But isar errors with,

ERROR: rtl8821cu-4.19.0-16-amd64-1.0-r0 do_transform_template: 
Error executing a python function in exec_python_func() autogenerated:
[...]
Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/work/build/tmp/work/debian-buster-amd64/rtl8821cu-4.19.0-16-amd64/1.0-r0/debian/control.tmpl'
mihaiolteanu commented 3 years ago

I can only find this entry from the changelog regarding DEBIAN_BUILD_DEPENDS,

Additional build dependencies of auto-debianized packages can now be defined by setting DEBIAN_BUILD_DEPENDS.

What does that mean?

jan-kiszka commented 3 years ago

It's hard to discuss build issues when only looking through keyholes (no full recipes, no full logs). In addition, the wilc driver is a beast, no longer building against all kernel versions. Take the recipe more as inspiration than as an actual thing to try outside of jailhouse-images. That xradio thing might be nicer.

DEBIAN_BUILD_DEPENDS injects packages into the Build-Depends: rule of the generated debian/control file. That's all. You are right, this is not described in the user manual. Feel free to send a patch that enhances it accordingly.

mihaiolteanu commented 3 years ago

the wilc driver is a beast

Noted! :))

I've only stripped the python function from that log to make it more readable. Here it is in all its glory (though not more useful, I believe),

ERROR: rtl8821cu-4.19.0-16-amd64-1.0-r0 do_transform_template: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_transform_template(d)
     0003:
File: '/work/isar/meta/classes/template.bbclass', lineno: 53, function: do_transform_template
     0049:                       else (template_file + ".out"))
     0050:        bb.note("{} {} [in: {} out: {}]".format(cmd, args,
     0051:                                                template_file, output_file))
     0052:        with contextlib.ExitStack() as stack:
 *** 0053:            input = stack.enter_context(open(template_file, 'rb'))
     0054:            output = stack.enter_context(open(output_file, 'wb'))
     0055:            process = stack.enter_context(subprocess.Popen([cmd, args], stdin=input,
     0056:                                          stdout=output, env=env))
     0057:            if process.wait() != 0:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/work/build/tmp/work/debian-buster-amd64/rtl8821cu-4.19.0-16-amd64/1.0-r0/debian/control.tmpl'

ERROR: Logfile of failure stored in: /work/build/tmp/work/debian-buster-amd64/rtl8821cu-4.19.0-16-amd64/1.0-r0/temp/log.do_transform_template.111
ERROR: Task (/work/meta-isar-aaa/recipes-support/8821cu/rtl8821cu.bb:do_transform_template) failed with exit code '1'
NOTE: Tasks Summary: Attempted 185 tasks of which 184 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /work/meta-isar-aaa/recipes-support/8821cu/rtl8821cu.bb:do_transform_template
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
2021-05-06 07:27:12 - ERROR    - Command returned non-zero exit status 1

Thanks for your time, Jan.

amikan commented 3 years ago

Yes, do_install step is performed before do_install_builddeps for dpkg-raw bbclass. So you're probably want install-driver.sh to be executed during do_dpkg_build task. This can be done by inherit dpkg (not dpkg-raw) with the custom debian/rules file, for example:

do_prepare_build[cleandirs] += "${S}/debian"
do_prepare_build() {
        deb_debianize

        cat <<EOF >> ${S}/debian/rules

override_dh_install:
        ../install-driver.sh
EOF
}
mihaiolteanu commented 3 years ago

I'm beginning to get the impression that to be able to use isar well you also have to know the internals of the debian build process. Am I wrong in this assumption? What one would call a leaky abstraction, maybe?

later-edit I've tried your example. I've moved deb_debianize at the end of the task to get rid of the errors,

do_prepare_build() {
    cat << EOF >> ${S}/debian/rules

override_dh_install:
    ../install-driver.sh
EOF

    deb_debianize
}

Though that doesn't install the stuff that install-driver.sh is supposed to. I think I get the idea, though. I will play with it further.

@amikan thank you for your time. end later-edit

Yes, do_install step is performed before do_install_builddeps for dpkg-raw bbclass. This can be done by inherit dpkg (not dpkg-raw) with the custom debian/rules file

Where does one get this kind of info regarding isar, apart from actually reading the isar source code? I believe that these things, like the DEBIAN_BUILD_DEPENDS for example, are also not specified in the user manual.

ismagulb commented 3 years ago

I'm beginning to get the impression that to be able to use isar well you also have to know the internals of the debian build process. Am I wrong in this assumption? What one would call a leaky abstraction, maybe?

Well, this is how bitbake is designed. E.g., you want to build a Debian package -- you do that in the recipe, with all gory details. deb_debianize is a band-aid help for quick transition from Yocto to Isar.

The clean way is: Don't use deb_debianize. Debianize your package in its git repo, with package/debian/control, rules, and everything that belongs to it (man dh_make). Then, inherit dpkg, and you have the abstraction. Example: meta-isar/recipes-app/hello-isar/hello-isar.bb. You may use the files generated by deb_debianize as a template (somewhere similar to build/tmp/work/debian-buster-armhf/yourpackage/0.1-r0).