Closed mihaiolteanu closed 3 years ago
Hello Mihai,
this is actually a Debian packaging question, not an Isar one. One doesn't have to do anything special for Isar; one creates a general Debian source package, then feeds it to Isar. I think the right forum is something like debian-mentors.
If you don't have deb_debianize working, don't bother with it. As mentioned in #66, find a dh_make tutorial and follow it. Roughly,
After you have this working, integrate it into Isar (e.g., similar to https://github.com/ilbers/isar/blob/master/meta-isar/recipes-app/hello-isar/hello-isar.bb).
Regarding the empty package, I guess debhelper tries to be smart and automatically detect the files that need to be copied. You may have to help it. debian/rules is a makefile with some magic behind the scenes that calls some hook targets. I also don't know them by heart (in the older days, the targets were pasted completely into debian/rules, and it was easier to understand), please check debhelper docs. You may also want to check other Debian source packages, maybe apt-get source coreutils
. Please note that not all packages use debhelper, there are also other Debian packaging systems.
I see a lot of suggestions floating around for using deb_debianize
and also for generating the actual debian files with here-docs.
deb_debianize
tries to generate the debian files if they are not in the WORKDIR, otherwise it doesn't touch them.
The WORKDIR would include the here-docs, as given in the do_prepare_build
and/or the SRC_URI-given files, as I understand.
I've tried both methods, but none of them worked for me.
Is that not the right approach?
In this particular and simplified example, I already know how the debian files look like and I just want to put them in the right location for isar to pick them up. How do I manage to do that? Is isar not designed for this?
An updated readme page on isar on how to start developing your own packages without a debian folder would be really helpful. Currently, I can only see this, https://github.com/ilbers/isar/blob/master/doc/user_manual.md#compilation-of-sources-missing-the-debian-directory
deb_debianize is only good if it works out of the box. If it doesn't, it is more confusing. After you are serious about using Isar, don't use deb_debianize :) . I agree that Isar docs on debianizing are rather sparing -- because it's documented better elsewhere; I've already taken a note to improve that section. As described above, the integration into the Isar build is the second step. It's easier to separate conceptual steps. Are you able to build hithere_1.0_amd64.deb from your source package without Isar?
I get sent to deb_debianize
every time (even from the docs).
And deb_debianize
contains comments such as,
# create the compat-file if there is no file with that name in WORKDIR
# create the control-file if there is no control-file in WORKDIR
# create rules if WORKDIR does not contain a rules-file
That, in my mind, makes me understand that I can use and combine deb_debianize
with my own files. In short, to play with all sorts of combinations until it just clicks! Is that the wrong impression that I'm having? :)
Yes, I've followed the steps from the debian packaging intro and I was able to build the hithere deb file successfully, check its contents and confirm that I have the binary, man pages, changelog, etc, in there. I've also installed it with dpkg -i and called the binary successfully.
Well, playing with the combinations didn't work till now, thus I suggest to learn by breaking down.
If you can build your binary package with dpkg-buildpackage, you don't need deb_debianize for the rest of your life. As the second step, just integrate your source package as shown in hello-isar.bb. Does this work?
As mentioned in #66, deb_debianize is intended as a transition help for Yocto developers who don't yet want to invest time into learning the Debian way of building packages. From the Debian perspective, building from recipes is wrong and bad (remember abstraction leak from #66?).
As the third step, if you still want to play with deb_debianize, you have to analyze the debian/* files created by it in tmp/work and adjust your recipes to bring them to the form that works with dpkg-buildpackage which you already have from the first step. Please don't use this in production, as we are migrating to sbuild and might remove building from recipes in the future.
Ok, I have the hithere-1.0.tar.gz
archive downloaded to the recipe's files folder. This archive does not have any debian files.
In this particular case, I do have the debian files, separately from the archive, since they are already given in the debian packaging intro.
This example would simulate a package where the upstream developer does not care about debianization and it is our job, using isar, to handle that. We have at least two packages right now where this is the case and we don't know how to move forward with it.
The hithere example uses debuild -us -uc
and not dpkg-buildpackage
.
From my understanding debuild contains the the dpkg-buildpackage step also (see my initial message above).
Back to our recipe. Here is the full hithere.bb that I'm trying right now,
DESCRIPTION = "hithere example from the official debian packaging tutorial"
MAINTAINER = "Mihai Olteanu <mihai_olteanu@fastmail.fm>"
PV = "1.0"
SRC_URI = "file://hithere-1.0.tar.gz"
SRC_URI[md5sum] = "5b2830fa1fcd44ce489774771625526e"
inherit dpkg
And the build output,
ERROR: hithere-1.0-r0 do_install_builddeps: Execution of '/work/build/tmp/work/debian-buster-amd64/hithere/1.0-r0/temp/run.do_install_builddeps.128' failed with exit code 1:
grep: debian/control: No such file or directory
W: Unable to locate package debian/control
mk-build-deps: Unable to find package name in `apt-cache showsrc debian/control'
WARNING: exit code 1 from a shell command.
ERROR: Logfile of failure stored in: /work/build/tmp/work/debian-buster-amd64/hithere/1.0-r0/temp/log.do_install_builddeps.128
Log data follows:
| DEBUG: Executing shell function do_install_builddeps
| grep: debian/control: No such file or directory
| W: Unable to locate package debian/control
| mk-build-deps: Unable to find package name in `apt-cache showsrc debian/control'
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/work/build/tmp/work/debian-buster-amd64/hithere/1.0-r0/temp/run.do_install_builddeps.128' failed with exit code 1:
| grep: debian/control: No such file or directory
| W: Unable to locate package debian/control
| mk-build-deps: Unable to find package name in `apt-cache showsrc debian/control'
| WARNING: exit code 1 from a shell command.
|
ERROR: Task (/work/meta-isar-ces/recipes-custom/hithere/hithere.bb:do_install_builddeps) failed with exit code '1'
NOTE: Tasks Summary: Attempted 72 tasks of which 64 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/work/meta-isar-ces/recipes-custom/hithere/hithere.bb:do_install_builddeps
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
2021-06-18 10:25:09 - ERROR - Command returned non-zero exit status 1
So I'm back again to missing debian files, and my natural instinct and understanding is that I should add them either to the WORKDIR or with deb_deb.... Ok, we're experts, I won't mention that function again! :)
So what are the next steps, then?
Ok, now I get your use case.
In such situations I usually fork the package to my internal git server, create a branch like master-debian, commit debian/* there, integrate as shown in hello-isar.bb, and then update master-debian from time to time. Does this work?
If you don't like this (why?), there are other Debian ways to do that.
[I haven't tested these:]
E.g., prepare a source package (dsc, patch, tar.gz), put it to an internal http server, use apt:// building, then update the source package from time to time.
Of course, there still are more Yocto-ish ways of doing that, like adding the Debian patch to the recipe (like in cowsay_git.bb) but using inherit dpkg, or forcing deb_debianize to generate what you want / overwriting its stuff with yours (like samefile_2.14.bb). Conceptually, all are the same -- get the right contents into debian/* in a suitable way, then use inherit dpkg.
Ok, I will try to fork the project (or just create a new github test project for the hithere
package, in this case) add the debian folder and create the isar recipe similar to hello-isar.bb
I now see that the hello repo has the debian folder (so no fooling around with isar creating those for you). To be fair, this is what this comment from the isar readme suggested as well,
Isar expects your sources to contain the debian folder and the above steps need to be described in it,
not in a task in a recipe.
You don't need to fork a package for that (except maybe during development, locally). You can also maintain complex debianization as local artifacts ("file://debian"
) in the recipe. Basically what Isar does for common components like the kernel or various bootloader/firmware components.
That's also a way. I just find forking the most comfortable to work with. It probably depends on the frequency of updates and how intensively you are going to work with this package later.
Sure, there are pros and cons. Forking permits usage independent of Isar, recipe artifacts in the same repo makes it a bit more self-contained.
I've tried the forking approach, as suggested by @ismagulb.
Here is what I did and the results, for future reference:
hithere.bb
isar recipe,
DESCRIPTION = "hithere example from the official debian packaging tutorial"
PV = "1.0"
SRC_URI = "git://github.com/mihaiolteanu/hithere;protocol=https;branch=main;destsuffix=${P}" SRCREV = "78500ecc9e658c33e500ac18a929c27d199fddaf"
inherit dpkg
- The build is successful, and the resulting .deb file contains the binary, man page, change log, etc. (unlike my initial attempt where it contained only the changelog),
$ dpkg -c ./build/tmp/work/debian-buster-amd64/hithere/1.0-r0/hithere_1.0-1_amd64.deb drwxr-xr-x root/root 0 2010-11-18 19:25 ./ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/bin/ -rwxr-xr-x root/root 14440 2010-11-18 19:25 ./usr/bin/hithere drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/doc/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/doc/hithere/ -rw-r--r-- root/root 154 2010-11-18 19:25 ./usr/share/doc/hithere/changelog.Debian.gz drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/man/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/man/man1/ -rw-r--r-- root/root 265 2010-11-18 19:25 ./usr/share/man/man1/hithere.1.gz
- The `dpkg -i` part (the package installation) is also successful, as confirmed by inspecting the resulting file system,
.../rootfs/usr/bin/hithere .../rootfs/usr/share/man/man1/hithere.1.gz .../rootfs/usr/share/doc/hithere .../rootfs/var/lib/dpkg/info/hithere.list .../rootfs/var/lib/dpkg/info/hithere.md5sums
Ok, now you have done the simplest form and can proceed to more complex structure if you want.
I've tried the file://debian
approach as suggested by @jan-kiszka.
Here is what I did and the results, for future reference:
no-debian
branch specified in the recipe does not contain any debian files),
DESCRIPTION = "hithere example from the official debian packaging tutorial"
PV = "1.0"
SRC_URI = "\ git://github.com/mihaiolteanu/hithere;protocol=https;branch=no-debian;destsuffix=${P} \ file://debian/" SRCREV = "bfdc34251d8e267e799c50b0a7d6cc58f1982d66"
inherit dpkg
do_prepare_build() { cp -r ${WORKDIR}/debian ${S}/ }
- Inside the recipe location, I've created the files/debian folder with all the contents (or files) as suggested in the [debian packaging intro](https://wiki.debian.org/Packaging/Intro),
$ ls -lahR hithere .: total 16K drwxr-xr-x 3 mihai mihai 4.0K Jun 18 15:32 . drwxr-xr-x 6 mihai mihai 4.0K Jun 17 10:43 .. drwxr-xr-x 3 mihai mihai 4.0K Jun 18 15:27 files -rw-r--r-- 1 mihai mihai 603 Jun 18 15:32 hithere.bb
./files: total 16K drwxr-xr-x 3 mihai mihai 4.0K Jun 18 15:27 . drwxr-xr-x 3 mihai mihai 4.0K Jun 18 15:32 .. drwxr-xr-x 2 mihai mihai 4.0K Jun 18 15:27 debian
./files/debian: total 32K drwxr-xr-x 2 mihai mihai 4.0K Jun 18 15:27 . drwxr-xr-x 3 mihai mihai 4.0K Jun 18 15:27 .. -rw-r--r-- 1 mihai mihai 146 Jun 17 16:08 changelog -rw-r--r-- 1 mihai mihai 3 Jun 17 16:09 compat -rw-r--r-- 1 mihai mihai 289 Jun 17 16:09 control -rw-r--r-- 1 mihai mihai 11 Jun 17 16:10 format -rw-r--r-- 1 mihai mihai 26 Jun 18 15:27 hithere.dirs -rw-r--r-- 1 mihai mihai 118 Jun 17 16:28 rules
- The build is successful, and the resulting .deb file contains the binary, man page, change log, etc,
$ dpkg -c ./build/tmp/work/debian-buster-amd64/hithere/1.0-r0/hithere_1.0-1_amd64.deb drwxr-xr-x root/root 0 2010-11-18 19:25 ./ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/bin/ -rwxr-xr-x root/root 14440 2010-11-18 19:25 ./usr/bin/hithere drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/doc/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/doc/hithere/ -rw-r--r-- root/root 154 2010-11-18 19:25 ./usr/share/doc/hithere/changelog.Debian.gz drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/man/ drwxr-xr-x root/root 0 2010-11-18 19:25 ./usr/share/man/man1/ -rw-r--r-- root/root 265 2010-11-18 19:25 ./usr/share/man/man1/hithere.1.gz
- The dpkg -i part (the package installation) is also successful, as confirmed by inspecting the resulting file system,
.../rootfs/usr/bin/hithere .../rootfs/usr/share/man/man1/hithere.1.gz .../rootfs/usr/share/doc/hithere .../rootfs/var/lib/dpkg/info/hithere.list .../rootfs/var/lib/dpkg/info/hithere.md5sums
@ismagulb yes, from here there is only debian stuff to learn and experiment with. Thank you for your help!
@jan-kiszka thank you for your suggestion!
Scope and Overview
I'm trying to understand how would one go about creating an isar recipe, step by step for a package that doesn't have the debian folder. Naturally, this would mean that we have to create our own debian files.
The official isar documentation, compilation of sources missing the debian directory, suggests that this is the case indeed and points one to the official debian documentation and to
deb_debianize
. How any of that works, is left as a homework for the user.Packaging Intro (hithere example)
The first step that I took was to understand how debian packaging works in general, outside isar. So I followed the official debian packaging intro. I've installed a debian docker container so as not to mess things up and did all the steps in the intro:
Replicate the hithere example with isar
Once I understood the basic steps needed to build a basic package, I've tried to replicate that with an isar recipe. The advantage of this approach, in my mind, is that I already know what the debian files should contain, so I am left with only one problem and not two. That is, replicating the build with isar and not at the same time trying to figure out what the debian files should contain.
Minimal recipe using only deb_debianize
I've started with a minimalist recipe inspired by
samefile_2.14.bb
, and namedhithere.bb
,This leads to a clean build, but the generated .deb package only contains the changelog and not the manual pages and, most importantly, the binary,
Some points that I'm figuring out as I go (some of which might be wrong):
cleandirs
is a way of telling isar to start with a clean debian folder.deb_debianize
generates the minimum required files for packaging (rules, control, compat, etc)deb_debianize
, we can create our own personalized debian files inside WORKDIR (with here-docs, for example) anddeb_debianize
will take those instead of creating its default ones.SRC_URI
(which will get copied in theWORKDIR,
according to the bitbake rules), anddeb_debianize
will install those; otherwise, it will install its default filesReplicate all the debian files from the intro using here-docs
Ok, since that didn't work, I've next tried to add all the debian files as seen in the official packages intro inside the
do_prepare_build
as here-docs and see what comes out of it.Some points:
Run the build,
The hithere official example runs the build with
debuild -us -uc
. It is possible that isar runs a different build command, thus the above error, see hereI think isar builds deb packages with
dpkg-buildpackage,
which is just one step in thedebuild
tool (that is,debuild
contains thedepk-buildpackage
call as a step, besides lintian, fakeroot, etc). See the isarbuild.sh
script file for details,${GBP_PREFIX}dpkg-buildpackage -a$target_arch -d --source-option=-I
But these are just assumptions on my side, which, even if they were right, they don't help me much in figuring out what the solution is to actually building this recipe.Manually remove every reference to
hithere
, except the recipe, and retry the build (sometimes I do get errors due to old, temp or cache files not being deleted).This time I get,
Remove this line, for now,
No error this time, but the .deb package still does not contain any binaries. For comparison, this is what the deb package contains on the docker container after a successful packaging,
This is what we want after isar completes the build, as well.
The rules and source/format files are there, for example,
But I think they are in the wrong place, as hithere-1.0 also contains debian files,
But I also get this warning,
Comparing the debian file contents reveals that they are the same, or contain the same info. But there are actually three different debian folders where debian files are present in the same
hithere
location. Most likely this is fine, and it's the way isar builds its stuff internally. But it's a bit tricky to understand how should one write a recipe by checking the implementation.Simplify and only keep one here-doc
Simplify the initial recipe, only add a compat file, for example, and do not call deb_debianize for the moment,
I do not expect this recipe to work, but only to understand how isar works. Clean every mention of hithere (except the recipe itself) and rebuild,
Perfect! Debian files are missing, so we get an error.
compat contains the 12345 string, as expected, hithere-1.0 contains an empty debian folder (presumably the compat was not copied here due to the error) and an error file,
Full recipe with here-docs but without deb_debianize
Try again the full recipe, but without the deb_debianize at the end of the
do_prepare_build
task.No debian/control file. Why? There is a debian/control file given in the task, as a here-doc. deb_debianize should only add one if it doesn't find one in the WORKDIR.
Full recipe but with override_dh_auto_install
Add back the
override_dh_auto_install,
But then I get other errors
Specify all the debian files in the files folder of the hithere recipe,
For a different approach, I've remove the here-docs and added all the debian files in the file folder of the hithere recipe,
All the debian files contain what the packaging intro contain and what I also used in the docker image to successfully build the debian package.
Rebuild,
That does not work either. debian/control is still missing.
Conclusion
I've tried all the combinations that I could think of to replicate the build steps from the packaging intro. At each step I get different errors. The big question is what is the correct approach to debianizing packages in isar? I am open and willing to write a detailed isar wiki article on how to debianize the hithere package once I'm clear with the process and the pitfalls. I'm choosing the hithere package since this is the official "packaging for beginners" suggestion for debian users who have no clue what packaging is. It might also be a good starting point for "isar for beginners" in the future. (feel free to adjust the formatting or grammar for the above issue)