search

armbian / build

Armbian Linux build framework generates custom Debian or Ubuntu image for x86, aarch64, riscv64 & armhf
https://www.armbian.com
GNU General Public License v2.0
3.99k stars 2.24k forks source link

[feature] Compile script: Add a BUILD_ONLY switch to the build procedure #4421

Closed mhoffrog closed 1 year ago

mhoffrog commented 1 year ago

Feature Add a BUILD_ONLY compile switch as a list of partial build artifacts to perform just a partial builds like `u-boot,kernel´

The motivation for this enhancement is to simplify and shorten U-Boot or other artifacts only builds.

:warning: Note: This description has been changed on 15-Nov-2022 as outcome of this issues discussion thread.

Solution Pull request #4455 has been created.

The-going commented 1 year ago

Stupid question: And if I want to build only the AUFS package, will I need the AUFS_ONLY key?

The benefit for all @armbian/build-scripts of us will be if we cancel this key KERNEL_ONLY and instead introduce a new key BUILD_ONLY="list packages"

hzyitc commented 1 year ago

I think we all forget this https://github.com/armbian/build/blob/c29e51c70828145785c3167d984e6857e342c96e/lib/functions/cli/cli-entrypoint.sh#L107-L111

We can use the following command to build U-boot only.

./compile.sh compile_uboot
The-going commented 1 year ago

We can use the following command to build U-boot only.

No we can't! Do you have tricky settings? If you can do it.

./compile.sh compile_uboot
....
[ o.k. ] * [l][c] sunxi_H616_GPU_enable_hack.patch 
[ o.k. ] * [l][c] xxx-disable-de2-to-improve-edid-detection.patch 
[ error ] ERROR in function compile_uboot [ functions/cli/cli-entrypoint.sh:110 -> functions/compilation/uboot.sh:82 -> functions/logging/traps.sh:0 ]
[ error ] ATF binary not found 
[ o.k. ] Process terminated
./compile.sh compile_atf compile_uboot
.....
Built /home...../build/sun50i_a64/debug/bl31.bin successfully

Only ATF is being assembled

The-going commented 1 year ago

@hzyitc I don't want to talk or discuss about what doesn't work in the build system. I want to say that we need a professional tool to work with only one or several packages. Collect only packages in the list excluding any other actions (git updates, hooks...), assemble in a clean target environment.

hzyitc commented 1 year ago

@The-going Yes. I think this is a good way to do this. Similar like make.

./compile.sh action1 action2

Just need a small modify to support multi actions.

The-going commented 1 year ago

No. it's bad. The entanglement in logic has reached a state where the probability of an error in the script is almost equal to one. A tool is required to work with only one package. Working with the package and not performing a single function.

hzyitc commented 1 year ago

So you mean a new script to build a package?

The-going commented 1 year ago

new script to build a package?

I want to implement a command that you just run with parameters. The team should only do what they were told. It will use the functions already available in the build system. And yes it can be a new executable script that is located in one of the /home/user/bin:/usr/bin directories.

The command must be packaged in a debian package and installed on the host system.

mhoffrog commented 1 year ago

I think we all forget this

https://github.com/armbian/build/blob/c29e51c70828145785c3167d984e6857e342c96e/lib/functions/cli/cli-entrypoint.sh#L107-L111

We can use the following command to build U-boot only.

./compile.sh compile_uboot

@hzyitc Yes, I am aware about this, but this does not include all the other tasks and validations the script framework does - like prepare_host() etc. - so I wanted to behave the compile procedure exactly as it does, if you use e.g. KERNEL_ONLY=yes up to the stage when U-Boot compile is finished. @The-going fully agree with your concerns - a more generic solution for selected package compiles would absolutely be preferred.

hzyitc commented 1 year ago

The command must be packaged in a debian package and installed on the host system.

I don't think this is a good idea since we're just a build system.

The do_default function is clean and can be easy to modify. So a new entry maybe enough.

The-going commented 1 year ago

@mhoffrog Marcus, you can hack the entire build system in a very simple way: 1) create a userpatches/lib.config file with the following contents

mybuild() {

        start=$(date +%s)

        display_alert "Downloading sources" "" "info"
        [[ -n $BOOTSOURCE ]] && fetch_from_repo "$BOOTSOURCE" "$BOOTDIR" "$BOOTBRANCH" "yes"
        [[ -n $ATFSOURCE ]] && fetch_from_repo "$ATFSOURCE" "$ATFDIR" "$ATFBRANCH" "yes"

        for option in $(tr ',' ' ' <<< "$CLEAN_LEVEL"); do
                        [[ $option != sources ]] && cleaning "$option"
        done

        # Don't build at all if the BOOTCONFIG is 'none'.
        [[ "${BOOTCONFIG}" != "none" ]] && {
                # Compile u-boot if packed .deb does not exist or use the one from repository
                if [[ ! -f "${DEB_STORAGE}"/${CHOSEN_UBOOT}_${REVISION}_${ARCH}.deb ]]; then
                        if [[ -n "${ATFSOURCE}" && "${REPOSITORY_INSTALL}" != *u-boot* ]]; then
                                compile_atf
                        fi
                        [[ "${REPOSITORY_INSTALL}" != *u-boot* ]] && compile_uboot
                fi
        }

        end=$(date +%s)
        runtime=$(((end - start) / 60))
        display_alert "Runtime" "$runtime min" "info"
}

This is taken from -> lib/functions/main/default-build.sh

leo@vm-jammy:~/armbian$ ./compile.sh test mybuild
[ warn ] This script requires root privileges, trying to use sudo 
[ o.k. ] Using config file [ /home/leo/armbian/userpatches/config-test.conf ]
[ .... ] Extension being added [ sunxi-tools :: added by functions/cli/cli-entrypoint.sh:105 -> functions/main/config-prepare.sh:135 -> functions/configuration/main-config.sh:174 -> config/sources/families/sun50iw1.conf:2 -> config/sources/families/include/sunxi64_common.inc:1 ]
[ o.k. ] Extension manager [ processed 3 Extension Methods calls and 3 Extension Method implementations ]
[ o.k. ] Using user configuration override [ /home/leo/armbian/userpatches/lib.config ]
[ o.k. ] Downloading sources 
[ o.k. ] Checking git sources [ u-boot v2022.07 ]
[ .... ]  Cleaning ....  [ 51 files ]
[ o.k. ] Checking git sources [ arm-trusted-firmware-sunxi-mainline af220ebbe467aa580e6b9ba554676f78ffec930f ]
[ .... ]  Cleaning ....  [ 2 files ]
[ o.k. ] Cleaning [ arm-trusted-firmware-sunxi-mainline/af220ebbe467aa580e6b9ba554676f78ffec930f ]
[ o.k. ] Compiling ATF 
[ o.k. ] Compiler version [ aarch64-none-linux-gnu-gcc 9.2.1 ]
[ o.k. ] Started patching process for [ atf sunxi64-bananapim64-edge ]
[ o.k. ] Looking for user patches in [ userpatches/atf/atf-sunxi64 ]
.......
  CAT     u-boot-dtb.bin
  MKIMAGE u-boot.img
  COPY    u-boot.dtb
  MKIMAGE u-boot-dtb.img
  COPY    u-boot.bin
  BINMAN  all
[ o.k. ] Building deb [ linux-u-boot-edge-bananapim64_22.11.0-trunk_arm64.deb ]
[ o.k. ] Runtime [ 0 min ]

I think you got the idea.

mhoffrog commented 1 year ago

@The-going many thanks for this hint! Yes I know that this can be patched via user patch, but I thought providing it in general would be useful and speed up the process for others as well. And if something is refactored on trunk I have to maintain my user patch and make sure it is doing all in a compliant way all the time.

The-going commented 1 year ago

And if something is refactored on trunk

That's right. We often want something more than the build system provides. This requires controlling her settings and scripts to get her to do some steps more correctly.

The-going commented 1 year ago

I don't think this is a good idea since we're just a build system.

The do_default function is clean and can be easy to modify. So a new entry maybe enough.

This is not enough. Just think about the fact that you wanted to build a library that depends on the ABI of this particular kernel. And then build an application that uses this library. And without much expense, install these packages into the system when the image is being made. Will you be able to do it at the moment? Example: Two close cores 5.19 and 6.0. Exported symbols for hardware decoding have changed. Your application requires recycling and reassembly. You did it. Good. And how to prescribe dependencies? The version of the kernel packages and the header library is 22.11.0-trunk in both cases. This is the tip of the iceberg. Continue further?

mhoffrog commented 1 year ago

@The-going If I get your last comment right, you would like to get this do_default function more structured or even more flexible in terms of custom/user hooking and configuration ?

If this is the case then this is a valid request. I'd then recommend to open a separate improvement request issue for that one.

My current purpose was just to get a quick and simple solution for now and for myself and others to only compile U-Boot without considering what existing methods I would have to call or what portion of do_default I would have to extract to a user extension config script.

That there is still space for improving the structure of the one or other existing script is no doubt I guess.

I leave it up to you guys @hzyitc , @The-going - whatever you prefer.

I really appreciate your great build framework and all the work you spend to it!

hzyitc commented 1 year ago

Just think about the fact that you wanted to build a library that depends on the ABI of this particular kernel. And then build an application that uses this library. And without much expense, install these packages into the system when the image is being made. Will you be able to do it at the moment?

This is too far from what we support. At lease we don't support to build any libraries.

Good. And how to prescribe dependencies? The version of the kernel packages and the header library is 22.11.0-trunk in both cases.

Yes, this is another problem. We didn't append the kernel version into the package version. But I remember thar someone is trying to solve it.

The-going commented 1 year ago

@hzyitc @mhoffrog Excuse me. Yesterday I wrote more than this relates to the topic of discussion.

I proceed from the following simple logic. Parents always buy trousers for a child of a slightly larger size and length. The child is growing fast.

I suggest entering a new key BUILD_ONLY="list name". Write a new function that will run the package build from the list. And add a selection here: https://github.com/armbian/build/blob/c29e51c70828145785c3167d984e6857e342c96e/lib/functions/cli/cli-entrypoint.sh#L107-L111 If the list is not empty, start a new function. To begin with, add only for the kernel and uboot name.

A little later I will add there the option of choosing packages to build in a chroot environment. By the way, there are libraries there. This strategy will not affect the logic of the build system as a whole and will allow you to gently switch to a new key later when the new function meets our requests.

The implementation in the pull request #4422 does not change anything fundamentally for me. I leave it to your consideration.

With respect.

mhoffrog commented 1 year ago

@The-going Your proposed looks perfect to me!

@hzyitc Please do not merge this pull request - I will replace it by a new one to revert my current commit(s) and will provide additional one(s) implementing the BUILD_ONLY switch as proposed.

By that I am supposed to refactor do_default a bit by breaking out the particular portions to dedicated methods. I guess I can provide it by EOB tomorrow.

The-going commented 1 year ago

I had in mind the following strategy: Create a file lib/functions/main/build_only.sh. Write one build_only function in it that accepts a list of arguments. And add a selection here: https://github.com/armbian/build/blob/c29e51c70828145785c3167d984e6857e342c96e/lib/functions/cli/cli-entrypoint.sh#L107-L111 If the list is not empty, start a new function. To begin with, add only for the kernel and uboot name.

if [ "$BUILD_ONLY" != "" ]; then
        build_only $BUILD_ONLY
elif [[ -z $1 ]]; then 
    do_default 
 else 
    eval "$@" 
 fi

There is no need to make changes with the KERNEL_ONLY key. No need to change the do_default function.

If the user wants to build only one package, he writes: ./compile.sh BUILD_ONLY="uboot" If there are two packages: ./compile.sh BUILD_ONLY="uboot kernel"

The-going commented 1 year ago

@mhoffrog Only when the new function is filled with correctly working code will we be able to afford to change the function of the do_default build system.

mhoffrog commented 1 year ago

@hzyitc

If the user wants to build only one package, he writes: ./compile.sh BUILD_ONLY="uboot" If there are two packages: ./compile.sh BUILD_ONLY="uboot kernel"

If you don´t mind, I would like to use the same syntax and artifact names as for

https://docs.armbian.com/Developer-Guide_Build-Options/ REPOSITORY_INSTALL (comma-separated list): list of core packages which will be installed from repository Available options: u-boot, kernel, ...

@The-going

Only when the new function is filled with correctly working code will we be able to afford to change the function of the do_default build system.

sure - I will run appropriate tests and assign you for code review.

Many thanks for all your comments so far!

hzyitc commented 1 year ago

BUILD_ONLY is a easy way to do this now. I think we can try it and drop KERNEL_ONLY.

@igorpecovnik How do you think about this.

BTW, GHA need to be changed.

igorpecovnik commented 1 year ago
mhoffrog commented 1 year ago

PR #4455 is created, implementation is done - with the following results:

@igorpecovnik

  • will BUILD_ONLY compile ATF within u-boot? If yes, then I am fine with

Yes, the implementation does exactly this - implemented in function build_uboot() in functions/main/build-tasks.sh

  • KERNEL_ONLY is used for build automation, so we need to keep it internally. Can be dropped once, but we need to spit out some warnings months before ...

Yes, realized this as well - KERNEL_ONLY option has been fully considered in the implementation and the build tasks performed are exactly the same as before, if BUILD_ONLY is not set. See Test 3 and 4 results on #4455.

@hzyitc @The-going I did try to leverage your code review by 4 partial commits to reflect that there was no functional change at all to the existing code of the previous default-build.sh - except for a few very small updates to improve the log output on getting sources for U-Boot and ATF as well as the output at the end of the build.

Please look to my particular full commit comments (there are more lines!) for the details.

4 relevant test scenarios have been performed (s. PR) and the build artifacts/steps for those were exactly as before.

In summary:

@hzyitc

BTW, GHA need to be changed.

From my point of view there is no reason why GHA is affected by the current implementation. The only change visible from outside is, that script default-build.sh is gone and script build-tasks.sh is there. But I guess this is nothing that should disturb GHA.

Many thanks in advance for your review efforts!