Build the minimum amount of atoms a package depends on

msdobrescu commented 2 years ago

Disclaimer: I am just beginning this kind of task, so, although a solution may be possible, I may not be able to find it yet.

Can't cleanly build media-sound/clementine. By "cleanly" I mean building only the atoms not yet present in some existing layer under mOS. I use the community repository of mOS.

For a start, I've made myself a diagram of layers dependency.

@startuml

scale 0.8

'skinparam linetype ortho
'skinparam linetype polyline

'skinparam nodesep 10
skinparam ranksep 100

skinparam artifact {
    backgroundColor #FFD700
}

skinparam package<<repo>> {
    backgroundColor #00FFFF
}

skinparam package<<collection>> {
    backgroundColor #DAA520

}

skinparam package<<layer>> {
    /'
    borderColor Transparent
    backgroundColor Transparent
    fontColor Transparent
    stereotypeFontColor Transparent
    '/

    backgroundColor #FFFFE0
}

skinparam package<<meta>> {
    backgroundColor #EE82EE
}

allow_mixing

package "os-commons" <<repo>> {
    collections "entities"
}

package "desktop" <<repo>> {
    package "apps" <<collection>> {
        artifact "vlc"
        artifact "pavucontrol-qt"
    }

    package "meta" <<collection>> {
        package "systemd-groups" <<meta>> {
        }    
    }    

    package "cinnamon" <<layer>> {
    }

    package "codecs" <<layer>> {
    }

    package "enlightenment" <<layer>> {
    }

    package "firmware" <<layer>> {
    }

    package "gnome" <<layer>> {
    }

    package "gnome-common" <<layer>> {
    }

    package "java" <<layer>> {
    }

    package "kde-apps-minimal" <<layer>> {
    }

    package "kde-common" <<layer>> {
    }

    package "kde-pim" <<layer>> {
    }

    package "lxqt" <<layer>> {
    }

    package "mate" <<layer>> {
    }

    package "net-tools" <<layer>> {
    }

    package "plasma" <<layer>> {
    }

    package "qt" <<layer>> {
    }

    package "sys-fs" <<layer>> {
    }

    package "system-x" <<collection>> as sxc {
        artifact "gcc-base"

        package "system-x" <<layer>> {
        }
    }

    package "wayfire" <<layer>> {
    }

    package "X" <<layer>> {
    }

    package "xfce" <<layer>> {
    }
}

"gnome-common" <-- "cinnamon"

"system-x" <-- "codecs"
"entities" <-- "codecs"

"X" <-- "enlightenment"

"X" <-- "gnome-common"
"entities" <-- "gnome-common"

"gnome-common" <-- "gnome"
"X" <-- "gnome"
"entities" <-- "gnome"

"X" <-- "java"

"plasma" <-- "kde-apps-minimal"
"qt" <-- "kde-common"
"entities" <-- "kde-common"

"plasma" <-- "kde-pim"
"entities" <-- "kde-pim"

"kde-common" <-- "lxqt"
"pavucontrol-qt" <-- "lxqt"

"X" <-- "mate"
"gnome-common" <-- "mate"

"system-x" <-- "net-tools"
"entities" <-- "net-tools"

"kde-common" <-- "plasma"
"vlc" <-- "plasma"
"entities" <-- "plasma"

"X" <-- "qt"

"system-x" <-- "sys-fs"

"entities" <-- "systemd-groups"

"gcc-base" <-- "system-x"
"systemd-groups" <-- "system-x"
"entities" <-- "system-x"

"system-x" <-- "X"
"sys-fs" <-- "X"
"codecs" <-- "X"
"net-tools" <-- "X"

"gnome-common" <-- "wayfire"
"gnome-common" <-- "xfce"

@enduml

This brings the question:

Why gnome layer requires gnome-common and X layers while gnome-common requires X already? In other words why requiring gnome-common is not enough (by bringing X in its requirements)? Or is it?

So, analyzing media-sound/clementine's deps, I have found that building it is depending on:

plasma layer primarly
gnome-commons layer due to mtp implemented by gnome-base/gvfs

And here we go...

Describe alternatives you've considered Incrementally, I've added the needed deps one by one to the community repository collection:

- &libs
  category: "libs"
  name: "chromaprint"
  version: "1.5.0"
  labels:
    emerge.jobs: "1"
    emerge.packages: >-
      media-libs/chromaprint
  atom: "media-libs/chromaprint"
  provides:
  - name: "chromaprint"
    category: "media-libs"
    version: ">=0"
  pre_emerge:
  - dev-cpp/gtest
  requires:
  - &layers-X
    category: "layers"
    name: "X"
    version: ">=0"
  build_requires:
  - <<: *layers-X

- <<: *libs
  name: "libmygpo-qt"
  version: "1.1.0"
  labels:
    emerge.jobs: "1"
    emerge.packages: >-
      media-libs/libmygpo-qt
  atom: "media-libs/libmygpo-qt"
  provides:
  - name: "libmygpo-qt"
    category: "media-libs"
    version: ">=0"
  pre_emerge:
  - dev-cpp/gtest
  requires:
  - &layers-qt
    category: "layers"
    name: "qt"
    version: ">=0"
  build_requires:
  - <<: *layers-qt

The trouble begins with gnome-base/gvfs.

In order to satisfy it, to avoid emerge stopping due to errors like these below, must refer gnome-commons layer.

Source: (here).

      Step 9/9 : RUN emerge -j ${JOBS} media-sound/clementine
 INFO    ---> Running in 212eab660414
 INFO   
        * IMPORTANT: 11 news items need reading for repository 'gentoo'.
        * Use eselect news read to view new items.
 INFO   [ebuild  N    ] gnome-base/gvfs-1.48.1-r1 
       [ebuild  N    ] media-sound/clementine-1.4.0_rc1 
 INFO   
       The following USE changes are necessary to proceed:
        (see "package.use" in the portage(5) man page for more details)
       # required by media-sound/clementine-1.4.0_rc1::gentoo[mtp]
       # required by media-sound/clementine (argument)
       >=gnome-base/gvfs-1.48.1-r1 mtp

       Use --autounmask-write to write changes to config files (honoring
       CONFIG_PROTECT). Carefully examine the list of proposed changes,
       paying special attention to mask or keyword changes that may expose
       experimental or unstable packages.
        INFO   The command '/bin/sh -c emerge -j ${JOBS} media-sound/clementine' returned a non-zero code: 1
  ERROR    Error: failed building package image: Could not push image: quay.io/mocaccinocache/desktop:f92d013b7212732aa056a8b3cbd7317bfe3beaa2b90614fafd1e2a9d7b8453ea clementine-apps-1.4.0.dockerfile: Could not build image: quay.io/mocaccinocache/desktop:f92d013b7212732aa056a8b3cbd7317bfe3beaa2b90614fafd1e2a9d7b8453ea clementine-apps-1.4.0.dockerfile: Failed running command: : exit status 1
  ERROR    Bailing out
exit status 1
Error: Process completed with exit code 1.

Clementine config has become this:

- <<: *apps
  name: "clementine"
  version: "1.4.0"
  labels:
    emerge.jobs: "1"
    emerge.packages: >-
      media-sound/clementine
  atom: "media-sound/clementine"
  provides:
  - name: "clementine"
    category: "media-sound"
    version: ">=0"
  pre_emerge:
  - dev-cpp/gtest
  - media-libs/chromaprint
  - media-libs/libmygpo-qt
  requires:
  - <<: *layers-plasma
  - <<: *layers-gnome-common
  - &libs-chromaprint
    category: "libs"
    name: "chromaprint"
    version: ">=0"
  - &libs-libmygpo-qt
    category: "libs"
    name: "libmygpo-qt"
    version: ">=0"
  build_requires:
  - <<: *layers-plasma
  - <<: *layers-gnome-common

The outcome is starting emerge qt layer, as seen here, but that is expected to be provided by the plasma layer inherently, isn't it?

  INFO    Building image quay.io/mocaccino/mocaccino-community:40dabfe5bb779e8925176e2c5cfab2060d862cdb9b2b0f338d57f95eef0d779e
 INFO   Sending build context to Docker daemon  14.85kB
 INFO   
 INFO   Step 1/9 : FROM quay.io/mocaccino/mocaccino-community:builder-ae8ba98acdf11e8208c134ec199901cd
 INFO    ---> 47494cd2783a
       Step 2/9 : COPY . /luetbuild
 INFO    ---> 2757dc43e60d
       Step 3/9 : WORKDIR /luetbuild
 INFO    ---> Running in f772eee9e5ce
 INFO   Removing intermediate container f772eee9e5ce
        ---> c99af5f46951
       Step 4/9 : ENV PACKAGE_NAME=qt
 INFO    ---> Running in 0a03324e9384
 INFO   Removing intermediate container 0a03324e9384
        ---> cbeccd427b2c
       Step 5/9 : ENV PACKAGE_VERSION=5.15.2+85
 INFO    ---> Running in d1e2a9d35b50
 INFO   Removing intermediate container d1e2a9d35b50
        ---> 14475feb83a6
       Step 6/9 : ENV PACKAGE_CATEGORY=layers
 INFO    ---> Running in 9b1b92aa8db9
 INFO   Removing intermediate container 9b1b92aa8db9
        ---> 0ccb3b79a2fb
       Step 7/9 : ENV JOBS=1
 INFO    ---> Running in 3c989ca53739
 INFO   Removing intermediate container 3c989ca53739
        ---> 993ab3f4b980
       Step 8/9 : RUN emerge -j ${JOBS} dev-qt/qdbus dev-qt/qtbluetooth dev-qt/qtcharts dev-qt/qtchooser dev-qt/qtconcurrent dev-qt/qtcore dev-qt/qtdbus dev-qt/qtdeclarative dev-qt/qtgraphicaleffects dev-qt/qtgui dev-qt/qthelp dev-qt/qtimageformats dev-qt/qtlockedfile dev-qt/qtmultimedia dev-qt/qtnetwork dev-qt/qtnetworkauth dev-qt/qtopengl dev-qt/qtpaths dev-qt/qtpositioning dev-qt/qtprintsupport dev-qt/qtquickcontrols dev-qt/qtquickcontrols2 dev-qt/qtscript dev-qt/qtscxml dev-qt/qtsensors dev-qt/qtserialport dev-qt/qtsingleapplication dev-qt/qtspeech dev-qt/qtsql dev-qt/qtsvg dev-qt/qttest dev-qt/qttranslations dev-qt/qtvirtualkeyboard dev-qt/qtwayland dev-qt/qtwebchannel dev-qt/qtwebsockets dev-qt/qtwidgets dev-qt/qtx11extras dev-qt/qtxml dev-qt/qtxmlpatterns dev-qt/qtwebengine
 INFO    ---> Running in 0c665a9cbcc0
 INFO   
        * IMPORTANT: 11 news items need reading for repository 'gentoo'.
        * Use eselect news read to view new items.
 INFO   >>> Verifying ebuild manifests
 INFO   >>> Running pre-merge checks for net-libs/nodejs-14.17.6
 INFO   >>> Running pre-merge checks for dev-qt/qtcore-5.15.2-r10
 INFO   >>> Emerging (1 of 48) dev-qt/qtchooser-66-r1::gentoo
 INFO   >>> Installing (1 of 48) dev-qt/qtchooser-66-r1::gentoo
 INFO   >>> Recording dev-qt/qtchooser in "world" favorites file...
 INFO   >>> Emerging (2 of 48) dev-libs/re2-0.2021.06.01::gentoo
 INFO   >>> Installing (2 of 48) dev-libs/re2-0.2021.06.01::gentoo
 INFO   >>> Emerging (3 of 48) app-arch/snappy-1.1.9::gentoo
 INFO   >>> Installing (3 of 48) app-arch/snappy-1.1.9::gentoo
 INFO   >>> Emerging (4 of 48) dev-lang/python-2.7.18_p13::gentoo
...

Describe the solution you'd like I would expect to pull all the necessary layers, specifying the top ones only, probably build the container in order to build only the remaining atoms.

In the case above, those atoms would be:

dev-cpp/gtest
media-libs/chromaprint
media-libs/libmygpo-qt
media-sound/clementine

Additional context I am aware sometimes the layers structure is not ideal, things can be moved around, but it's not always possible (not saying here it's the case for media-sound/clementine).

mudler commented 2 years ago

That's more an issue on how a repository was designed, than rather an issue with luet itself. Or I am missing something?

msdobrescu commented 2 years ago

Actually, I thought this myself. But, seems some KDE apps need Gnome libs sometimes, due to missing those implementation in a KDE/Qt lib. This would need, for instance, to build some intermediary layer with the common stuff. Eventually, due to deep deps, some may end needing merging some layers, like qt or plasma and gonme-common. On the user side the result is the same, indeed, but on repo builder side does not make sense, probably.

My question above is still in place:

Why gnome layer requires gnome-common and X layers while gnome-common requires X already? In other words why requiring gnome-common is not enough (by bringing X in its requirements)? Or is it?

I mean, why would need that (apparently) redundancy?

Then, why qt is emerged for the presented setup? What layer is brought of the required ones then? Why gnome-common and not plasma? I presume it's gnome-common, because it seems to start emerging qt - and I expect to continue up to plasma.

For such a case, seems necessary to merge the containers somehow, to end up with plasma + gnome-common and build. Otherwise takes forever and we shoot flies with a big cannon...

I would analyze this particular case if it's possible to move atoms in a way that it makes it build with a smaller print, but this is just a case.

mudler commented 2 years ago

Actually, I thought this myself. But, seems some KDE apps need Gnome libs sometimes, due to missing those implementation in a KDE/Qt lib. This would need, for instance, to build some intermediary layer with the common stuff. Eventually, due to deep deps, some may end needing merging some layers, like qt or plasma and gonme-common. On the user side the result is the same, indeed, but on repo builder side does not make sense, probably.

Yes, you can definitely have a building layer that contains all the deps, and have smaller packages extracting just the required from it - there is no need to make that layer installable for the user

msdobrescu commented 2 years ago

Yes, you can definitely have a building layer that contains all the deps, and have smaller packages extracting just the required from it - there is no need to make that layer installable for the user

That seems out of my reach as mOS user. Or I don't have the knowledge how, given the current layers definition.

mudler commented 2 years ago

Yes, you can definitely have a building layer that contains all the deps, and have smaller packages extracting just the required from it - there is no need to make that layer installable for the user

That seems out of my reach as mOS user. Or I don't have the knowledge how, given the current layers definition.

That's why I think this is a mOS issue rather then a luet one... Nevertheless https://luet-lab.github.io/docs/docs/concepts/packages/specfile/#building-strategies should give you some indications: you can hust select the parts you are interested to from a layer and create packages as necessary

msdobrescu commented 2 years ago

Is this what you mean? The outcome is here. Is it right? If not, what should I change?

msdobrescu commented 2 years ago

I think I've solved this. Config is here. Log is here.

mudler commented 2 years ago

Closing the issue then :+1:

mudler / luet

Build the minimum amount of atoms a package depends on #269