Dependencies of mentioned packages are not being cached?

matinlotfali commented 2 years ago

Hi. I am using your apt action to cache my apt dependencies in my repo https://github.com/matinlotfali/KDE-Rounded-Corners I noticed that when I list my packages in the yml file like this:

    - name: Cache APT Packages
      uses: awalsh128/cache-apt-pkgs-action@v1.0.2
      with:
        packages: gettext extra-cmake-modules qttools5-dev libqt5x11extras5-dev libkf5configwidgets-dev libkf5crash-dev libkf5globalaccel-dev libkf5kio-dev libkf5notifications-dev kinit-dev kwin-dev
        version: 20.04
        refresh: true

It successfully installs them and then reports that it has saved only 2 MB in the post job. But I believe installing these packages will include around 300 MB of dependencies that are not being kept in the cache and the list is too long to include all of them (It would be the whole Kubuntu installation).

As a result, the next time the action is run, it reuses the cache and because dependencies are not met, the build fails in the CMAKE.

matinlotfali commented 2 years ago

Is it a duplicate of issue #13 ?

awalsh128 commented 2 years ago

Sorry for the lag responding, was away from a system. I don't believe this is related to #13 since you verified that these are the actual deps. I need to follow up on #13 because it isn't so clear to me after working on it that the original speculation is true.

I need to cleanup staging but I think the next step is to add some more debugging to the action's outputs and logging. Could you provide some info for me?

The package list that gets outputted.
The post job logs with the reported size.
The entire log from the action.

matinlotfali commented 2 years ago

When I run the command below:

sudo apt install -y gettext extra-cmake-modules qttools5-dev libqt5x11extras5-dev libkf5configwidgets-dev libkf5crash-dev libkf5globalaccel-dev libkf5kio-dev libkf5notifications-dev kinit-dev kwin-dev

on the ubuntu-22.04 Github runner, The following NEW packages will be installed:

  accountsservice appmenu-gtk-module-common appmenu-gtk3-module appstream
  apt-config-icons apt-config-icons-hidpi apt-config-icons-large
  apt-config-icons-large-hidpi avahi-daemon bluedevil bluez bluez-obexd breeze
  breeze-cursor-theme breeze-gtk-theme breeze-icon-theme bubblewrap bup
  bup-doc catdoc debconf-kde-data debconf-kde-helper docbook-xml docbook-xsl
  drkonqi extra-cmake-modules fonts-hack fonts-noto-cjk fonts-noto-core
  fonts-noto-hinted fonts-noto-mono fonts-noto-ui-core fonts-noto-unhinted
  frameworkintegration gamin gdb geoclue-2.0 gettext gstreamer1.0-gl
  gstreamer1.0-plugins-base gtk2-engines-pixbuf hwdata i965-va-driver
  ibus-data ieee-data iio-sensor-proxy intel-media-va-driver
  kaccounts-providers kactivities-bin kactivitymanagerd kde-cli-tools
  kde-cli-tools-data kde-config-gtk-style kde-config-screenlocker
  kde-config-sddm kde-config-updates kde-style-breeze kde-style-oxygen-qt5
  kdeconnect kded5 kdoctools5 keditbookmarks kgamma5 khelpcenter khotkeys
  khotkeys-data kinfocenter kinit kinit-dev kio kio-extras kio-extras-data
  kmenuedit kpackagelauncherqml kpackagetool5 kpeople-vcard kscreen
  ksshaskpass ksystemstats ktexteditor-data ktexteditor-katepart kup-backup
  kuserfeedback-doc kwalletmanager kwayland-data kwayland-integration
  kwin-common kwin-data kwin-dev kwin-style-breeze kwin-x11 kwrited
  layer-shell-qt liba52-0.7.4 libaa1 libaacs0 libaccounts-glib0
  libaccounts-qt5-1 libaccountsservice0 libappimage0 libappmenu-gtk3-parser0
  libappstreamqt2 libaribb24-0 libasound2-plugins libass9 libasyncns0
  libavahi-core7 libavc1394-0 libavcodec58 libavformat58 libavutil56
  libbabeltrace1 libbdplus0 libbluray2 libboost-regex1.74.0 libc6-dbg libcaca0
  libcanberra-pulse libcanberra0 libcddb2 libcdparanoia0 libchromaprint1
  libcodec2-1.0 libcolorcorrect5 libdaemon0 libdbusmenu-qt5-2 libdc1394-25
  libdca0 libdebconf-kde1 libdebuginfod-common libdebuginfod1 libdmtx0b
  libdouble-conversion3 libdrm-intel1 libdvbpsi10 libdvdnav4 libdvdread8
  libebml5 libeditorconfig0 libegl-dev libegl-mesa0 libegl1 libegl1-mesa-dev
  libepoxy-dev libepub0 libevdev2 libevent-2.1-7 libexiv2-27 libfaad2
  libfakekey0 libfftw3-single3 libflac8 libfuse2 libgamin0 libgif7 libgit2-1.1
  libgl-dev libgles-dev libgles1 libgles2 libglu1-mesa libglu1-mesa-dev
  libglvnd-core-dev libglvnd-dev libglx-dev libgme0 libgpgmepp6 libgps28
  libgrantlee-templates5 libgraphene-1.0-0 libgsettings-qt1 libgsm1
  libgstreamer-gl1.0-0 libgstreamer-plugins-base1.0-0 libhfstospell11
  libhttp-parser2.9 libhyphen0 libibus-1.0-5 libical3 libidn12 libigdgmm12
  libimobiledevice6 libinput-bin libinput10 libipt2 libixml10 libjack-jackd2-0
  libkaccounts2 libkate1 libkdecorations2-5v5 libkdecorations2private9
  libkdsoap1 libkf5activities5 libkf5activitiesstats1 libkf5archive5
  libkf5attica-dev libkf5attica-doc libkf5attica5 libkf5auth-data
  libkf5auth-dev libkf5auth-dev-bin libkf5auth-doc libkf5auth5 libkf5authcore5
  libkf5baloo5 libkf5balooengine5 libkf5bluezqt-data libkf5bluezqt6
  libkf5bookmarks-data libkf5bookmarks-dev libkf5bookmarks-doc
  libkf5bookmarks5 libkf5calendarevents5 libkf5codecs-data libkf5codecs-dev
  libkf5codecs-doc libkf5codecs5 libkf5completion-data libkf5completion-dev
  libkf5completion-doc libkf5completion5 libkf5config-bin libkf5config-data
  libkf5config-dev libkf5config-dev-bin libkf5config-doc libkf5configcore5
  libkf5configgui5 libkf5configwidgets-data libkf5configwidgets-dev
  libkf5configwidgets-doc libkf5configwidgets5 libkf5contacts-data
  libkf5contacts5 libkf5coreaddons-data libkf5coreaddons-dev
  libkf5coreaddons-dev-bin libkf5coreaddons-doc libkf5coreaddons5
  libkf5crash-dev libkf5crash-doc libkf5crash5 libkf5dbusaddons-bin
  libkf5dbusaddons-data libkf5dbusaddons-dev libkf5dbusaddons-doc
  libkf5dbusaddons5 libkf5declarative-data libkf5declarative5 libkf5dnssd-data
  libkf5dnssd5 libkf5doctools5 libkf5filemetadata-bin libkf5filemetadata-data
  libkf5filemetadata3 libkf5globalaccel-bin libkf5globalaccel-data
  libkf5globalaccel-dev libkf5globalaccel-doc libkf5globalaccel5
  libkf5globalaccelprivate5 libkf5guiaddons-bin libkf5guiaddons-data
  libkf5guiaddons-dev libkf5guiaddons-doc libkf5guiaddons5 libkf5holidays-data
  libkf5holidays5 libkf5i18n-data libkf5i18n-dev libkf5i18n-doc libkf5i18n5
  libkf5i18nlocaledata5 libkf5iconthemes-bin libkf5iconthemes-data
  libkf5iconthemes-dev libkf5iconthemes-doc libkf5iconthemes5 libkf5idletime5
  libkf5itemmodels5 libkf5itemviews-data libkf5itemviews-dev
  libkf5itemviews-doc libkf5itemviews5 libkf5jobwidgets-data
  libkf5jobwidgets-dev libkf5jobwidgets-doc libkf5jobwidgets5 libkf5js5
  libkf5kcmutils-data libkf5kcmutils5 libkf5kdelibs4support-data
  libkf5kdelibs4support5 libkf5kdelibs4support5-bin libkf5khtml-bin
  libkf5khtml-data libkf5khtml5 libkf5kio-dev libkf5kio-doc libkf5kiocore5
  libkf5kiofilewidgets5 libkf5kiogui5 libkf5kiontlm5 libkf5kiowidgets5
  libkf5kirigami2-5 libkf5networkmanagerqt6 libkf5newstuff-data
  libkf5newstuff5 libkf5newstuffcore5 libkf5notifications-data
  libkf5notifications-dev libkf5notifications-doc libkf5notifications5
  libkf5notifyconfig-data libkf5notifyconfig5 libkf5package-data
  libkf5package5 libkf5parts-data libkf5parts-plugins libkf5parts5
  libkf5people-data libkf5people5 libkf5peoplebackend5 libkf5peoplewidgets5
  libkf5plasma5 libkf5plasmaquick5 libkf5prison5 libkf5pty-data libkf5pty5
  libkf5pulseaudioqt3 libkf5purpose-bin libkf5purpose5 libkf5quickaddons5
  libkf5runner5 libkf5screen-bin libkf5screen7 libkf5service-bin
  libkf5service-data libkf5service-dev libkf5service-doc libkf5service5
  libkf5solid-dev libkf5solid-doc libkf5solid5 libkf5solid5-data
  libkf5sonnet5-data libkf5sonnetcore5 libkf5sonnetui5 libkf5style5
  libkf5su-bin libkf5su-data libkf5su5 libkf5syndication5abi1
  libkf5syntaxhighlighting-data libkf5syntaxhighlighting5 libkf5sysguard-data
  libkf5texteditor-bin libkf5texteditor5 libkf5textwidgets-data
  libkf5textwidgets5 libkf5threadweaver5 libkf5wallet-bin libkf5wallet-data
  libkf5wallet5 libkf5waylandclient5 libkf5waylandserver5
  libkf5widgetsaddons-data libkf5widgetsaddons-dev libkf5widgetsaddons-doc
  libkf5widgetsaddons5 libkf5windowsystem-data libkf5windowsystem-dev
  libkf5windowsystem-doc libkf5windowsystem5 libkf5xmlgui-bin
  libkf5xmlgui-data libkf5xmlgui-dev libkf5xmlgui-doc libkf5xmlgui5
  libkfontinst5 libkfontinstui5 libkpmcore11 libkscreenlocker5 libksgrd9
  libksysguardformatter1 libksysguardsensorfaces1 libksysguardsensors1
  libksysguardsystemstats1 libkuserfeedbackcore1 libkwalletbackend5-5
  libkwaylandserver5 libkwineffects13 libkwinglutils13 libkwinxrenderutils13
  libkworkspace5-5 liblayershellqtinterface5 libldb2 liblirc-client0
  liblua5.2-0 libmad0 libmarkdown2 libmatroska7 libmbedcrypto7 libmbedtls14
  libmbedx509-1 libmd4c0 libmfx1 libminizip1 libmp3lame0 libmpcdec6 libmpeg2-4
  libmpg123-0 libmtdev1 libmtp-common libmtp-runtime libmtp9 libmysofa1
  libnfs13 libnl-route-3-200 libnotificationmanager1 libnotify4 libnss-mdns
  libogg0 libopengl-dev libopengl0 libopenmpt-modplug1 libopenmpt0 libopus0
  liborc-0.4-0 liboxygenstyle5-5 liboxygenstyleconfig5-5 libpackagekitqt5-1
  libpam-kwallet-common libpam-kwallet5 libpciaccess0 libpcsclite1
  libphonon4qt5-4 libphonon4qt5-data libpipewire-0.3-0 libpipewire-0.3-common
  libpipewire-0.3-modules libplacebo192 libplasma-geolocation-interface5
  libplist3 libpolkit-qt5-1-1 libpoppler-qt5-1 libpoppler118 libpostproc55
  libpowerdevilcore2 libpowerdevilui5 libprocesscore9 libprocessui9
  libproxy-tools libpulse-mainloop-glib0 libpulse0 libpulsedsp
  libqaccessibilityclient-qt5-0 libqalculate-data libqalculate22 libqca-qt5-2
  libqca-qt5-2-plugins libqmobipocket2 libqrencode4 libqt5concurrent5
  libqt5core5a libqt5dbus5 libqt5designer5 libqt5designercomponents5
  libqt5gui5 libqt5help5 libqt5hunspellinputmethod5 libqt5multimedia5
  libqt5multimedia5-plugins libqt5multimediagsttools5 libqt5multimediaquick5
  libqt5multimediawidgets5 libqt5network5 libqt5opengl5 libqt5opengl5-dev
  libqt5positioning5 libqt5printsupport5 libqt5qml5 libqt5qmlmodels5
  libqt5qmlworkerscript5 libqt5quick5 libqt5quickcontrols2-5
  libqt5quickparticles5 libqt5quickshapes5 libqt5quicktemplates2-5
  libqt5quickwidgets5 libqt5script5 libqt5scripttools5 libqt5sensors5
  libqt5sql5 libqt5sql5-sqlite libqt5svg5 libqt5test5 libqt5texttospeech5
  libqt5virtualkeyboard5 libqt5waylandclient5 libqt5waylandcompositor5
  libqt5webchannel5 libqt5webengine-data libqt5webengine5 libqt5webenginecore5
  libqt5webenginewidgets5 libqt5webkit5 libqt5widgets5 libqt5x11extras5
  libqt5x11extras5-dev libqt5xml5 libraw1394-11 libresid-builder0c2a
  libsamplerate0 libscim8v5 libsdl-image1.2 libsdl1.2debian libshine3
  libshout3 libsidplay2 libsignon-plugins-common1 libsignon-qt5-1 libsmbclient
  libsnapd-glib1 libsnapd-qt1 libsndfile1 libsndio7.0
  libsource-highlight-common libsource-highlight4v5 libsoxr0
  libspa-0.2-modules libspatialaudio0 libspeechd2 libspeex1 libspeexdsp1
  libsquashfuse0 libsrt1.4-gnutls libssh-gcrypt-4 libswresample3 libswscale5
  libtag1v5 libtag1v5-vanilla libtaskmanager6 libtdb1 libtheora0 libtwolame0
  libudfread0 libupnp13 libupower-glib3 libusbmuxd6 libva-drm2 libva-wayland2
  libva-x11-2 libva2 libvdpau1 libvisual-0.4-0 libvlc-bin libvlc5 libvlccore9
  libvoikko1 libvorbis0a libvorbisenc2 libvorbisfile3 libvpx7 libvulkan-dev
  libwacom-bin libwacom-common libwacom9 libweather-ion7
  libwebrtc-audio-processing1 libwoff1 libx264-163 libxapian30
  libxcb-composite0 libxcb-cursor0 libxcb-damage0 libxcb-dpms0 libxcb-icccm4
  libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-record0
  libxcb-render-util0 libxcb-res0 libxcb-shape0-dev libxcb-util1
  libxcb-xfixes0-dev libxcb-xinerama0 libxcb-xinput0 libxcb-xkb1 libxcb-xv0
  libxkbcommon-x11-0 libxml2-utils libxres1 libxvidcore4 libzvbi-common
  libzvbi0 libzxingcore1 media-player-info mesa-utils mesa-utils-bin
  mesa-va-drivers mesa-vdpau-drivers milou ocl-icd-libopencl1 oxygen-sounds
  par2 partitionmanager phonon4qt5 phonon4qt5-backend-vlc pipewire
  pipewire-bin pipewire-media-session plasma-browser-integration
  plasma-desktop plasma-desktop-data plasma-discover
  plasma-discover-backend-fwupd plasma-discover-backend-snap
  plasma-discover-common plasma-disks plasma-framework plasma-integration
  plasma-pa plasma-systemmonitor plasma-thunderbolt plasma-workspace
  plasma-workspace-data polkit-kde-agent-1 poppler-data power-profiles-daemon
  powerdevil powerdevil-data pulseaudio pulseaudio-module-gsettings
  pulseaudio-utils python3-fuse python3-ldb python3-pylibacl python3-pyqt5
  python3-pyqt5.sip python3-pyxattr python3-talloc python3-tornado qdbus-qt5
  qdoc-qt5 qhelpgenerator-qt5 qml-module-gsettings1.0
  qml-module-org-kde-activities qml-module-org-kde-bluezqt
  qml-module-org-kde-draganddrop qml-module-org-kde-kcm
  qml-module-org-kde-kconfig qml-module-org-kde-kcoreaddons
  qml-module-org-kde-kholidays qml-module-org-kde-kio
  qml-module-org-kde-kirigami2 qml-module-org-kde-kitemmodels
  qml-module-org-kde-kquickcontrols qml-module-org-kde-kquickcontrolsaddons
  qml-module-org-kde-ksysguard qml-module-org-kde-kwindowsystem
  qml-module-org-kde-newstuff qml-module-org-kde-people
  qml-module-org-kde-prison qml-module-org-kde-purpose
  qml-module-org-kde-qqc2desktopstyle qml-module-org-kde-quickcharts
  qml-module-org-kde-runnermodel qml-module-org-kde-solid
  qml-module-org-kde-sonnet qml-module-org-kde-userfeedback
  qml-module-qt-labs-folderlistmodel qml-module-qt-labs-platform
  qml-module-qt-labs-qmlmodels qml-module-qt-labs-settings
  qml-module-qtgraphicaleffects qml-module-qtmultimedia qml-module-qtqml
  qml-module-qtqml-models2 qml-module-qtquick-controls
  qml-module-qtquick-controls2 qml-module-qtquick-dialogs
  qml-module-qtquick-layouts qml-module-qtquick-particles2
  qml-module-qtquick-privatewidgets qml-module-qtquick-shapes
  qml-module-qtquick-templates2 qml-module-qtquick-virtualkeyboard
  qml-module-qtquick-window2 qml-module-qtquick2 qml-module-qtwebengine
  qml-module-ubuntu-onlineaccounts qt5-assistant qt5-gtk-platformtheme
  qt5-qmake qt5-qmake-bin qtattributionsscanner-qt5 qtbase5-dev
  qtbase5-dev-tools qtchooser qtscript5-dev qtspeech5-speechd-plugin
  qttools5-dev qttools5-dev-tools qttranslations5-l10n
  qtvirtualkeyboard-plugin qtwayland5 rtkit samba-libs sgml-base sgml-data
  signon-plugin-oauth2 smartmontools socat software-properties-qt
  sonnet-plugins sound-theme-freedesktop sshfs systemsettings upower usbmuxd
  va-driver-all vdpau-driver-all vlc-data vlc-plugin-base
  vlc-plugin-video-output vulkan-tools wpasupplicant xdg-desktop-portal
  xdg-desktop-portal-kde xml-core xsettingsd

Need to get 412 MB of archives. After this operation, 1414 MB of additional disk space will be used.

However, it only caches the packages that I mention in the action (first comment) which is only 2 MB. I will provide logs later.

pablgonz commented 2 years ago

@matinlotfali Just for fun :D

name: Kubuntu 22.04

on:
  push:
    branches: [ master, latest-shapecorners-code, kwin_5.24 ]
  pull_request:
    branches: [ master ]

env:
  # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.)
  BUILD_TYPE: Release

jobs:
  ubuntu2204:
    # The CMake configure and build commands are platform agnostic and should work equally
    # well on Windows or Mac.  You can convert this to a matrix build if you need
    # cross-platform coverage.
    # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix
    runs-on: ubuntu-22.04

    steps:
    - name: Check out repository
      uses: actions/checkout@v3

    # Speed up your github actions on ubuntu-latest
    - name: Run  github-actions-tune
      uses: abbbi/github-actions-tune@v1

    - name: Refresh Ubuntu Packages
      run: sudo apt-fast -y update

    - name: Install Dependencies
      run: sudo apt-fast install -y --no-install-recommends gettext extra-cmake-modules qttools5-dev libqt5x11extras5-dev libkf5configwidgets-dev libkf5crash-dev libkf5globalaccel-dev libkf5kio-dev libkf5notifications-dev kinit-dev kwin-dev libqt5opengl5-dev

    - name: Configure CMake
      # Configure CMake in a 'build' subdirectory. `CMAKE_BUILD_TYPE` is only required if you are using a single-configuration generator such as make.
      # See https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html?highlight=cmake_build_type
      run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}}

    - name: Build
      # Build your program with the given configuration
      run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}}

#     - name: Test
#       working-directory: ${{github.workspace}}/build
#       # Execute tests defined by the CMake configuration.
#       # See https://cmake.org/cmake/help/latest/manual/ctest.1.html for more detail
#       run: ctest -C ${{env.BUILD_TYPE}}

(https://github.com/pablgonz/KDE-Rounded-Corners/actions/workflows/kubuntu2204.yml)

It doesn't solve the request, but, it reduces considerably the execution time, I don't know anything about the code you compile, but if you know the essential libraries you can (I tried it but it wasn't fun anymore ;).

I hope it will be of help

Saludos

matinlotfali commented 2 years ago

That is very good! I probably use this. Thanks a lot @pablgonz

awalsh128 commented 2 years ago

Hey @matinlotfali sorry for the wait; I was away for 3 weeks without electronics. I have a change that I am doing to consider the dependencies as well and will test against your use case. The cache supports up to 5GB so I don't see this being a problem. Hoping to get something out soon.

awalsh128 commented 2 years ago

Okay, looks like we might have something in staging (tests passed). Update the action to ref @staging instead of the version and let me know how it goes. If this works I may need some feedback from you afterwards.

pablgonz commented 2 years ago

@awalsh128 Tested in my fork (https://github.com/pablgonz/KDE-Rounded-Corners/actions/runs/2678911156) OK

awalsh128 commented 2 years ago

You rock @pablgonz. I'll cut a release later today.

matinlotfali commented 2 years ago

Thanks both of you @pablgonz and @awalsh128 ! I am closing the issue.

pablgonz commented 2 years ago

Okay, looks like we might have something in staging (tests passed). Update the action to ref @staging instead of the version and let me know how it goes. If this works I may need some feedback from you afterwards.

The branch @master have a problem:

E: No packages found
07:18:38   * Caching extra-cmake-modules:5.92.0-0ubuntu1 to /home/runner/cache-apt-pkgs/extra-cmake-modules:5.92.0-0ubuntu1.tar.gz...
dpkg-query: error: --listfiles needs a valid package name but 'extra-cmake-modules:5.92.0-0ubuntu1' is not: illegal architecture name in specifier 'extra-cmake-modules:5.92.0-0ubuntu1': character '.' not allowed (only letters, digits and characters '-')

The @stage works OK

awalsh128 commented 2 years ago

Yeah, thanks for noting, I have a pending commit to fix this. This should be in staging though. Going to revert master so it is stable later this weekend.

awalsh128 / cache-apt-pkgs-action

Dependencies of mentioned packages are not being cached? #21