search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.1k stars 959 forks source link

[question] GH Action workflow is caching missing packages? #15847

Closed manuelpagliuca closed 5 months ago

manuelpagliuca commented 5 months ago

What is your question?

Good morning, I'm attempting to make a GH Action workflow to run with conan, I've updated the previous ADKs with new binary versions, but it seems that this GH workflow can't access to the ADK and it is still caching something else.

So I tried to force the building pipeline using the --build=missing flag on the conan install command, but this doesn't fetch anything and keeps saying Cached even if the caching key is a new one and nothing is registerd with that.

Then when it reaches the compilation steps it fails because it is searching for a file which is positioned somewhere else in the old ADK.

image

This is the log of the conan configuration and install, and the other workflow is identical in this

Run mkdir build_Browser
  mkdir build_Browser
  md build-[2](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:2)5
  cd build-25
  conan config install ..\code\conan\config
  conan user github --remote jfrog --password "***"
  conan install ..\code --profile win_ac25 --settings build_type=MinSizeRel --build=missing
  cd ..\build_Browser
  conan install ..\code --profile cef_release --settings build_type=Release --build=missing
  cd ..
  shell: C:\Windows\system[3](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:3)2\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "{0}""
  env:
    BUILD_TYPE: MinSizeRel
    CONAN_REVISIONS_ENABLED: 0
    JfrogPlannerSuiteVersion: 0.0.0.0
    AZURE_CREDENTIALS: {"clientId":"***","clientSecret":"***","subscriptionId":"***","tenantId":"***"}
    AZ_CLI_VERSION: 2.25.0
    SUBSCRIPTION: ***
A subdirectory or file build_Browser already exists.
A subdirectory or file build-25 already exists.
Defining remotes from remotes.txt
Copying file cef_release to C:\Users\bimplugin\.conan\profiles
Copying file cef_release_macos to C:\Users\bimplugin\.conan\profiles
Copying file cef_release_macos_arm6[4](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:4) to C:\Users\bimplugin\.conan\profiles
Copying file macos to C:\Users\bimplugin\.conan\profiles
Copying file macos_arm64 to C:\Users\bimplugin\.conan\profiles
Copying file win_ac24 to C:\Users\bimplugin\.conan\profiles
Copying file win_ac2[5](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:5) to C:\Users\bimplugin\.conan\profiles
Copying file win_ac2[6](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:6) to C:\Users\bimplugin\.conan\profiles
Copying file win_ac2[7](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:7) to C:\Users\bimplugin\.conan\profiles
Copying file win_dbg_ac25 to C:\Users\bimplugin\.conan\profiles
Copying file win_dbg_ac26 to C:\Users\bimplugin\.conan\profiles
Copying file win_dbg_ac27 to C:\Users\bimplugin\.conan\profiles
User of remote 'jfrog' is already 'github'
Configuration:
[settings]
arch=x[8](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:8)6_64
arch_build=x86_64
build_type=MinSizeRel
compiler=Visual Studio
compiler.cppstd=17
compiler.runtime=MD
compiler.toolset=v142
compiler.version=16
os=Windows
os_build=Windows
cmake:build_type=Release
ninja:build_type=Release
[options]
*:archicad=25
[build_requires]
[env]
conanfile.py: Installing package
Requirements
    archicad_adk/25.3002 from 'jfrog' - Cache
    boost/1.77.0 from 'jfrog' - Cache
    bzip2/1.0.8 from 'jfrog' - Cache
    catch2/2.13.6 from 'jfrog' - Cache
    fakeit/2.0.[9](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:9) from 'jfrog' - Cache
    fmt/7.1.3 from 'jfrog' - Cache
    gsl-lite/0.38.1 from 'jfrog' - Cache
    magic_enum/0.7.3 from 'jfrog' - Cache
    range-v3/0.5.0 from 'jfrog' - Cache
    spdlog/1.8.5 from 'jfrog' - Cache
    zlib/1.2.11 from 'jfrog' - Cache
Packages
    archicad_adk/25.3002:ca33edce272a279b24f87dc0d4cf5bbdcffbc187 - Cache
    boost/1.77.0:2dbbe13082b9b3bd3696299be2025f072dc96031 - Cache
    bzip2/1.0.8:786618c232bc8fb9697b338c7d6b440edfbafc64 - Cache
    catch2/2.13.6:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    fakeit/2.0.9:95972c9e061c46d3c1856bb1ab1d009b4121c367 - Cache
    fmt/7.1.3:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    gsl-lite/0.38.1:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    magic_enum/0.7.3:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    range-v3/0.5.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    spdlog/1.8.5:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    zlib/1.2.11:757854854851bee2af59d0[10](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:10)9[11](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:11)4aea6e5aff305 - Cache
Build requirements
    cmake/3.22.0 from 'jfrog' - Cache
Build requirements packages
    cmake/3.22.0:0a420ff5c47119e668867cdb51baff0eca1fdb68 - Cache
Installing (downloading, building) binaries...
archicad_adk/25.3002: Already installed!
bzip2/1.0.8: Already installed!
bzip2/1.0.8: Appending PATH environment variable: C:\Users\bimplugin\.conan\data\bzip2\1.0.8\_\_\package\786618c232bc8fb9697b338c7d6b440edfbafc64\bin
catch2/2.13.6: Already installed!
cmake/3.22.0: Already installed!
cmake/3.22.0: Appending PATH environment variable: C:\Users\bimplugin\.conan\data\cmake\3.22.0\_\_\package\0a420ff5c47119e668867cdb51baff0eca1fdb68\bin
fakeit/2.0.9: Already installed!
fmt/7.1.3: Already installed!
gsl-lite/0.38.1: Already installed!
magic_enum/0.7.3: Already installed!
range-v3/0.5.0: Already installed!
zlib/1.2.11: Already installed!
boost/1.77.0: Already installed!
boost/1.77.0: Disabled magic autolinking (smart and magic decisions)
spdlog/1.8.5: Already installed!
conanfile.py: Applying build-requirement: cmake/3.22.0
conanfile.py: Generator cmake created conanbuildinfo.cmake
conanfile.py: Generator virtualenv created activate.bat
conanfile.py: Generator virtualenv created deactivate.bat
conanfile.py: Generator virtualenv created environment.bat.env
conanfile.py: Generator virtualenv created activate.ps1
conanfile.py: Generator virtualenv created deactivate.ps1
conanfile.py: Generator virtualenv created environment.ps1.env
conanfile.py: Generator virtualenv created activate.sh
conanfile.py: Generator virtualenv created deactivate.sh
conanfile.py: Generator virtualenv created environment.sh.env
conanfile.py: Generator txt created conanbuildinfo.txt
conanfile.py: Aggregating env generators
conanfile.py: Generated conaninfo.txt
conanfile.py: Generated graphinfo
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.cppstd=17
compiler.runtime=MD
compiler.toolset=v142
compiler.version=16
os=Windows
os_build=Windows
cmake:build_type=Release
ninja:build_type=Release
[options]
[build_requires]
[env]
conanfile.py: Installing package
Requirements
    archicad_adk/26.3000 from 'jfrog' - Cache
    boost/1.77.0 from 'jfrog' - Cache
    bzip2/1.0.8 from 'jfrog' - Cache
    catch2/2.13.6 from 'jfrog' - Cache
    fakeit/2.0.9 from 'jfrog' - Cache
    fmt/7.1.3 from 'jfrog' - Cache
    gsl-lite/0.38.1 from 'jfrog' - Cache
    magic_enum/0.7.3 from 'jfrog' - Cache
    range-v3/0.[12](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:12).0 from 'jfrog' - Cache
    spdlog/1.8.5 from 'jfrog' - Cache
    zlib/1.2.11 from 'jfrog' - Cache
Packages
    archicad_adk/26.3000:ca33edce272a279b24f87dc0d4cf5bbdcffbc187 - Cache
    boost/1.77.0:92322e62bf8feadbdf3ded4d807f2c5532a289cf - Cache
    bzip2/1.0.8:d16a91eadaaf5829b928b12d2f836ff7680d3df5 - Cache
    catch2/2.[13](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:13).6:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    fakeit/2.0.9:95972c9e061c46d3c1856bb1ab1d009b4121c367 - Cache
    fmt/7.1.3:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    gsl-lite/0.38.1:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    magic_enum/0.7.3:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    range-v3/0.12.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    spdlog/1.8.5:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache
    zlib/1.2.11:3fb49604f9c2f729b85ba31[15](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:15)852006824e72cab - Cache
Build requirements
    cmake/3.22.0 from 'jfrog' - Cache
Build requirements packages
    cmake/3.22.0:0a420ff5c47119e668867cdb51baff0eca1fdb68 - Cache
Installing (downloading, building) binaries...
archicad_adk/26.3000: Already installed!
bzip2/1.0.8: Already installed!
bzip2/1.0.8: Appending PATH environment variable: C:\Users\bimplugin\.conan\data\bzip2\1.0.8\_\_\package\d[16](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:16)a91eadaaf5829b928b12d2f836ff7680d3df5\bin
catch2/2.13.6: Already installed!
cmake/3.22.0: Already installed!
cmake/3.22.0: Appending PATH environment variable: C:\Users\bimplugin\.conan\data\cmake\3.22.0\_\_\package\0a420ff5c471[19](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:20)e668867cdb51baff0eca1fdb68\bin
fakeit/2.0.9: Already installed!
fmt/7.1.3: Already installed!
gsl-lite/0.38.1: Already installed!
magic_enum/0.7.3: Already installed!
range-v3/0.12.0: Already installed!
zlib/1.2.11: Already installed!
boost/1.77.0: Already installed!
boost/1.77.0: Disabled magic autolinking (smart and magic decisions)
spdlog/1.8.5: Already installed!
conanfile.py: Applying build-requirement: cmake/3.[22](https://github.com/jfrog-group/BIM-ArchiCAD-Plugin/actions/runs/8232144854/job/22508926740#step:9:23).0
conanfile.py: Generator virtualenv created activate.bat
conanfile.py: Generator virtualenv created deactivate.bat
conanfile.py: Generator virtualenv created environment.bat.env
conanfile.py: Generator virtualenv created activate.ps1
conanfile.py: Generator virtualenv created deactivate.ps1
conanfile.py: Generator virtualenv created environment.ps1.env
conanfile.py: Generator virtualenv created activate.sh
conanfile.py: Generator virtualenv created deactivate.sh
conanfile.py: Generator virtualenv created environment.sh.env
conanfile.py: Generator txt created conanbuildinfo.txt
conanfile.py: Generator cmake created conanbuildinfo.cmake
conanfile.py: Aggregating env generators
conanfile.py: Generated conaninfo.txt
conanfile.py: Generated graphinfo

This is the conan install action:

      - name: Configure conan and install dependencies for plugin and browser
        run: |
          mkdir build_Browser
          md build-${{ matrix.adk }}
          cd build-${{ matrix.adk }}
          conan config install ..\code\conan\config
          conan user github --remote jfrog --password "${{ secrets.CONAN_TOKEN_BIMPLUGIN_GITHUB }}"
          conan install ..\code --profile win_ac${{ matrix.adk }} --settings build_type=${{ env.BUILD_TYPE }} --build=missing
          cd ..\build_Browser
          conan install ..\code --profile cef_release --settings build_type=Release --build=missing
          cd ..

Could you please help me debugging this issue? I feel like I'm missing some pieces of the puzzle, thanks in advance!

Have you read the CONTRIBUTING guide?

memsharded commented 5 months ago

Hi @manuelpagliuca

So I tried to force the building pipeline using the --build=missing flag on the conan install command, but this doesn't fetch anything and keeps saying Cached even if the caching key is a new one and nothing is registerd with that.

Sure, the --build=missing only builds when the binary is not existing, but if the binary is existing in the cache, it will use it, not build it again

CONAN_REVISIONS_ENABLED: 0

This could be related. Using revisions has been encouraged for many years now, working without revisions gives 0 control of modifications. Code can change, but if there is no version bump, nothing will change because revisions are not there, and the cache packages will still be used, because they are all the same.

Furthermore, it seems there are some things too broken there, reading the logs:

Please, read carefully the logs, and try to understand what they are printing, those logs with those things doesn't make any sense to me.

Also, this is the kind of issue we cannot help much, debugging and understanding issues in the users CI that we cannot reproduce is mostly impossible. This is why we insist in having things that we can reproduce on our end. It is fine if you setup a toy project in Github, with GH actions, that is open source and fully self-contained, so it can be reproduced. But otherwise there is simply not enough information that would allow us to understand what is happening. Thanks very much!

manuelpagliuca commented 5 months ago

HI @memsharded first of all thanks for answering my question there!

I would like to know if there is a way for forcing the download the binaries anyway, or if it is possible to remove the current cache from the workflow.

manuelpagliuca commented 5 months ago

I solved it by appending the flag --update on conan install, thanks anyway!