[Bug]: rdeps.py dont seems work anymore.

sorasoras commented 8 months ago

Describe the bug

A clear and concise description of what the bug or problem is.

To Reproduce

Precise version of rocBLAS installed or rocBLAS commit hash if building from source. Steps to reproduce the behavior:

build rocblas 5.5 from source for windows
See error on logfile log.txt

TorreZuk commented 8 months ago

Not sure if you saw examples of details in the issue template but adding some more information will help.
You say doesn't seem to work anymore, so when did it work for you? I see drives C: D: and W: all referred to as well as proxy errors, but could be windows updates make the hard coded vcpkg release tag (in rdeps.py) incompatible. You can try with newer vcpkg releases. Can you confirm if you set your env VCPKG_PATH and don't use a vpkg install from visual studio that you updated?
Also good to know your visual studio version along with vcpkg version. I can try to reproduce with latest visual studio version and the 5.5 branch tip, on Windows 11 latest patch.

sorasoras commented 8 months ago

But the mirrors is no longer available 404 Not Found

TorreZuk commented 8 months ago

I'll see if I can find out what is up with the mirror list issue but for 5.5 you want to match this vcpkg: git clone -b 2022.05.10 https://github.com/microsoft/vcpkg The -b flag gives you release tag 2022.05.10 that should give you download lists for the version that was tested as working. If you just clone you get an unreleased version, not to say all later releases have been tested but is the original non-mirror download site retired or just temporarily down needs to be checked. Make sure you are not blocking it directly by fetch/ping test. The old vcpkg tag could also have problems with newer toolchain....

sorasoras commented 8 months ago

I dont think switch to 2022.05.10 vcpkg help. logs.txt The only workaround i can think of is to download them one by one and load it as cache.

Also, I just find this out at here "https://github.com/microsoft/vcpkg/issues/35359#issuecomment-1828882746" I guess this is related?

TorreZuk commented 8 months ago

Okay yes looks like the same issue, guess the dependency msys2 download is too stale if they remove them after 1.75 years after any update. That isn't very nice if you want something to keep building for 2 years at least. You can try later release tags of vcpkg but I would start moving forward from 2022.05.10 as new dependencies could also break things. Let us know and I'll look too for a newer candidate.

TorreZuk commented 8 months ago

Bad news is that the first vcpkg release that fixes the gtest to use a still available mingw package is 2023.06.20 but that breaks backward compatibility in msgpack as the new versions change target names. So it looks like manual installation of components by version will be required, the required components to install are all listed in rdeps.xml so just moving gtest forward is what I'll try next. vcpkg releases or commits won't imply any build reproducibility if distributions availability may drop anytime after.

TorreZuk commented 8 months ago

So to start with commit of 2022.05.10 and add new gtest only to avoid mingw change you can use a local file vcpkg.json with contents: { "dependencies": [ "gtest", "msgpack", "openblas" ], "builtin-baseline": "14e7bb4ae24616ec54ff6b2f6ef4e8659434ea44", "overrides": [ { "name": "gtest", "version": "1.13.0" } ] } So running vcpkg at location of this new file builds the deps (e.g. root dir). I tested and works with VS 17.8.2 and building 5.5 branch: %VCPKG_PATH%\vcpkg --triplet x64-windows install Then the same as before to build python rmake.py -c -a ... I need to look into how to patch this old release as the branch is normally frozen

sorasoras commented 8 months ago

So to start with commit of 2022.05.10 and add new gtest only to avoid mingw change you can use a local file vcpkg.json with contents: { "dependencies": [ "gtest", "msgpack", "openblas" ], "builtin-baseline": "14e7bb4ae24616ec54ff6b2f6ef4e8659434ea44", "overrides": [ { "name": "gtest", "version": "1.13.0" } ] } So running vcpkg at location of this new file builds the deps (e.g. root dir). I tested and works with VS 17.8.2 and building 5.5 branch: %VCPKG_PATH%\vcpkg --triplet x64-windows install Then the same as before to build python rmake.py -c -a ... I need to look into how to patch this old release as the branch is normally frozen

Errors occurred while parsing W:\git\vcpkg\vcpkg.json $ (a manifest): missing required field 'name' (an identifier) $ (a manifest): expected a versioning field (example: version-string) See https://github.com/Microsoft/vcpkg/tree/master/docs/users/manifests.md for more information.

vcpkg.json am I missing something in vcpkg.json?

Nevermind, I have to use latest ver vcpkg to load this vcpkg.

mononoSaya commented 8 months ago

how shuold i do？ I has already been done the vcpkg.json. But run python rdeps.py are identical errors.

sorasoras commented 8 months ago

So to start with commit of 2022.05.10 and add new gtest only to avoid mingw change you can use a local file vcpkg.json with contents: { "dependencies": [ "gtest", "msgpack", "openblas" ], "builtin-baseline": "14e7bb4ae24616ec54ff6b2f6ef4e8659434ea44", "overrides": [ { "name": "gtest", "version": "1.13.0" } ] } So running vcpkg at location of this new file builds the deps (e.g. root dir). I tested and works with VS 17.8.2 and building 5.5 branch: %VCPKG_PATH%\vcpkg --triplet x64-windows install Then the same as before to build python rmake.py -c -a ... I need to look into how to patch this old release as the branch is normally frozen logs2.txt

I make the change with the json you provide, it does not seems to resolve the issue

https://github.com/microsoft/vcpkg/issues/35516 This issue I look up kind of similar to this one.

mononoSaya commented 8 months ago

rocblas.txt That is log. ..

TorreZuk commented 8 months ago

Sorry that json must have been run against vckpg 2023.06.23 on my workstation, they must have changed the parser. I'll see if I can find a json for the original branch vcpkg 2022.05.10 version that you can you try, or clone 2023.06.23 and retry.
You can manually install the 3 dependencies and point to them via CMAKE_PREFIX_PATH, or try vcpkg install with what I tested with your vcpkg verison. These are what get found by json and 06.23:

gtest:x86-windows -> 1.13.0 -- C:\develop\github\vcpkg\buildtrees\versioning_\versions\gtest\3f70f42192aacc5f8242c14d959d02a802f7559c
msgpack:x86-windows -> 3.3.0#2 -- C:\develop\github\vcpkg\buildtrees\versioning_\versions\msgpack\33927863a8eae2ea7c816389e7ef3dbdcc00382a
openblas:x86-windows -> 0.3.20 -- C:\develop\github\vcpkg\buildtrees\versioning_\versions\openblas\abd8d8b0d0286d963399d79e3e8b6f7f8e12117b

TorreZuk commented 8 months ago

Okay with 2022.05.10 it appears it doesn't accept the newer json package overrides (requires other tags as well to parse but the newer dependency versions aren't allowed) so please try that json with 2023.06.23 or later vcpkg commit and let me know how it goes. I don't think we can convince mingw to restore the older versions so seems like we will have to give updates when older versions drop off availability.

sorasoras commented 8 months ago

Okay with 2022.05.10 it appears it doesn't accept the newer json package overrides (requires other tags as well to parse but the newer dependency versions aren't allowed) so please try that json with 2023.06.23 or later vcpkg commit and let me know how it goes. I don't think we can convince mingw to restore the older versions so seems like we will have to give updates when older versions drop off availability.

I was able to download all stuff except "mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst" log3.txt I found the package via google but I don't how to manually load onto pkg. vckpg 2023.06.23 no longer available so i use 23/6/15

https://mirrors.piconets.webwerks.in/msys2-mirror/mingw/i686/mingw-w64-i686-crt-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst

mononoSaya commented 8 months ago

This is my version E:\llama.cpp\Rocm\rocBLAS>vcpkg version vcpkg package management program version 2023-09-15-ac02a9f660977426b8ec6392919fbb1d51b10998

TorreZuk commented 8 months ago

@mononoSaya okay is that is from vcpkg-tool right? 9-15 is likely too new so requires msgpack changes that are incompatibl with Tensile used in rocBLAS in 5.5 Probably best if we stick with the vcpkg repo, not the tool vcpkg-tool repo to keep it simple, so this is what should still exist there: git clone -b 2023.06.20 https://github.com/microsoft/vcpkg Then bootstrap that one as per bootstrap-vcpkg.bat and set env VCPKG_PATH to where you installed. Putting the json above in rocblas root directory that then you should get the required versions with

%VCPKG_PATH%\vcpkg --triplet x64-windows install

Otherwise you just need to use manual methods to install the versions listed earlier. The overriding change is client testing for gtest so doesn't change the rocblas and tensile library build.

sorasoras commented 8 months ago

@mononoSaya okay is that is from vcpkg-tool right? 9-15 is likely too new so requires msgpack changes that are incompatibl with Tensile used in rocBLAS in 5.5 Probably best if we stick with the vcpkg repo, not the tool vcpkg-tool repo to keep it simple, so this is what should still exist there: git clone -b 2023.06.20 https://github.com/microsoft/vcpkg Then bootstrap that one as per bootstrap-vcpkg.bat and set env VCPKG_PATH to where you installed. Putting the json above in rocblas root directory that then you should get the required versions with

%VCPKG_PATH%\vcpkg --triplet x64-windows install

Otherwise you just need to use manual methods to install the versions listed earlier. The overriding change is client testing for gtest so doesn't change the rocblas and tensile library build.

Thanks, I was able pass all deploy and compiled .

sorasoras commented 8 months ago

logfailed.txt I encounter another bug I think. I was able to get one compile success for one gpu arch. Then, I try to target another arch, but i encounter another bug, but
"CMake Error at C:/vcpkg/scripts/buildsystems/vcpkg.cmake:853 (_find_package): By not providing "Findmsgpack.cmake" in CMAKE_MODULE_PATH this project has asked CMake to find a package configuration file provided by "msgpack", but CMake did not find one.

Could not find a package configuration file provided by "msgpack" with any of the following names:

msgpackConfig.cmake
msgpack-config.cmake

Add the installation prefix of "msgpack" to CMAKE_PREFIX_PATH or set "msgpack_DIR" to a directory containing one of the above files. If "msgpack" provides a separate development package or SDK, be sure it has been installed. Call Stack (most recent call first): build/release/virtualenv/Lib/site-packages/Tensile/Source/lib/CMakeLists.txt:100 (find_package)"

thanks.

sorasoras commented 8 months ago

O nevermind, I was just not able compile for gfx1032/gfx1035 but gfx1031/gfx1010 are fine.

ekg commented 8 months ago

Will gfx1035 support ever land?

TorreZuk commented 8 months ago

Unsupported gfx will not work as GPU targets for building. The msgpack error is independent of this and due to a breaking change in newer msgpack versions, the changed cmake targets will have to be accomodated by fixes in our (Tensile https://github.com/ROCm/Tensile/pull/1836) CMake code in later releases. It can be avoided with the custom listed dependency versions provided earlier (and specific vcpkg version). While this is closed I will still try and determine how to provide dependency patches so that rdeps.py can be used once again for earlier and future releases.

@ekg while I agree with the request to support more gfx I can not say if direct support will ever come. Only on Linux you can try to use an override to make the gfx1035 report itself as the supported gfx1030 using environment variable, HSA_OVERRIDE_GFX_VERSION=10.3.0
While this may work it is not officially supported, Please add your voice to https://github.com/ROCm/ROCm/issues/2654 as this issue is a building only topic for dependencies.

cgmb commented 8 months ago

Will gfx1035 support ever land?

On Ubuntu 23.10 or Debian Unstable, you can install librocblas-dev to get a version of rocBLAS that has been tested on gfx1035. Those results suggest that Torre's HSA_OVERRIDE_GFX_VERSION=10.3.0 recommendation is likely to work for users of the AMD binary packages and other Linux distributions. Unfortunately, the methods used to enable gfx1035 on Linux are not applicable to Windows.

patientx commented 3 months ago

Download manually from here and put inside to downloads folder inside vcpkg folder. It works without any other tricks. (both or one of these files is giving errors : msys-mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst , msys-mingw-w64-i686-pkg-config-0.29.2-3-any.pkg.tar.zst) https://sandwich.sh/archive/other/pineapple-src/externals/vcpkg/downloads/index.html

ROCm / rocBLAS

[Bug]: rdeps.py dont seems work anymore. #1375

Describe the bug

To Reproduce