natowi
commented
9 months ago Does ZLUDA work when using the GPU in FeatureExtraction? (enable sift and disable the advanced setting: "force cpu")
Is the same error thrown?
Closed harakiru closed 6 months ago
natowi
commented
9 months ago Does ZLUDA work when using the GPU in FeatureExtraction? (enable sift and disable the advanced setting: "force cpu")
Is the same error thrown?
harakiru
commented
9 months ago I get a different error this time
Choosing device 0: AMD Radeon RX 6800 XT [ZLUDA]
terminate called after throwing an instance of 'std::runtime_error'
what(): /tmp/AliceVisionDeps_build/popsift/src/popsift/gauss_filter.cu:245
cudaMemcpyToSymbol failed for Gauss kernel initialization: operation not supported
However FeatureExtraction can be done on the CPU, and with a powerful system speed is not much of an issue. The real blocker is the DepthMap node, which has to use CUDA.
the-mush
commented
9 months ago just tried it out and I have the exact same problem, also with a 6800XT :(
20943204920434
commented
9 months ago It looks like a function cudaMemcpyToSymbol() is used here. Obviously it isn't supported. As a solution I can suggest making a patch to substitute the function with cudaMemcpy(), which is similar according to this. Actually, there is a problem, we don't have a clear API compatibility paper, so we cannot say if some function is supported or not.
P.S. I don't know even basic cuda, so I may be wrong sometimes, feel free to correct me.
LeMoonStar
commented
9 months ago I can reproduce the issue using an Radeon RX 7700 xt (+ AMD Ryzen 7600 on Fedora Linux 39)
I also noticed that the first warning is slightly different for me: (error code: 999) cudaErrorUnknown insead of (error code: 801) cudaErrorNotSupported the original issue had.
$ LD_LIBRARY_PATH='/home/user/Downloads/zluda-3-linux/zluda' ROCR_VISIBLE_DEVICES="GPU-****" '/home/user/Downloads/Meshroom-2023.3.0-linux/Meshroom-2023.3.0/Meshroom'
[...]
Hardware :
Detected core count : 12
OpenMP will use 12 cores
Detected available memory : 22386 Mo
[18:49:20.033367][warning] Cannot get available memory information for CUDA gpu device 0:
(error code: 999) cudaErrorUnknown
[18:49:19.830787][warning] CUDA-Enabled GPU.
Device information:
- id: 0
- name: AMD Radeon RX 7700 XT [ZLUDA]
- compute capability: 8.8
- clock frequency (kHz): 2276000
- total device memory: 12272 MB
- device memory available: 0 MB
- per-block shared memory: 65536
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 2048
- max block sizes: {1024,1024,1024}
- max grid sizes: {2147483647,65536,65536}
- max 2D array texture: {16384,16384}
- max 3D array texture: {16384,16384,8192}
- max 2D linear texture: {0,0,0}
- max 2D layered texture: {16384,16384,65535}
- number of SM(x)s: 27
- registers per SM(x): 65536
- registers per block: 65536
- concurrent kernels: yes
- mapping host memory: yes
- unified addressing: yes
- texture alignment: 256 byte
- pitch alignment: 256 byte
CUDA build version: 11.3
[18:49:20.033493][info] Supported CUDA-Enabled GPU detected.
[18:49:20.062344][info] Found 1 image dimension(s):
[18:49:20.062372][info] - [4032x3024]
[18:49:20.072950][info] Overall maximum dimension: [4032x3024]
[18:49:20.072990][info] Tiling information:
- parameters:
- buffer width: 1024 px
- buffer height: 1024 px
- padding: 64 px
- maximum downscale: 4
- maximum image width: 2016 px
- maximum image height: 1512 px
- maximum effective tile width: 896 px
- maximum effective tile height: 896 px
- # tiles on X-side: 3
- # tiles on Y-side: 2
- effective tile width: 672 px
- effective tile height: 756 px
- tile list:
- tile (1/6) size: 736x820 px, roi: [x: 0-736, y: 0-820]
- tile (2/6) size: 736x756 px, roi: [x: 0-736, y: 756-1512]
- tile (3/6) size: 736x820 px, roi: [x: 672-1408, y: 0-820]
- tile (4/6) size: 736x756 px, roi: [x: 672-1408, y: 756-1512]
- tile (5/6) size: 672x820 px, roi: [x: 1344-2016, y: 0-820]
- tile (6/6) size: 672x756 px, roi: [x: 1344-2016, y: 756-1512]
[18:49:20.073007][info] SGM parameters:
- scale: 2
- stepXY: 2
[18:49:20.073025][info] Refine parameters:
- scale: 1
- stepXY: 1
[18:49:20.073049][info] Number of GPU devices: 1, number of CPU threads: 12
================================================================================
====================== Command line failed with an error =======================
Could not allocate pinned host memory in /opt/AliceVision_git/src/aliceVision/depthMap/cuda/host/memory.hpp:395, unknown error: unknown error
================================================================================EDIT: The final error message is different as well, I get an unknown error: unknown error, not the operation not supported: operation not supported shown in the original issue.
20943204920434
commented
8 months ago I've started making a troubleshooting info:
Then i've tried running meshroom with ltrace, but the program freezes and doesn't respond at all. There is an output in my terminal, so i'll attach it too. Suddenly, it started working. I'm attaching the logs.
So we get a bunch of 500 (CUDA_ERROR_NOT_FOUND) errors from cuGetProcAddress(). I would like to know what does that mean in this context. Maybe @vosen as a mantainer can help?
P.S. I've used monstree-mini6 dataset to test it.
vosen
commented
8 months ago cuGetProcAddress returning 500 is benign pretty much always. There's a handful of CUDA driver functions which are either too esoteric (nvsci) or just difficult to support cleanly (OS-specific interop: DirectX, EGL) where ZLUDA does not even advertise possibility of supporting them.
It's not int the logs, but I infer that ZLUDA fails to compile a PTX module at startup, which usually is not hard to fix. I'll have a look
vosen
commented
7 months ago I was wrong, this was suffering.
Meshroom seems to work now. Since it is a rather big change and I don't really know how to use Meshroom I'd like someone who is interested in this issue to give it a try:
src/cmake/Dependencies.cmake change
ExternalProject_Add(${CCTAG_TARGET}
GIT_REPOSITORY https://github.com/alicevision/CCTag
GIT_TAG v1.0.3to
ExternalProject_Add(${CCTAG_TARGET}
GIT_REPOSITORY https://github.com/alicevision/CCTag
GIT_TAG ee7a02dc329efe1e43a4afad36cb0e37644ba957I can provide the binaries, but I'd prefer if someone built from scratch to double check everything. If you have questions please ask.
natowi
commented
7 months ago It would be great if you could share the binaries for testing :)
vosen
commented
7 months ago ZLUDA: https://files.catbox.moe/pba9kq.zip AliceVision: https://gofile.io/d/ITFhF0
gerberger
commented
6 months ago I I've tried running the full and mini6 image set from https://github.com/alicevision/dataset_monstree, using the binaries shared by Natowi at https://github.com/alicevision/Meshroom/issues/595, but when it gets to the DepthMap scene, it fails, with nothing in the log. I've tried with the log verbosity set to various levels. cmd prompt has this: ERROR:root:Error on node computation: Error on node "DepthMap_1(0)": Log:
WARNING:root:Downgrade status on node "DepthMapFilter_1(0)" from Status.SUBMITTED to Status.NONE WARNING:root:Downgrade status on node "Meshing_1" from Status.SUBMITTED to Status.NONE WARNING:root:Downgrade status on node "MeshFiltering_1" from Status.SUBMITTED to Status.NONE WARNING:root:Downgrade status on node "Texturing_1" from Status.SUBMITTED to Status.NONE
I'm running an AMD Ryzen 5 3600, with an RX 5700XT, on Windows 11 23H2, with Adrenalin 24.4.1
edit: Feature extraction using sift, and unticking force cpu works fine
aesxsc
commented
5 months ago I did everything @gerberger did, and it still fails with the same error. Also this error pops up
Adrenalin 24.5.1, RX 5500XT
bonnyr
commented
3 months ago I can confirm this is still happening although I do not get that popup. Same output, no logs... All stages until that point run fine.
This is with a scene using my own images, likely not to be producing useful model, but not the known image set mentioned above.
The full, mini3 and mini6 monster scene fails in the same way at the same point
Adrenalin 24.6.1, RX 6750 XT
Tried to run the same command directly under zluda, but no dice
Would like to know if there are more logs or any other way I can get more info out
dngulin
commented
2 months ago It works only on Windows, because of an underlying ROCm issue
Can you please clarify what is the ROCm issue? I wonder if it is possible to run Meshroom + ZLUDA on linux.
vosen
commented
2 months ago @dngulin for some reason AMD enabled support for mipmapped arrays (textures) on Windows ROCm, but not on Linux ROCm. Meshroom relies on this functionality
bonnyr-f5
commented
2 months ago But why is this happening on windows for me? Or is this a different issue?
@dngulin for some reason AMD enabled support for mipmapped arrays (textures) on Windows ROCm, but not on Linux ROCm. Meshroom relies on this functionality
Hiuzuki
commented
1 month ago Hello, does this method still work? I tried to use Meshroom with my RX 5500 XT using Zluda, I get an error when I get to Depthmap, it seems that Meshroom doesn't even detect the GPU, only the CPU. I installed the CUDA toolkit but it didn't work, I even tried to install the Nvidia graphics driver, but obviously it didn't work.
vosen
commented
1 month ago @aesxsc @bonnyr-f5 I guess your error probably comes from DepthMap using CUDA Runtime, but not bundling it with the app. Extracting it from CUDA installer and dropping it in a path where DepthMap looks for .dll would make the error go away ~but still won't make DepthMap work because of aforementioned ROCm Windows issue~
Edit: Sorry, I got this wrong way around. It's Linux ROCm that is broken, not Windows one
mpaterakis
commented
5 hours ago To anyone still having crashes: Use CUDA 11.0, add the installation path (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin) to your system's Path, and use this pre-compiled package by natowi.
Works on my 6750 XT.
Relevant line of code: https://github.com/alicevision/AliceVision/blob/develop/src/aliceVision/depthMap/cuda/host/memory.hpp