search

JuliaDSP / MFCC.jl

Mel Frequency Cepstral Coefficients calculation for Julia
BSD 2-Clause "Simplified" License
33 stars 18 forks source link

Dependency hell for CUDA.jl #27

Closed jhigginbotham64 closed 1 year ago

jhigginbotham64 commented 2 years ago

v0.3.1 is only compatible with v2.6.3 of CUDA.jl (currently on v3.5.0). This breaks audio processing projects that want to use it with the latest version of Flux (which uses the latest CUDA.jl).

davidavdav commented 2 years ago

Hi, I feel your pain. Dependency hell is not a nice place to visit, or be stuck in for a longer period.

You have to give me a little more context—how is MFCC not compatible with CUDA? Is it through dependencies that are incompatible with CUDA?

racoiaws commented 2 years ago

Hello!

I have also encountered this issue while trying to run https://github.com/FluxML/model-zoo/tree/master/contrib/audio/speech-blstm.

Setup is Julia 1.7.1 on Ubuntu 20.04.2 x86_64

Minimal example:

  1. Create a new Julia env
  2. Add Flux and CUDA
  3. Add MFCC
  4. Notice the downgrades, also Flux and CUDA fail to precompile
Logs

``` ~/julia-sandbox> julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.7.1 (2021-12-22) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | (@v1.7) pkg> activate . Activating new project at `~/julia-sandbox` (julia-sandbox) pkg> add Flux, CUDA Updating registry at `~/.julia/registries/General.toml` Resolving package versions... Updating `~/julia-sandbox/Project.toml` [052768ef] + CUDA v3.6.4 [587475ba] + Flux v0.12.8 Updating `~/julia-sandbox/Manifest.toml` [621f4979] + AbstractFFTs v1.1.0 [1520ce14] + AbstractTrees v0.3.4 [79e6a3ab] + Adapt v3.3.3 [4fba245c] + ArrayInterface v3.2.2 [ab4f0b2a] + BFloat16s v0.2.0 [fa961155] + CEnum v0.4.1 [052768ef] + CUDA v3.6.4 [082447d4] + ChainRules v1.23.0 [d360d2e6] + ChainRulesCore v1.11.6 [9e997f8a] + ChangesOfVariables v0.1.2 [944b1d66] + CodecZlib v0.7.0 [3da002f7] + ColorTypes v0.11.0 [5ae59095] + Colors v0.12.8 [bbf7d656] + CommonSubexpressions v0.3.0 [34da2185] + Compat v3.41.0 [9a962f9c] + DataAPI v1.9.0 [864edb3b] + DataStructures v0.18.11 [163ba53b] + DiffResults v1.0.3 [b552c78f] + DiffRules v1.9.0 [ffbed154] + DocStringExtensions v0.8.6 [e2ba6199] + ExprTools v0.1.7 [1a297f60] + FillArrays v0.12.7 [53c48c17] + FixedPointNumbers v0.8.4 [587475ba] + Flux v0.12.8 [f6369f11] + ForwardDiff v0.10.25 [d9f16b24] + Functors v0.2.7 [0c68f7d7] + GPUArrays v8.1.3 [61eb1bfa] + GPUCompiler v0.13.10 [7869d1d1] + IRTools v0.4.4 [615f187c] + IfElse v0.1.1 [3587e190] + InverseFunctions v0.1.2 [92d709cd] + IrrationalConstants v0.1.1 [692b3bcd] + JLLWrappers v1.4.0 [e5e0dc1b] + Juno v0.8.4 [929cbde3] + LLVM v4.7.1 [2ab3a3ac] + LogExpFunctions v0.3.6 [1914dd2f] + MacroTools v0.5.9 [e89f7d12] + Media v0.5.0 [e1d29d7a] + Missings v1.0.2 [872c559c] + NNlib v0.7.34 [a00861dc] + NNlibCUDA v0.1.11 [77ba4419] + NaNMath v0.3.6 [bac558e1] + OrderedCollections v1.4.1 [21216c6a] + Preferences v1.2.3 [74087812] + Random123 v1.4.2 [e6cf234a] + RandomNumbers v1.5.3 [c1ae055f] + RealDot v0.1.0 [189a3867] + Reexport v1.2.2 [ae029012] + Requires v1.3.0 [a2af1166] + SortingAlgorithms v1.0.1 [276daf66] + SpecialFunctions v2.0.0 [aedffcd0] + Static v0.4.1 [90137ffa] + StaticArrays v1.3.2 [82ae8749] + StatsAPI v1.2.0 [2913bbd2] + StatsBase v0.33.14 [a759f4b9] + TimerOutputs v0.5.14 [3bb67fe8] + TranscodingStreams v0.9.6 [a5390f91] + ZipFile v0.9.4 [e88e6eb3] + Zygote v0.6.33 [700de1a5] + ZygoteRules v0.2.2 [dad2f222] + LLVMExtra_jll v0.0.13+1 [efe28fd5] + OpenSpecFun_jll v0.5.5+0 [0dad84c5] + ArgTools [56f22d72] + Artifacts [2a0f44e3] + Base64 [ade2ca70] + Dates [8bb1440f] + DelimitedFiles [8ba89e20] + Distributed [f43a241f] + Downloads [b77e0a4c] + InteractiveUtils [4af54fe1] + LazyArtifacts [b27032c2] + LibCURL [76f85450] + LibGit2 [8f399da3] + Libdl [37e2e46d] + LinearAlgebra [56ddb016] + Logging [d6f4376e] + Markdown [a63ad114] + Mmap [ca575930] + NetworkOptions [44cfe95a] + Pkg [de0858da] + Printf [9abbd945] + Profile [3fa0cd96] + REPL [9a3f8284] + Random [ea8e919c] + SHA [9e88b42a] + Serialization [1a1011a3] + SharedArrays [6462fe0b] + Sockets [2f01184e] + SparseArrays [10745b16] + Statistics [fa267f1f] + TOML [a4e569a6] + Tar [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode [e66e0078] + CompilerSupportLibraries_jll [deac9b47] + LibCURL_jll [29816b5a] + LibSSH2_jll [c8ffd9c3] + MbedTLS_jll [14a3606d] + MozillaCACerts_jll [4536629a] + OpenBLAS_jll [05823500] + OpenLibm_jll [83775a58] + Zlib_jll [8e850b90] + libblastrampoline_jll [8e850ede] + nghttp2_jll [3f19e933] + p7zip_jll (julia-sandbox) pkg> add MFCC Resolving package versions... Updating `~/julia-sandbox/Project.toml` [052768ef] ↓ CUDA v3.6.4 ⇒ v2.6.3 [587475ba] ↓ Flux v0.12.8 ⇒ v0.12.1 [ca7b5df7] + MFCC v0.3.1 Updating `~/julia-sandbox/Manifest.toml` [621f4979] ↓ AbstractFFTs v1.1.0 ⇒ v1.0.1 [4fba245c] - ArrayInterface v3.2.2 [ab4f0b2a] ↓ BFloat16s v0.2.0 ⇒ v0.1.0 [a74b3585] + Blosc v0.7.2 [052768ef] ↓ CUDA v3.6.4 ⇒ v2.6.3 [082447d4] ↓ ChainRules v1.23.0 ⇒ v0.8.25 [d360d2e6] ↓ ChainRulesCore v1.11.6 ⇒ v0.10.13 [9e997f8a] - ChangesOfVariables v0.1.2 [717857b8] + DSP v0.6.10 [b552c78f] ↓ DiffRules v1.9.0 ⇒ v1.5.0 [7a1cc6ca] + FFTW v1.4.5 [5789e2e9] + FileIO v1.12.0 [587475ba] ↓ Flux v0.12.8 ⇒ v0.12.1 [0c68f7d7] ↓ GPUArrays v8.1.3 ⇒ v6.4.1 [61eb1bfa] ↓ GPUCompiler v0.13.10 ⇒ v0.10.0 [f67ccb44] + HDF5 v0.13.7 [615f187c] - IfElse v0.1.1 [842dd82b] + InlineStrings v1.1.1 [d8418881] + Intervals v1.5.0 [3587e190] - InverseFunctions v0.1.2 [c8e1da08] + IterTools v1.4.0 [929cbde3] ↓ LLVM v4.7.1 ⇒ v3.9.0 [2ab3a3ac] ↓ LogExpFunctions v0.3.6 ⇒ v0.3.0 [ca7b5df7] + MFCC v0.3.1 [c03570c3] + Memoize v0.4.4 [78c3b35d] + Mocking v0.7.3 [a00861dc] - NNlibCUDA v0.1.11 [6fe1bfb0] + OffsetArrays v1.10.8 [69de0a69] + Parsers v2.2.0 [f27b6e38] + Polynomials v1.2.1 [74087812] - Random123 v1.4.2 [e6cf234a] - RandomNumbers v1.5.3 [c1ae055f] - RealDot v0.1.0 [3cdcf5f2] + RecipesBase v1.2.1 [6c6a2e73] + Scratch v1.1.0 [276daf66] ↓ SpecialFunctions v2.0.0 ⇒ v0.10.3 [aedffcd0] - Static v0.4.1 [f269a46b] + TimeZones v1.7.1 [8149f6b0] + WAV v1.2.0 [e88e6eb3] ↓ Zygote v0.6.33 ⇒ v0.6.17 [0b7ba130] + Blosc_jll v1.21.1+0 [f5851436] + FFTW_jll v3.3.10+0 [0234f1f7] + HDF5_jll v1.10.5+7 [1d5cc7b8] + IntelOpenMP_jll v2018.0.3+2 [dad2f222] - LLVMExtra_jll v0.0.13+1 [5ced341a] + Lz4_jll v1.9.3+0 [856f044c] + MKL_jll v2021.1.1+2 [3161d3a3] + Zstd_jll v1.5.0+0 [05823500] - OpenLibm_jll Precompiling project... ✗ CUDA ✗ Flux 0 dependencies successfully precompiled in 5 seconds (81 already precompiled) 2 dependencies errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the packages ``` ``` ~/julia-sandbox> cat Project.toml [deps] CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba" Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c" MFCC = "ca7b5df7-6146-5dcc-89ec-36256279a339" ```

Hope this is helpful

racoiaws commented 2 years ago

We can reveal the culprit by explicitly requiring recent LLVM and Flux

(julia-sandbox) pkg> add LLVM@4, Flux@0.12, MFCC
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package CUDA [052768ef]:
 CUDA [052768ef] log:
 ├─possible versions are: 0.1.0-3.6.4 or uninstalled
 ├─restricted by julia compatibility requirements to versions: [2.3.0, 2.5.0-3.6.4] or uninstalled
 ├─restricted by compatibility requirements with Flux [587475ba] to versions: 2.1.0-3.6.4, leaving only versions: [2.3.0, 2.5.0-3.6.4]
 │ └─Flux [587475ba] log:
 │   ├─possible versions are: 0.4.1-0.12.8 or uninstalled
 │   └─restricted to versions 0.12 by an explicit requirement, leaving only versions 0.12.0-0.12.8
 ├─restricted by compatibility requirements with LLVM [929cbde3] to versions: 3.3.3-3.6.4 or uninstalled, leaving only versions: 3.3.3-3.6.4
 │ └─LLVM [929cbde3] log:
 │   ├─possible versions are: 0.9.0-4.7.1 or uninstalled
 │   └─restricted to versions 4.7.1 by an explicit requirement, leaving only versions 4.7.1
 └─restricted by compatibility requirements with SpecialFunctions [276daf66] to versions: 0.1.0-2.6.3 or uninstalled — no versions left
   └─SpecialFunctions [276daf66] log:
     ├─possible versions are: 0.7.0-2.0.0 or uninstalled
     └─restricted by compatibility requirements with MFCC [ca7b5df7] to versions: 0.7.0-0.10.3
       └─MFCC [ca7b5df7] log:
         ├─possible versions are: 0.3.1 or uninstalled
         └─restricted to versions 0.3.1 by an explicit requirement, leaving only versions 0.3.1

MFCC requires an old version of SpecialFunctions (0.7.0-0.10.3), which is only compatible with old CUDA (2.6.3), which is only compatible with old LLVM (3.3.3-3.6.4)

davidavdav commented 2 years ago

Hello,

I am sorry to hear that this happened to you. Any clue what special function MFCC needs? It is probably not such a fancy function, I am sure we can make this more modern

racoiaws commented 2 years ago

There is a comment about erfinv in the code: https://github.com/JuliaDSP/MFCC.jl/blob/c87c6ab8a6773fc7dca4a4e1cd641b9695d9e5fe/src/MFCC.jl#L16 Maybe it's the only one MFCC.jl uses, hard to tell

davidavdav commented 2 years ago

The Project.toml doesn't specify any version in dependencies, where is this coming from?

racoiaws commented 2 years ago

I think I found it. For some reason, Julia General Registry thinks that SpecialFunctions has to have version under 0.10 https://github.com/JuliaRegistries/General/blob/master/M/MFCC/Compat.toml

Current workaround

Install MFCC from git:

(sandbox) pkg> add https://github.com/JuliaDSP/MFCC.jl, HDF5@0.14

To avoid explicit HDF5 line in Project.toml, one can use branch from #25:

(sandbox) pkg> add https://github.com/maetshju/MFCC.jl#fixhdf5
davidavdav commented 1 year ago

I think I managed to update all dependencies (they weren't so strong anyways), basically because I figured out how to do it:

(@v1.8) pkg> activate .
(@v1.8) pkg> update
## try new version of dependency in Project.toml
(@v1.8) pkg> update
(@v1.8) pkg> test

fix problems in the code along the way, and repeat.

Can you maybe check if v0.3.3 still causes these issues for you?