search

FourierFlows / FourierFlows.jl

Tools for building fast, hackable, pseudospectral partial differential equation solvers on periodic domains
https://bit.ly/FourierFlows
MIT License
207 stars 29 forks source link

FourierFlows on GPUs #277

Closed navidcy closed 3 years ago

navidcy commented 3 years ago

There is some discussion on running FourierFlows.jl on GPUs in the docs but perhaps, we should revisit it to add clarifications. I'm raising while in despair being confused and puzzled with why FourierFlows is unable to "see" the GPU sometimes on an HPC cluster.

cc @liasiegelman

glwagner commented 3 years ago

It could help to document what's going wrong with FourierFlows on the cluster here so that process is documented (and hopefully the solution).

navidcy commented 3 years ago

Yes. Just need to find some time and make sure the issue is reproducible.

glwagner commented 3 years ago

If you mean whether the same steps lead to the same behavior on the same cluster then that's important just to ensure that there's no mistake in documenting the steps.

But I don't think we need any more reproducibility (eg on another cluster); any documented information at all is valuable and can be discussed.

liasiegelman commented 3 years ago

Brief summary on the skylab GPU of the NASA cluster with JULIA_CUDA_USE_BINARYBUILDER: false , CUDA toolkit 11.0.2, CUDA driver 11.0.0 NVIDIA driver 450.51.5

with julia1.5.4 and cuda2.4.3 FourierFlows can build the TwoDGrid on the GPU and GeophysicalFlows creates a problem on the GPU

with julia1.6.1 and both cuda3.1.0 and cuda2.6.3, using FourierFlows returns

┌ Debug: CUDA-enabled GPU(s) detected:
└ @ FourierFlows ~/.julia/packages/FourierFlows/e7mT6/src/FourierFlows.jl:127
┌ Debug: CuDevice(0): Tesla V100-SXM2-32GB
└ @ FourierFlows ~/.julia/packages/FourierFlows/e7mT6/src/FourierFlows.jl:129
however TwoDgrid(GPU(),10,2) returns 
ERROR: MethodError: no method matching /(::Int64, ::GPU)
Closest candidates are:
  /(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, ::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:93
  /(::Union{Integer, Complex{var"#s79"} where var"#s79"<:Union{Integer, Rational}}, ::Rational) at rational.jl:322
  /(::Union{Int16, Int32, Int64, Int8, UInt16, UInt32, UInt64, UInt8}, ::BigInt) at gmp.jl:544
  ...
Stacktrace:
 [1] TwoDGrid(nx::GPU, Lx::Int64, ny::Int64, Ly::Int64; x0::Float64, y0::Float64, nthreads::Int64, effort::UInt32, T::Type{Float64}, dealias::Float64, ArrayType::Type)
   @ FourierFlows ~/.julia/packages/FourierFlows/e7mT6/src/domains.jl:126
 [2] TwoDGrid(nx::GPU, Lx::Int64, ny::Int64, Ly::Int64) (repeats 2 times)
   @ FourierFlows ~/.julia/packages/FourierFlows/e7mT6/src/domains.jl:126
 [3] top-level scope
   @ REPL[19]:1

One idea may be to check the init function https://github.com/FourierFlows/FourierFlows.jl/blob/844684479013516ceb4a2ddd8909470f4d60973f/src/FourierFlows.jl#L119

The error reads like https://github.com/FourierFlows/FourierFlows.jl/blob/bb5ff153f92dc96cf53821df0be5f3d08992bfd6/src/FourierFlows.jl#L116 Was never called in FourierFlows.jl and therefore there TwoGrid overload https://github.com/FourierFlows/FourierFlows.jl/blob/bb5ff153f92dc96cf53821df0be5f3d08992bfd6/src/CuFourierFlows.jl#L6 Is not there!

navidcy commented 3 years ago

To add to that, seems like on the NASA Pleiades Cluster FourierFlows.jl v0.6.18 worked as expected with CUDA.jl v2.4.3 and Julia 1.5.4. However, with Julia v1.6.1, both with CUDA.jl v2.6.3 and v.3.1.0, while CUDA.has_cuda() returns true, FourierFlows.jl does not "see" the GPU and does not overloads the GPU-related methods for grids, etc.

But I don't think it's CUDA v3 that creates issues because the tests on FourierFlows.jl v0.6.18 seem to run OK on Julia v1.6; see buildkite log

navidcy commented 3 years ago

OK, I can confirm that I get the same behavior on Australia's HPC. Weird.

(base) [nc3020@gadi-gpu-v100-0023 testfolder]$ julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.1 (2021-04-23)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

julia> using FourierFlows
[ Info: FourierFlows will use 48 threads

julia> TwoDGrid(GPU(), 10, 2.0)
ERROR: MethodError: no method matching /(::Int64, ::GPU)
Closest candidates are:
  /(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, ::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:93
  /(::Union{Integer, Complex{var"#s79"} where var"#s79"<:Union{Integer, Rational}}, ::Rational) at rational.jl:322
  /(::Union{Int16, Int32, Int64, Int8, UInt16, UInt32, UInt64, UInt8}, ::BigInt) at gmp.jl:544
  ...
Stacktrace:
 [1] TwoDGrid(nx::GPU, Lx::Int64, ny::Float64, Ly::Int64; x0::Float64, y0::Float64, nthreads::Int64, effort::UInt32, T::Type{Float64}, dealias::Float64, ArrayType::Type)
   @ FourierFlows /g/data/v45/nc3020/.julia/packages/FourierFlows/e7mT6/src/domains.jl:126
 [2] TwoDGrid(nx::GPU, Lx::Int64, ny::Float64, Ly::Int64) (repeats 2 times)
   @ FourierFlows /g/data/v45/nc3020/.julia/packages/FourierFlows/e7mT6/src/domains.jl:126
 [3] top-level scope
   @ REPL[2]:1
 [4] top-level scope
   @ /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/src/initialization.jl:81

julia>

shell> nvidia-smi
Sun May  9 07:45:48 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:B1:00.0 Off |                    0 |
| N/A   31C    P0    56W / 300W |    308MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   1483166      C   ...ta/v45/nc3020/julia/julia      305MiB |
+-----------------------------------------------------------------------------+

(testfolder) pkg> st
      Status `/g/data/v45/nc3020/testfolder/Project.toml`
  [2aec4490] FourierFlows v0.6.18
  [44ee3b1c] GeophysicalFlows v0.12.2

shell> cat Manifest.toml
# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "485ee0867925449198280d4af84bdb46a2a404d0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.0.1"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "f1b523983a58802c4695851926203b36e28f09db"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.0"

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[AxisAlgorithms]]
deps = ["LinearAlgebra", "Random", "SparseArrays", "WoodburyMatrices"]
git-tree-sha1 = "a4d07a1c313392a77042855df46c5f534076fab9"
uuid = "13072b0f-2c55-5437-9ae7-d433b7a33950"
version = "1.0.0"

[[BFloat16s]]
deps = ["LinearAlgebra", "Test"]
git-tree-sha1 = "4af69e205efc343068dc8722b8dfec1ade89254a"
uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b"
version = "0.1.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"

[[CUDA]]
deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CompilerSupportLibraries_jll", "DataStructures", "ExprTools", "GPUArrays", "GPUCompiler", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "MacroTools", "Memoize", "Printf", "Random", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "SpecialFunctions", "Statistics", "TimerOutputs"]
git-tree-sha1 = "d4fa6486e94c4087f1d081d7be2d501a170bd51d"
uuid = "052768ef-5323-5732-b1bb-66c8b64840ba"
version = "3.1.0"

[[ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "bd0cc939d94b8bd736dce5bbbe0d635db9f94af7"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "0.9.41"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "0a817fbe51c976de090aa8c997b7b719b786118d"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.28.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "4437b64df1e0adccc3e5d1adbc3ac741095e4677"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.9"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[DocStringExtensions]]
deps = ["LibGit2", "Markdown", "Pkg", "Test"]
git-tree-sha1 = "9d4f64f79012636741cf01133158a54b24924c32"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.4"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[ExprTools]]
git-tree-sha1 = "10407a39b87f29d47ebaca8edbc75d7c302ff93e"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.3"

[[FFTW]]
deps = ["AbstractFFTs", "FFTW_jll", "LinearAlgebra", "MKL_jll", "Preferences", "Reexport"]
git-tree-sha1 = "746f68839306977040653ebbd249e39c15420b8a"
uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341"
version = "1.4.1"

[[FFTW_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "5a0d4b6a22a34d17d53543bd124f4b08ed78e8b0"
uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a"
version = "3.3.9+7"

[[FileIO]]
deps = ["Pkg", "Requires", "UUIDs"]
git-tree-sha1 = "d74fd0dae131104471c849f25ba1179fe0fadf47"
uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549"
version = "1.8.2"

[[FourierFlows]]
deps = ["CUDA", "DocStringExtensions", "FFTW", "Interpolations", "JLD2", "LinearAlgebra", "Random", "Reexport", "Statistics"]
git-tree-sha1 = "e134ecc0cc475af41ade2a337c74a5755e648337"
uuid = "2aec4490-903f-5c70-9b11-9bed06a700e1"
version = "0.6.18"

[[GPUArrays]]
deps = ["AbstractFFTs", "Adapt", "LinearAlgebra", "Printf", "Random", "Serialization", "Statistics"]
git-tree-sha1 = "711b448634c8e1c3597eca3a72aeecf0d19c7d3b"
uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7"
version = "6.4.0"

[[GPUCompiler]]
deps = ["DataStructures", "ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "Scratch", "Serialization", "TimerOutputs", "UUIDs"]
git-tree-sha1 = "42d635f6d87af125b86288df3819f805fb4d851a"
uuid = "61eb1bfa-7361-4325-ad38-22787b887f55"
version = "0.11.5"

[[GeophysicalFlows]]
deps = ["CUDA", "DocStringExtensions", "FFTW", "FourierFlows", "JLD2", "LinearAlgebra", "Random", "Reexport", "SpecialFunctions", "StaticArrays", "Statistics"]
git-tree-sha1 = "8ec20ff56f798ca37b09605f23e9be4d8b63673c"
uuid = "44ee3b1c-bc02-53fa-8355-8e347616e15e"
version = "0.12.2"

[[IntelOpenMP_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d979e54b71da82f3a65b62553da4fc3d18c9004c"
uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0"
version = "2018.0.3+2"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[Interpolations]]
deps = ["AxisAlgorithms", "LinearAlgebra", "OffsetArrays", "Random", "Ratios", "SharedArrays", "SparseArrays", "StaticArrays", "WoodburyMatrices"]
git-tree-sha1 = "1e0e51692a3a77f1eeb51bf741bdd0439ed210e7"
uuid = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59"
version = "0.13.2"

[[JLD2]]
deps = ["DataStructures", "FileIO", "MacroTools", "Mmap", "Pkg", "Printf", "Reexport", "Requires", "TranscodingStreams", "UUIDs"]
git-tree-sha1 = "c50e77bb661b0e5aea126404246b07c7937691c5"
uuid = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
version = "0.4.5"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[LLVM]]
deps = ["CEnum", "Libdl", "Printf", "Unicode"]
git-tree-sha1 = "a220efe4a6bc1c71809d002eb9ed9209ce5a86fb"
uuid = "929cbde3-209d-540e-8aea-75f648917ca0"
version = "3.7.0"

[[LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MKL_jll]]
deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"]
git-tree-sha1 = "c253236b0ed414624b083e6b72bfe891fbd2c7af"
uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7"
version = "2021.1.1+1"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "6a8a2a625ab0dea913aba95c11370589e0239ff0"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.6"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[Memoize]]
deps = ["MacroTools"]
git-tree-sha1 = "2b1dfcba103de714d31c033b5dacc2e4a12c7caa"
uuid = "c03570c3-d221-55d1-a50c-7939bbd78826"
version = "0.4.4"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[OffsetArrays]]
deps = ["Adapt"]
git-tree-sha1 = "87a728aebb76220bd72855e1c85284c5fdb9774c"
uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
version = "1.7.0"

[[OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b9b8b8ed236998f91143938a760c2112dceeb2b4"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.4+0"

[[OrderedCollections]]
git-tree-sha1 = "4fa2ba51070ec13fcc7517db714445b4ab986bdf"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.0"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "ea79e4c9077208cd3bc5d29631a26bc0cff78902"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.1"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[RandomNumbers]]
deps = ["Random", "Requires"]
git-tree-sha1 = "441e6fc35597524ada7f85e13df1f4e10137d16f"
uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143"
version = "1.4.0"

[[Ratios]]
git-tree-sha1 = "37d210f612d70f3f7d57d488cb3b6eff56ad4e41"
uuid = "c84ed2f1-dad5-54f0-aa8e-dbefe2724439"
version = "0.4.0"

[[Reexport]]
git-tree-sha1 = "57d8440b0c7d98fc4f889e478e80f268d534c9d5"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.0.0"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.1.3"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Scratch]]
deps = ["Dates"]
git-tree-sha1 = "ad4b278adb62d185bbcb6864dc24959ab0627bf6"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.0.3"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[SpecialFunctions]]
deps = ["ChainRulesCore", "OpenSpecFun_jll"]
git-tree-sha1 = "5919936c0e92cff40e57d0ddf0ceb667d42e5902"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "1.3.0"

[[StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "fb46e45ef2cade8be20bb445b3ffeca3c6d6f7d3"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.1.3"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TimerOutputs]]
deps = ["Printf"]
git-tree-sha1 = "32cdbe6cd2d214c25a0b88f985c9e0092877c236"
uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
version = "0.5.8"

[[TranscodingStreams]]
deps = ["Random", "Test"]
git-tree-sha1 = "7c53c35547de1c5b9d46a4797cf6d8253807108c"
uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
version = "0.9.5"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[WoodburyMatrices]]
deps = ["LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "59e2ad8fd1591ea019a5259bd012d7aee15f995c"
uuid = "efce3f68-66dc-5838-9240-27a6d6f5f9b6"
version = "0.5.3"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"
navidcy commented 3 years ago

What's also confusing is that https://github.com/FourierFlows/FourierFlows.jl/blob/844684479013516ceb4a2ddd8909470f4d60973f/src/FourierFlows.jl#L114-L117 is not part of the __init()__ function. So it should be executed every time CUDA.has_cuda() is true, no?

liasiegelman commented 3 years ago

I agree, maybe we could add a debug string here to monitor if it is loaded or not ?

navidcy commented 3 years ago

I agree, maybe we could add a debug string here to monitor if it is loaded or not ?

278

navidcy commented 3 years ago

@liasiegelman, did you try on bruno with Julia 1.6 to call using FourierFlows? Is GPU functionality available?

liasiegelman commented 3 years ago

@navidcy I didn't. currently running julia1.5.3 and cuda v2.3.0 on bruno, can try on monday if we don't figure it out by then

navidcy commented 3 years ago

Seems that include("CuFourierFlows.jl") is never called... why? I even deleted the.julia` folder whatsoever to make sure nothing clashes...

(base) [nc3020@gadi-gpu-v100-0023 testfolder2]$ JULIA_DEBUG=all julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.1 (2021-04-23)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |

(testfolder2) pkg> st
      Status `/g/data/v45/nc3020/testfolder2/Project.toml`
  [2aec4490] FourierFlows v0.6.18 `https://github.com/FourierFlows/FourierFlows.jl.git#master`

julia> using FourierFlows
┌ Debug: Creating new cache for "/g/data/v45/nc3020/testfolder2/Manifest.toml"
└ @ Base loading.jl:225
┌ Debug: Creating new cache for "/g/data/v45/nc3020/julia/usr/share/julia/stdlib/v1.6/CompilerSupportLibraries_jll/Project.toml"
└ @ Base loading.jl:225
┌ Debug: Using LLVM 11.0.1 at /g/data/v45/nc3020/julia/usr/bin/../lib/libLLVM-11jl.so
└ @ LLVM /g/data/v45/nc3020/.julia/packages/LLVM/hHQuD/src/LLVM.jl:85
[ Info: FourierFlows will use 48 threads
┌ Debug: Initializing CUDA driver
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/src/initialization.jl:100
┌ Debug: Trying to use local installation...
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/bindeps.jl:225
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME, CUDA_ROOT
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:401
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:119
┌ Debug: Looking for binary nvdisasm
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/bin"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:129
┌ Debug: Found binary nvdisasm at /apps/cuda/11.2.2/bin/nvdisasm
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:135
┌ Debug: Request to look for binary compute-sanitizer
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:119
┌ Debug: Looking for binary compute-sanitizer
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/bin"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:129
┌ Debug: Found binary compute-sanitizer at /apps/cuda/11.2.2/bin/compute-sanitizer
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:135
┌ Debug: CUDA toolkit identified as 11.2.2 (providing nvdisasm 11.2.152)
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:485
┌ Debug: Request to look for library cublas 11.4.1
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcublas.so.11.4.1, libcublas.so.11.4, libcublas.so.11
│   locations =
│    4-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcublas.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cusparse 11.4.1
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcusparse.so.11.4.1, libcusparse.so.11.4, libcusparse.so.11
│   locations =
│    4-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcusparse.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cusolver 11.1.0
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcusolver.so.11.1.0, libcusolver.so.11.1, libcusolver.so.11
│   locations =
│    4-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcusolver.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cufft 10.4.1
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcufft.so.10.4.1, libcufft.so.10.4, libcufft.so.10
│   locations =
│    4-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcufft.so.10 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library curand 10.2.3
│   locations =
│    1-element Vector{String}:
│     "/apps/cuda/11.2.2"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcurand.so.10.2.3, libcurand.so.10.2, libcurand.so.10
│   locations =
│    4-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcurand.so.10 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cupti 2020.3.1
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcupti.so.2020.3.1
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcupti.so.2020.3.1 at /apps/cuda/11.2.2/extras/CUPTI/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library nvToolsExt 1.0.0
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libnvToolsExt.so.1.0.0, libnvToolsExt.so.1.0, libnvToolsExt.so.1
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libnvToolsExt.so.1.0.0 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for libcudadevrt
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:539
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    6-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:568
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:575
┌ Debug: Request to look for libdevice
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:506
┌ Debug: Look for libdevice
│   locations =
│    3-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/nvvm/libdevice"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:518
┌ Debug: Found unified device library at /apps/cuda/11.2.2/nvvm/libdevice/libdevice.10.bc
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:523
┌ Debug: Request to look for library cublas 11.4.1
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcublas.so.11.4.1, libcublas.so.11.4, libcublas.so.11
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcublas.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cusparse 11.4.1
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcusparse.so.11.4.1, libcusparse.so.11.4, libcusparse.so.11
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcusparse.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cusolver 11.1.0
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcusolver.so.11.1.0, libcusolver.so.11.1, libcusolver.so.11
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcusolver.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cufft 10.4.1
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcufft.so.10.4.1, libcufft.so.10.4, libcufft.so.10
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcufft.so.10 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library curand 10.2.3
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcurand.so.10.2.3, libcurand.so.10.2, libcurand.so.10
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcurand.so.10 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Request to look for library cusolverMg 11.1.0
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcusolverMg.so.11.1.0, libcusolverMg.so.11.1, libcusolverMg.so.11
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Found library libcusolverMg.so.11 at /apps/cuda/11.2.2/lib64
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:108
┌ Debug: Found local CUDA 11.2.2 at /apps/cuda/11.2.2, /apps/cuda/11.2.2/extras/CUPTI
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/bindeps.jl:296
┌ Debug: Request to look for library cudnn 8.0.0
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcudnn.so.8.0.0, libcudnn.so.8.0, libcudnn.so.8
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Request to look for library cutensor 1.0.0
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcutensor.so.1.0.0, libcutensor.so.1.0, libcutensor.so.1
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Request to look for library cutensor nothing
│   locations =
│    2-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/extras/CUPTI"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:80
┌ Debug: Looking for library libcutensor.so
│   locations =
│    8-element Vector{String}:
│     "/apps/cuda/11.2.2"
│     "/apps/cuda/11.2.2/lib"
│     "/apps/cuda/11.2.2/lib64"
│     "/apps/cuda/11.2.2/libx64"
│     "/apps/cuda/11.2.2/extras/CUPTI"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib"
│     "/apps/cuda/11.2.2/extras/CUPTI/lib64"
│     "/apps/cuda/11.2.2/extras/CUPTI/libx64"
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/discovery.jl:100
┌ Debug: Toolchain with LLVM 11.0.1, CUDA driver 11.2.0 and toolkit 11.2.2 supports devices 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2, 7.5 and 8.0; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5 and 7.0
└ @ CUDA /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/deps/compatibility.jl:227
┌ Debug: CUDA-enabled GPU(s) detected:
└ @ FourierFlows /g/data/v45/nc3020/.julia/packages/FourierFlows/CPu74/src/FourierFlows.jl:128
┌ Debug: CUDA.CuDevice(0): Tesla V100-SXM2-32GB
└ @ FourierFlows /g/data/v45/nc3020/.julia/packages/FourierFlows/CPu74/src/FourierFlows.jl:130

julia> TwoDGrid(GPU(), 10, 2.0)
ERROR: MethodError: no method matching /(::Int64, ::GPU)
Closest candidates are:
  /(::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, ::Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}) at int.jl:93
  /(::Union{Integer, Complex{var"#s79"} where var"#s79"<:Union{Integer, Rational}}, ::Rational) at rational.jl:322
  /(::Union{Int16, Int32, Int64, Int8, UInt16, UInt32, UInt64, UInt8}, ::BigInt) at gmp.jl:544
  ...
Stacktrace:
 [1] TwoDGrid(nx::GPU, Lx::Int64, ny::Float64, Ly::Int64; x0::Float64, y0::Float64, nthreads::Int64, effort::UInt32, T::Type{Float64}, dealias::Float64, ArrayType::Type)
   @ FourierFlows /g/data/v45/nc3020/.julia/packages/FourierFlows/CPu74/src/domains.jl:126
 [2] TwoDGrid(nx::GPU, Lx::Int64, ny::Float64, Ly::Int64) (repeats 2 times)
   @ FourierFlows /g/data/v45/nc3020/.julia/packages/FourierFlows/CPu74/src/domains.jl:126
 [3] top-level scope
   @ REPL[3]:1
 [4] top-level scope
   @ /g/data/v45/nc3020/.julia/packages/CUDA/k52QH/src/initialization.jl:81
liasiegelman commented 3 years ago

this is weird. Would it be an option to put include("CuFourierFlows.jl") inside the init function in the if CUDA.has_cuda() ?

navidcy commented 3 years ago

I'm closing this...