search

JuliaGPU / CUDA.jl

CUDA programming in Julia.
https://juliagpu.org/cuda/
Other
1.21k stars 218 forks source link

Calls to `has_cudnn` running on wrong `CuDevice`? #978

Closed darsnack closed 3 years ago

darsnack commented 3 years ago

Describe the bug

I am developing a package that depends on Flux, and I need to run some benchmarks on the GPU. There are other users on the system (cyclops) taking up all the memory on several devices. Before doing anything else, I call

using CUDA
CUDA.device!(CUDA.CuDevice(7))

I call CUDA.memory_status() to ensure that the device I selected is not in use (it isn't). When using my package, it triggers this line in Flux which OOMs. I suspect has_cudnn is running some code on a device other than the one I selected.

To reproduce

You need two GPUs with one GPU maxed out on memory consumption. The following should trigger the bug:

using CUDA
CUDA.device!(CuDevice(0)) # replace with device id for vacant GPU
using Flux
Manifest.toml

``` # This file is machine-generated - editing it directly is not advised [[AbstractFFTs]] deps = ["LinearAlgebra"] git-tree-sha1 = "485ee0867925449198280d4af84bdb46a2a404d0" uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c" version = "1.0.1" [[AbstractTrees]] git-tree-sha1 = "03e0550477d86222521d254b741d470ba17ea0b5" uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" version = "0.3.4" [[Adapt]] deps = ["LinearAlgebra"] git-tree-sha1 = "84918055d15b3114ede17ac6a7182f68870c16f7" uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" version = "3.3.1" [[ArgTools]] uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" [[ArrayInterface]] deps = ["IfElse", "LinearAlgebra", "Requires", "SparseArrays", "Static"] git-tree-sha1 = "045ff5e1bc8c6fb1ecb28694abba0a0d55b5f4f5" uuid = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9" version = "3.1.17" [[ArrayLayouts]] deps = ["FillArrays", "LinearAlgebra", "SparseArrays"] git-tree-sha1 = "b53ddb9ea93ed75506a9cfcae4a6514ceffb1997" uuid = "4c555306-a7a7-4459-81d9-ec55ddd5c99a" version = "0.7.0" [[Artifacts]] uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" [[AxisAlgorithms]] deps = ["LinearAlgebra", "Random", "SparseArrays", "WoodburyMatrices"] git-tree-sha1 = "a4d07a1c313392a77042855df46c5f534076fab9" uuid = "13072b0f-2c55-5437-9ae7-d433b7a33950" version = "1.0.0" [[AxisArrays]] deps = ["Dates", "IntervalSets", "IterTools", "RangeArrays"] git-tree-sha1 = "f31f50712cbdf40ee8287f0443b57503e34122ef" uuid = "39de3d68-74b9-583c-8d2d-e117c070f3a9" version = "0.4.3" [[BFloat16s]] deps = ["LinearAlgebra", "Test"] git-tree-sha1 = "4af69e205efc343068dc8722b8dfec1ade89254a" uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b" version = "0.1.0" [[BSON]] git-tree-sha1 = "92b8a8479128367aaab2620b8e73dff632f5ae69" uuid = "fbb218c0-5317-5bc6-957e-2ee96dd4b1f0" version = "0.3.3" [[Base64]] uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" [[Blosc]] deps = ["Blosc_jll"] git-tree-sha1 = "84cf7d0f8fd46ca6f1b3e0305b4b4a37afe50fd6" uuid = "a74b3585-a348-5f62-a45c-50e91977d574" version = "0.7.0" [[Blosc_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Lz4_jll", "Pkg", "Zlib_jll", "Zstd_jll"] git-tree-sha1 = "e747dac84f39c62aff6956651ec359686490134e" uuid = "0b7ba130-8d10-5ba8-a3d6-c5182647fed9" version = "1.21.0+0" [[BufferedStreams]] deps = ["Compat", "Test"] git-tree-sha1 = "5d55b9486590fdda5905c275bb21ce1f0754020f" uuid = "e1450e63-4bb3-523b-b2a4-4ffa8c0fd77d" version = "1.0.0" [[CEnum]] git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9" uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82" version = "0.4.1" [[CUDA]] deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CompilerSupportLibraries_jll", "DataStructures", "ExprTools", "GPUArrays", "GPUCompiler", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "Memoization", "Printf", "Random", "Random123", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "SpecialFunctions", "TimerOutputs"] git-tree-sha1 = "f6f6d2fc7a80b7710b2db4ecb1f59a1b2c2a715a" uuid = "052768ef-5323-5732-b1bb-66c8b64840ba" version = "3.3.0" [[CatIndices]] deps = ["CustomUnitRanges", "OffsetArrays"] git-tree-sha1 = "a0f80a09780eed9b1d106a1bf62041c2efc995bc" uuid = "aafaddc9-749c-510e-ac4f-586e18779b91" version = "0.2.2" [[ChainRules]] deps = ["ChainRulesCore", "Compat", "LinearAlgebra", "Random", "Statistics"] git-tree-sha1 = "720fa9a9ce61ff18842a40f501d6a1f8ba771c64" uuid = "082447d4-558c-5d27-93f4-14fc19e9eca2" version = "0.8.6" [[ChainRulesCore]] deps = ["Compat", "LinearAlgebra", "SparseArrays"] git-tree-sha1 = "d659e42240c2162300b321f05173cab5cc40a5ba" uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" version = "0.10.4" [[CodecZlib]] deps = ["TranscodingStreams", "Zlib_jll"] git-tree-sha1 = "ded953804d019afa9a3f98981d99b33e3db7b6da" uuid = "944b1d66-785c-5afd-91f1-9de20f533193" version = "0.7.0" [[ColorBlendModes]] deps = ["ColorTypes", "FixedPointNumbers"] git-tree-sha1 = "0f8a8ca226003fefc686bf0a652e9386ca9fcb96" uuid = "60508b50-96e1-4007-9d6c-f475c410f16b" version = "0.2.3" [[ColorTypes]] deps = ["FixedPointNumbers", "Random"] git-tree-sha1 = "32a2b8af383f11cbb65803883837a149d10dfe8a" uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f" version = "0.10.12" [[ColorVectorSpace]] deps = ["ColorTypes", "FixedPointNumbers", "LinearAlgebra", "SpecialFunctions", "Statistics", "TensorCore"] git-tree-sha1 = "42a9b08d3f2f951c9b283ea427d96ed9f1f30343" uuid = "c3611d14-8923-5661-9e6a-0046d554d3a4" version = "0.9.5" [[Colors]] deps = ["ColorTypes", "FixedPointNumbers", "Reexport"] git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40" uuid = "5ae59095-9a9b-59fe-a467-6f913c188581" version = "0.12.8" [[CommonSubexpressions]] deps = ["MacroTools", "Test"] git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" version = "0.3.0" [[Compat]] deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"] git-tree-sha1 = "e4e2b39db08f967cc1360951f01e8a75ec441cab" uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" version = "3.30.0" [[CompilerSupportLibraries_jll]] deps = ["Artifacts", "Libdl"] uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" [[ComputationalResources]] git-tree-sha1 = "52cb3ec90e8a8bea0e62e275ba577ad0f74821f7" uuid = "ed09eef8-17a6-5b46-8889-db040fac31e3" version = "0.3.2" [[ConstructionBase]] deps = ["LinearAlgebra"] git-tree-sha1 = "1dc43957fb9a1574fa1b7a449e101bd1fd3a9fb7" uuid = "187b0558-2788-49d3-abe0-74a17ed4e7c9" version = "1.2.1" [[CoordinateTransformations]] deps = ["LinearAlgebra", "StaticArrays"] git-tree-sha1 = "6d1c23e740a586955645500bbec662476204a52c" uuid = "150eb455-5306-5404-9cee-2592286d6298" version = "0.6.1" [[CustomUnitRanges]] git-tree-sha1 = "537c988076d001469093945f3bd0b300b8d3a7f3" uuid = "dc8bdbbb-1ca9-579f-8c36-e416f6a65cce" version = "1.0.1" [[DSP]] deps = ["FFTW", "IterTools", "LinearAlgebra", "Polynomials", "Random", "Reexport", "SpecialFunctions", "Statistics"] git-tree-sha1 = "2a63cb5fc0e8c1f0f139475ef94228c7441dc7d0" uuid = "717857b8-e6f2-59f4-9121-6e50c889abd2" version = "0.6.10" [[DataAPI]] git-tree-sha1 = "dfb3b7e89e395be1e25c2ad6d7690dc29cc53b1d" uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" version = "1.6.0" [[DataAugmentation]] deps = ["ColorBlendModes", "CoordinateTransformations", "Distributions", "ImageDraw", "ImageTransformations", "Images", "Interpolations", "LinearAlgebra", "MosaicViews", "OffsetArrays", "Parameters", "Random", "Rotations", "Setfield", "StaticArrays", "Statistics", "Test"] git-tree-sha1 = "99017f6bd6ebae582a2dbc7d70d3b1d0cd498888" uuid = "88a5189c-e7ff-4f85-ac6b-e6158070f02e" version = "0.2.2" [[DataLoaders]] deps = ["DocStringExtensions", "LearnBase", "MLDataPattern", "Parameters", "Random", "ThreadPools"] git-tree-sha1 = "5ca678562d2cbb0417f56b31ef083b31c654c9ce" uuid = "2e981812-ef13-4a9c-bfa0-ab13047b12a9" version = "0.1.2" [[DataStructures]] deps = ["Compat", "InteractiveUtils", "OrderedCollections"] git-tree-sha1 = "4437b64df1e0adccc3e5d1adbc3ac741095e4677" uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" version = "0.18.9" [[Dates]] deps = ["Printf"] uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" [[DelimitedFiles]] deps = ["Mmap"] uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab" [[DiffResults]] deps = ["StaticArrays"] git-tree-sha1 = "c18e98cba888c6c25d1c3b048e4b3380ca956805" uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5" version = "1.0.3" [[DiffRules]] deps = ["NaNMath", "Random", "SpecialFunctions"] git-tree-sha1 = "214c3fcac57755cfda163d91c58893a8723f93e9" uuid = "b552c78f-8df3-52c6-915a-8e097449b14b" version = "1.0.2" [[Distances]] deps = ["LinearAlgebra", "Statistics", "StatsAPI"] git-tree-sha1 = "abe4ad222b26af3337262b8afb28fab8d215e9f8" uuid = "b4f34e82-e78d-54a5-968a-f98e89d6e8f7" version = "0.10.3" [[Distributed]] deps = ["Random", "Serialization", "Sockets"] uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" [[Distributions]] deps = ["FillArrays", "LinearAlgebra", "PDMats", "Printf", "QuadGK", "Random", "SparseArrays", "SpecialFunctions", "Statistics", "StatsBase", "StatsFuns"] git-tree-sha1 = "a837fdf80f333415b69684ba8e8ae6ba76de6aaa" uuid = "31c24e10-a181-5473-b8eb-7969acd0382f" version = "0.24.18" [[DocStringExtensions]] deps = ["LibGit2"] git-tree-sha1 = "a32185f5428d3986f47c2ab78b1f216d5e6cc96f" uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" version = "0.8.5" [[Downloads]] deps = ["ArgTools", "LibCURL", "NetworkOptions"] uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" [[EllipsisNotation]] deps = ["ArrayInterface"] git-tree-sha1 = "8041575f021cba5a099a456b4163c9a08b566a02" uuid = "da5c29d0-fa7d-589e-88eb-ea29b0a81949" version = "1.1.0" [[ExprTools]] git-tree-sha1 = "10407a39b87f29d47ebaca8edbc75d7c302ff93e" uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04" version = "0.1.3" [[EzXML]] deps = ["Printf", "XML2_jll"] git-tree-sha1 = "0fa3b52a04a4e210aeb1626def9c90df3ae65268" uuid = "8f5d6c58-4d21-5cfd-889c-e3ad7ee6a615" version = "1.1.0" [[FFTViews]] deps = ["CustomUnitRanges", "FFTW"] git-tree-sha1 = "70a0cfd9b1c86b0209e38fbfe6d8231fd606eeaf" uuid = "4f61f5a4-77b1-5117-aa51-3ab5ef4ef0cd" version = "0.3.1" [[FFTW]] deps = ["AbstractFFTs", "FFTW_jll", "LinearAlgebra", "MKL_jll", "Preferences", "Reexport"] git-tree-sha1 = "ae8de3350af3026008be4ba23e1e905ab2011d20" uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341" version = "1.4.2" [[FFTW_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "5a0d4b6a22a34d17d53543bd124f4b08ed78e8b0" uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a" version = "3.3.9+7" [[FileIO]] deps = ["Pkg", "Requires", "UUIDs"] git-tree-sha1 = "ca36405da56db2f4730b29cd2ad2bf5869baea3c" uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549" version = "1.9.1" [[FillArrays]] deps = ["LinearAlgebra", "Random", "SparseArrays"] git-tree-sha1 = "31939159aeb8ffad1d4d8ee44d07f8558273120a" uuid = "1a297f60-69ca-5386-bcde-b61e274b549b" version = "0.11.7" [[FixedPointNumbers]] deps = ["Statistics"] git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc" uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93" version = "0.8.4" [[Flux]] deps = ["AbstractTrees", "Adapt", "CUDA", "CodecZlib", "Colors", "DelimitedFiles", "Functors", "Juno", "LinearAlgebra", "MacroTools", "NNlib", "NNlibCUDA", "Pkg", "Printf", "Random", "Reexport", "SHA", "Statistics", "StatsBase", "Test", "ZipFile", "Zygote"] git-tree-sha1 = "0b3c6d0ce57d3b793eabd346ccc8f605035ef079" uuid = "587475ba-b771-5e3f-ad9e-33799f191a9c" version = "0.12.4" [[ForwardDiff]] deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "LinearAlgebra", "NaNMath", "Printf", "Random", "SpecialFunctions", "StaticArrays"] git-tree-sha1 = "e2af66012e08966366a43251e1fd421522908be6" uuid = "f6369f11-7733-5829-9624-2563aa707210" version = "0.10.18" [[Functors]] deps = ["MacroTools"] git-tree-sha1 = "a7bb2af991c43dcf5c3455d276dd83976799634f" uuid = "d9f16b24-f501-4c13-a1f2-28368ffc5196" version = "0.2.1" [[Future]] deps = ["Random"] uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820" [[GPUArrays]] deps = ["AbstractFFTs", "Adapt", "LinearAlgebra", "Printf", "Random", "Serialization", "Statistics"] git-tree-sha1 = "3683030b5479249abaa18aa930fc02307fed05d3" uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7" version = "7.0.0" [[GPUCompiler]] deps = ["DataStructures", "ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "TimerOutputs", "UUIDs"] git-tree-sha1 = "e12c5495c19b852e8840bf2640b539383dda3954" uuid = "61eb1bfa-7361-4325-ad38-22787b887f55" version = "0.12.0" [[Graphics]] deps = ["Colors", "LinearAlgebra", "NaNMath"] git-tree-sha1 = "2c1cf4df419938ece72de17f368a021ee162762e" uuid = "a2bd30eb-e257-5431-a919-1863eab51364" version = "1.1.0" [[HDF5]] deps = ["Blosc", "Compat", "HDF5_jll", "Libdl", "Mmap", "Random", "Requires"] git-tree-sha1 = "1d18a48a037b14052ca462ea9d05dee3ac607d23" uuid = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f" version = "0.15.5" [[HDF5_jll]] deps = ["Artifacts", "JLLWrappers", "LibCURL_jll", "Libdl", "OpenSSL_jll", "Pkg", "Zlib_jll"] git-tree-sha1 = "fd83fa0bde42e01952757f01149dd968c06c4dba" uuid = "0234f1f7-429e-5d53-9886-15a909be8d59" version = "1.12.0+1" [[IRTools]] deps = ["InteractiveUtils", "MacroTools", "Test"] git-tree-sha1 = "c67e7515a11f726f44083e74f218d134396d6510" uuid = "7869d1d1-7146-5819-86e3-90919afe41df" version = "0.4.2" [[IdentityRanges]] deps = ["OffsetArrays"] git-tree-sha1 = "be8fcd695c4da16a1d6d0cd213cb88090a150e3b" uuid = "bbac6d45-d8f3-5730-bfe4-7a449cd117ca" version = "0.3.1" [[IfElse]] git-tree-sha1 = "28e837ff3e7a6c3cdb252ce49fb412c8eb3caeef" uuid = "615f187c-cbe4-4ef1-ba3b-2fcf58d6d173" version = "0.1.0" [[ImageAxes]] deps = ["AxisArrays", "ImageCore", "Reexport", "SimpleTraits"] git-tree-sha1 = "794ad1d922c432082bc1aaa9fa8ffbd1fe74e621" uuid = "2803e5a7-5153-5ecf-9a86-9b4c37f5f5ac" version = "0.6.9" [[ImageContrastAdjustment]] deps = ["ColorVectorSpace", "ImageCore", "ImageTransformations", "Parameters"] git-tree-sha1 = "2e6084db6cccab11fe0bc3e4130bd3d117092ed9" uuid = "f332f351-ec65-5f6a-b3d1-319c6670881a" version = "0.3.7" [[ImageCore]] deps = ["AbstractFFTs", "Colors", "FixedPointNumbers", "Graphics", "MappedArrays", "MosaicViews", "OffsetArrays", "PaddedViews", "Reexport"] git-tree-sha1 = "da98af44e78a2a9dd131ac12cf19b574f551dace" uuid = "a09fc81d-aa75-5fe9-8630-4744c3626534" version = "0.8.21" [[ImageDistances]] deps = ["ColorVectorSpace", "Distances", "ImageCore", "ImageMorphology", "LinearAlgebra", "Statistics"] git-tree-sha1 = "6378c34a3c3a216235210d19b9f495ecfff2f85f" uuid = "51556ac3-7006-55f5-8cb3-34580c88182d" version = "0.2.13" [[ImageDraw]] deps = ["Distances", "ImageCore", "LinearAlgebra"] git-tree-sha1 = "6ed6e945d909f87c3013e391dcd3b2a56e48b331" uuid = "4381153b-2b60-58ae-a1ba-fd683676385f" version = "0.2.5" [[ImageFiltering]] deps = ["CatIndices", "ColorVectorSpace", "ComputationalResources", "DataStructures", "FFTViews", "FFTW", "ImageCore", "LinearAlgebra", "OffsetArrays", "Requires", "SparseArrays", "StaticArrays", "Statistics", "TiledIteration"] git-tree-sha1 = "bf96839133212d3eff4a1c3a80c57abc7cfbf0ce" uuid = "6a3955dd-da59-5b1f-98d4-e7296123deb5" version = "0.6.21" [[ImageIO]] deps = ["FileIO", "Netpbm", "PNGFiles", "TiffImages", "UUIDs"] git-tree-sha1 = "d2b485a4db3d24b58a7558d5e47979c6e9ba1c8e" uuid = "82e4d734-157c-48bb-816b-45c225c6df19" version = "0.5.5" [[ImageMagick]] deps = ["FileIO", "ImageCore", "ImageMagick_jll", "InteractiveUtils", "Libdl", "Pkg", "Random"] git-tree-sha1 = "5bc1cb62e0c5f1005868358db0692c994c3a13c6" uuid = "6218d12a-5da1-5696-b52f-db25d2ecc6d1" version = "1.2.1" [[ImageMagick_jll]] deps = ["JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pkg", "Zlib_jll", "libpng_jll"] git-tree-sha1 = "1c0a2295cca535fabaf2029062912591e9b61987" uuid = "c73af94c-d91f-53ed-93a7-00f77d67a9d7" version = "6.9.10-12+3" [[ImageMetadata]] deps = ["AxisArrays", "ColorVectorSpace", "ImageAxes", "ImageCore", "IndirectArrays"] git-tree-sha1 = "ae76038347dc4edcdb06b541595268fca65b6a42" uuid = "bc367c6b-8a6b-528e-b4bd-a4b897500b49" version = "0.9.5" [[ImageMorphology]] deps = ["ColorVectorSpace", "ImageCore", "LinearAlgebra", "TiledIteration"] git-tree-sha1 = "68e7cbcd7dfaa3c2f74b0a8ab3066f5de8f2b71d" uuid = "787d08f9-d448-5407-9aad-5290dd7ab264" version = "0.2.11" [[ImageQualityIndexes]] deps = ["ColorVectorSpace", "ImageCore", "ImageDistances", "ImageFiltering", "OffsetArrays", "Statistics"] git-tree-sha1 = "1198f85fa2481a3bb94bf937495ba1916f12b533" uuid = "2996bd0c-7a13-11e9-2da2-2f5ce47296a9" version = "0.2.2" [[ImageShow]] deps = ["Base64", "FileIO", "ImageCore", "OffsetArrays", "Requires", "StackViews"] git-tree-sha1 = "832abfd709fa436a562db47fd8e81377f72b01f9" uuid = "4e3cecfd-b093-5904-9786-8bbb286a6a31" version = "0.3.1" [[ImageTransformations]] deps = ["AxisAlgorithms", "ColorVectorSpace", "CoordinateTransformations", "IdentityRanges", "ImageCore", "Interpolations", "OffsetArrays", "Rotations", "StaticArrays"] git-tree-sha1 = "d966631de06f36c8cd4bec4bb2e8fa731db16ed9" uuid = "02fcd773-0e25-5acc-982a-7f6622650795" version = "0.8.12" [[Images]] deps = ["AxisArrays", "Base64", "ColorVectorSpace", "FileIO", "Graphics", "ImageAxes", "ImageContrastAdjustment", "ImageCore", "ImageDistances", "ImageFiltering", "ImageIO", "ImageMagick", "ImageMetadata", "ImageMorphology", "ImageQualityIndexes", "ImageShow", "ImageTransformations", "IndirectArrays", "OffsetArrays", "Random", "Reexport", "SparseArrays", "StaticArrays", "Statistics", "StatsBase", "TiledIteration"] git-tree-sha1 = "8b714d5e11c91a0d945717430ec20f9251af4bd2" uuid = "916415d5-f1e6-5110-898d-aaa5f9f070e0" version = "0.24.1" [[IndirectArrays]] git-tree-sha1 = "c2a145a145dc03a7620af1444e0264ef907bd44f" uuid = "9b13fd28-a010-5f03-acff-a1bbcff69959" version = "0.5.1" [[InfiniteArrays]] deps = ["ArrayLayouts", "DSP", "FillArrays", "Infinities", "LazyArrays", "LinearAlgebra", "Statistics"] git-tree-sha1 = "c1c3ba6fe2c20721dc44e7db2df43c2652e60089" uuid = "4858937d-0d70-526a-a4dd-2d5cb5dd786c" version = "0.10.7" [[Infinities]] git-tree-sha1 = "06570a7404df6901f0fa09733dde6f7ae4c68b0a" uuid = "e1ba4f0e-776d-440f-acd9-e1d2e9742647" version = "0.1.0" [[IntelOpenMP_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "d979e54b71da82f3a65b62553da4fc3d18c9004c" uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0" version = "2018.0.3+2" [[InteractiveUtils]] deps = ["Markdown"] uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" [[Interpolations]] deps = ["AxisAlgorithms", "LinearAlgebra", "OffsetArrays", "Random", "Ratios", "SharedArrays", "SparseArrays", "StaticArrays", "WoodburyMatrices"] git-tree-sha1 = "1e0e51692a3a77f1eeb51bf741bdd0439ed210e7" uuid = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59" version = "0.13.2" [[IntervalSets]] deps = ["Dates", "EllipsisNotation", "Statistics"] git-tree-sha1 = "3cc368af3f110a767ac786560045dceddfc16758" uuid = "8197267c-284f-5f27-9208-e0e47529a953" version = "0.5.3" [[Intervals]] deps = ["Dates", "Printf", "RecipesBase", "Serialization", "TimeZones"] git-tree-sha1 = "323a38ed1952d30586d0fe03412cde9399d3618b" uuid = "d8418881-c3e1-53bb-8760-2df7ec849ed5" version = "1.5.0" [[IterTools]] git-tree-sha1 = "05110a2ab1fc5f932622ffea2a003221f4782c18" uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e" version = "1.3.0" [[JLLWrappers]] deps = ["Preferences"] git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e" uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" version = "1.3.0" [[JpegTurbo_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "d735490ac75c5cb9f1b00d8b5509c11984dc6943" uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" version = "2.1.0+0" [[Juno]] deps = ["Base64", "Logging", "Media", "Profile"] git-tree-sha1 = "07cb43290a840908a771552911a6274bc6c072c7" uuid = "e5e0dc1b-0480-54bc-9374-aad01c23163d" version = "0.8.4" [[LLVM]] deps = ["CEnum", "Libdl", "Printf", "Unicode"] git-tree-sha1 = "b499c68a45249b0385585c62f4a9b62b5db8e691" uuid = "929cbde3-209d-540e-8aea-75f648917ca0" version = "3.7.1" [[LazyArrays]] deps = ["ArrayLayouts", "FillArrays", "LinearAlgebra", "MacroTools", "MatrixFactorizations", "SparseArrays", "StaticArrays"] git-tree-sha1 = "893ba0b12e57054fab3fee55349389c939a017e5" uuid = "5078a376-72f3-5289-bfd5-ec5146d43c02" version = "0.21.6" [[LazyArtifacts]] deps = ["Artifacts", "Pkg"] uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3" [[LearnBase]] deps = ["LinearAlgebra", "StatsBase"] git-tree-sha1 = "47e6f4623c1db88570c7a7fa66c6528b92ba4725" uuid = "7f8f8fb0-2700-5f03-b4bd-41f8cfc144b6" version = "0.3.0" [[LibCURL]] deps = ["LibCURL_jll", "MozillaCACerts_jll"] uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" [[LibCURL_jll]] deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" [[LibGit2]] deps = ["Base64", "NetworkOptions", "Printf", "SHA"] uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" [[LibSSH2_jll]] deps = ["Artifacts", "Libdl", "MbedTLS_jll"] uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" [[Libdl]] uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" [[Libiconv_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "8d22e127ea9a0917bc98ebd3755c8bd31989381e" uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" version = "1.16.1+0" [[Libtiff_jll]] deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"] git-tree-sha1 = "340e257aada13f95f98ee352d316c3bed37c8ab9" uuid = "89763e89-9b03-5906-acba-b20f662cd828" version = "4.3.0+0" [[LinearAlgebra]] deps = ["Libdl"] uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" [[LogExpFunctions]] deps = ["DocStringExtensions", "LinearAlgebra"] git-tree-sha1 = "1ba664552f1ef15325e68dc4c05c3ef8c2d5d885" uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" version = "0.2.4" [[Logging]] uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" [[Lz4_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "5d494bc6e85c4c9b626ee0cab05daa4085486ab1" uuid = "5ced341a-0733-55b8-9ab6-a4889d929147" version = "1.9.3+0" [[MAT]] deps = ["BufferedStreams", "CodecZlib", "HDF5", "SparseArrays"] git-tree-sha1 = "5c62992f3d46b8dce69bdd234279bb5a369db7d5" uuid = "23992714-dd62-5051-b70f-ba57cb901cac" version = "0.10.1" [[MKL_jll]] deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"] git-tree-sha1 = "c253236b0ed414624b083e6b72bfe891fbd2c7af" uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7" version = "2021.1.1+1" [[MLDataPattern]] deps = ["LearnBase", "MLLabelUtils", "Random", "SparseArrays", "StatsBase"] git-tree-sha1 = "e99514e96e8b8129bb333c69e063a56ab6402b5b" uuid = "9920b226-0b2a-5f5f-9153-9aa70a013f8b" version = "0.5.4" [[MLLabelUtils]] deps = ["LearnBase", "MappedArrays", "StatsBase"] git-tree-sha1 = "3211c1fdd1efaefa692c8cf60e021fb007b76a08" uuid = "66a33bbf-0c2b-5fc8-a008-9da813334f0a" version = "0.5.6" [[MacroTools]] deps = ["Markdown", "Random"] git-tree-sha1 = "6a8a2a625ab0dea913aba95c11370589e0239ff0" uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" version = "0.5.6" [[MappedArrays]] deps = ["FixedPointNumbers"] git-tree-sha1 = "b92bd220c95a8bbe89af28f11201fd080e0e3fe7" uuid = "dbb5928d-eab1-5f90-85c2-b9b0edb7c900" version = "0.3.0" [[Markdown]] deps = ["Base64"] uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" [[MatrixFactorizations]] deps = ["ArrayLayouts", "LinearAlgebra", "Printf", "Random"] git-tree-sha1 = "304dec6e95a14d2284759645078f7e4f0189ea39" uuid = "a3b82374-2e81-5b9e-98ce-41277c0e4c87" version = "0.8.3" [[MbedTLS_jll]] deps = ["Artifacts", "Libdl"] uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" [[Media]] deps = ["MacroTools", "Test"] git-tree-sha1 = "75a54abd10709c01f1b86b84ec225d26e840ed58" uuid = "e89f7d12-3494-54d1-8411-f7d8b9ae1f27" version = "0.5.0" [[Memoization]] deps = ["MacroTools"] git-tree-sha1 = "ed3f3b19264e531b29d1d6a266336ccec5e597bd" uuid = "6fafb56a-5788-4b4e-91ca-c0cea6611c73" version = "0.1.10" [[Metalhead]] deps = ["Artifacts", "BSON", "Flux", "Functors", "LazyArtifacts"] git-tree-sha1 = "da783d33d01be7c3e614ac1cdb69930fee8ec501" repo-rev = "darsnack/vision-refactor" repo-url = ".." uuid = "dbeba491-748d-5e0e-a39e-b530a07fa0cc" version = "0.6.0" [[Missings]] deps = ["DataAPI"] git-tree-sha1 = "4ea90bd5d3985ae1f9a908bd4500ae88921c5ce7" uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" version = "1.0.0" [[Mmap]] uuid = "a63ad114-7e13-5084-954f-fe012c677804" [[Mocking]] deps = ["ExprTools"] git-tree-sha1 = "916b850daad0d46b8c71f65f719c49957e9513ed" uuid = "78c3b35d-d492-501b-9361-3d52fe80e533" version = "0.7.1" [[MosaicViews]] deps = ["MappedArrays", "OffsetArrays", "PaddedViews"] git-tree-sha1 = "614e8d77264d20c1db83661daadfab38e8e4b77e" uuid = "e94cdb99-869f-56ef-bcf0-1ae2bcbe0389" version = "0.2.4" [[MozillaCACerts_jll]] uuid = "14a3606d-f60d-562e-9121-12d972cd8159" [[NNlib]] deps = ["Adapt", "ChainRulesCore", "Compat", "LinearAlgebra", "Pkg", "Requires", "Statistics"] git-tree-sha1 = "0bf1fbb9dc557f2af9fb7e1337366d69de0dc78c" uuid = "872c559c-99b0-510c-b3b7-b6c96a88d5cd" version = "0.7.21" [[NNlibCUDA]] deps = ["CUDA", "LinearAlgebra", "NNlib", "Random", "Statistics"] git-tree-sha1 = "bd8b29bf75be7a6c2b288b4b9a4e8903d0376ac1" uuid = "a00861dc-f156-4864-bf3c-e6376f28a68d" version = "0.1.3" [[NaNMath]] git-tree-sha1 = "bfe47e760d60b82b66b61d2d44128b62e3a369fb" uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3" version = "0.3.5" [[Netpbm]] deps = ["ColorVectorSpace", "FileIO", "ImageCore"] git-tree-sha1 = "09589171688f0039f13ebe0fdcc7288f50228b52" uuid = "f09324ee-3d7c-5217-9330-fc30815ba969" version = "1.0.1" [[NetworkOptions]] uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" [[OffsetArrays]] deps = ["Adapt"] git-tree-sha1 = "1381a7142eefd4cd12f052a4d2d790fe21bd1d55" uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" version = "1.9.2" [[OpenSSL_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "15003dcb7d8db3c6c857fda14891a539a8f2705a" uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" version = "1.1.10+0" [[OpenSpecFun_jll]] deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1" uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e" version = "0.5.5+0" [[OrderedCollections]] git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c" uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" version = "1.4.1" [[PDMats]] deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"] git-tree-sha1 = "4dd403333bcf0909341cfe57ec115152f937d7d8" uuid = "90014a1f-27ba-587c-ab20-58faa44d9150" version = "0.11.1" [[PNGFiles]] deps = ["Base64", "CEnum", "ImageCore", "IndirectArrays", "OffsetArrays", "libpng_jll"] git-tree-sha1 = "520e28d4026d16dcf7b8c8140a3041f0e20a9ca8" uuid = "f57f5aa1-a3ce-4bc8-8ab9-96f992907883" version = "0.3.7" [[PaddedViews]] deps = ["OffsetArrays"] git-tree-sha1 = "0fa5e78929aebc3f6b56e1a88cf505bb00a354c4" uuid = "5432bcbf-9aad-5242-b902-cca2824c8663" version = "0.5.8" [[ParameterSchedulers]] deps = ["Flux", "InfiniteArrays"] git-tree-sha1 = "63430ecbe6b07fde701205ce45a0b01980876c30" uuid = "d7d3b36b-41b8-4d0d-a2bf-768c6151755e" version = "0.2.3" [[Parameters]] deps = ["OrderedCollections", "UnPack"] git-tree-sha1 = "2276ac65f1e236e0a6ea70baff3f62ad4c625345" uuid = "d96e819e-fc66-5662-9728-84c9c7592b0a" version = "0.12.2" [[Pkg]] deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" [[PkgVersion]] deps = ["Pkg"] git-tree-sha1 = "a7a7e1a88853564e551e4eba8650f8c38df79b37" uuid = "eebad327-c553-4316-9ea0-9fa01ccd7688" version = "0.1.1" [[Polynomials]] deps = ["Intervals", "LinearAlgebra", "OffsetArrays", "RecipesBase"] git-tree-sha1 = "0b15f3597b01eb76764dd03c3c23d6679a3c32c8" uuid = "f27b6e38-b328-58d1-80ce-0feddd5e7a45" version = "1.2.1" [[Preferences]] deps = ["TOML"] git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a" uuid = "21216c6a-2e73-6563-6e65-726566657250" version = "1.2.2" [[Printf]] deps = ["Unicode"] uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" [[Profile]] deps = ["Printf"] uuid = "9abbd945-dff8-562f-b5e8-e1ebf5ef1b79" [[ProgressMeter]] deps = ["Distributed", "Printf"] git-tree-sha1 = "afadeba63d90ff223a6a48d2009434ecee2ec9e8" uuid = "92933f4c-e287-5a05-a399-4b506db050ca" version = "1.7.1" [[QuadGK]] deps = ["DataStructures", "LinearAlgebra"] git-tree-sha1 = "12fbe86da16df6679be7521dfb39fbc861e1dc7b" uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc" version = "2.4.1" [[REPL]] deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" [[Random]] deps = ["Serialization"] uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" [[Random123]] deps = ["Libdl", "Random", "RandomNumbers"] git-tree-sha1 = "5fbcfdea78c76cf16be611c9cb1d194f3e448e37" uuid = "74087812-796a-5b5d-8853-05524746bad3" version = "1.4.0" [[RandomNumbers]] deps = ["Random", "Requires"] git-tree-sha1 = "441e6fc35597524ada7f85e13df1f4e10137d16f" uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143" version = "1.4.0" [[RangeArrays]] git-tree-sha1 = "b9039e93773ddcfc828f12aadf7115b4b4d225f5" uuid = "b3c3ace0-ae52-54e7-9d0b-2c1406fd6b9d" version = "0.3.2" [[Ratios]] git-tree-sha1 = "37d210f612d70f3f7d57d488cb3b6eff56ad4e41" uuid = "c84ed2f1-dad5-54f0-aa8e-dbefe2724439" version = "0.4.0" [[RecipesBase]] git-tree-sha1 = "b3fb709f3c97bfc6e948be68beeecb55a0b340ae" uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" version = "1.1.1" [[Reexport]] git-tree-sha1 = "5f6c21241f0f655da3952fd60aa18477cf96c220" uuid = "189a3867-3050-52da-a836-e630ba90ab69" version = "1.1.0" [[Requires]] deps = ["UUIDs"] git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621" uuid = "ae029012-a4dd-5104-9daa-d747884805df" version = "1.1.3" [[Rmath]] deps = ["Random", "Rmath_jll"] git-tree-sha1 = "bf3188feca147ce108c76ad82c2792c57abe7b1f" uuid = "79098fc4-a85e-5d69-aa6a-4863f24498fa" version = "0.7.0" [[Rmath_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "68db32dff12bb6127bac73c209881191bf0efbb7" uuid = "f50d1b31-88e8-58de-be2c-1cc44531875f" version = "0.3.0+0" [[Rotations]] deps = ["LinearAlgebra", "StaticArrays", "Statistics"] git-tree-sha1 = "2ed8d8a16d703f900168822d83699b8c3c1a5cd8" uuid = "6038ab10-8711-5258-84ad-4b1120ba62dc" version = "1.0.2" [[SHA]] uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" [[Serialization]] uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" [[Setfield]] deps = ["ConstructionBase", "Future", "MacroTools", "Requires"] git-tree-sha1 = "d5640fc570fb1b6c54512f0bd3853866bd298b3e" uuid = "efcf1570-3423-57d1-acb7-fd33fddbac46" version = "0.7.0" [[SharedArrays]] deps = ["Distributed", "Mmap", "Random", "Serialization"] uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" [[SimpleTraits]] deps = ["InteractiveUtils", "MacroTools"] git-tree-sha1 = "daf7aec3fe3acb2131388f93a4c409b8c7f62226" uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" version = "0.9.3" [[Sockets]] uuid = "6462fe0b-24de-5631-8697-dd941f90decc" [[SortingAlgorithms]] deps = ["DataStructures"] git-tree-sha1 = "2ec1962eba973f383239da22e75218565c390a96" uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" version = "1.0.0" [[SparseArrays]] deps = ["LinearAlgebra", "Random"] uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" [[SpecialFunctions]] deps = ["ChainRulesCore", "LogExpFunctions", "OpenSpecFun_jll"] git-tree-sha1 = "a50550fa3164a8c46747e62063b4d774ac1bcf49" uuid = "276daf66-3868-5448-9aa4-cd146d93841b" version = "1.5.1" [[StackViews]] deps = ["OffsetArrays"] git-tree-sha1 = "46e589465204cd0c08b4bd97385e4fa79a0c770c" uuid = "cae243ae-269e-4f55-b966-ac2d0dc13c15" version = "0.1.1" [[Static]] deps = ["IfElse"] git-tree-sha1 = "2740ea27b66a41f9d213561a04573da5d3823d4b" uuid = "aedffcd0-7271-4cad-89d0-dc628f76c6d3" version = "0.2.5" [[StaticArrays]] deps = ["LinearAlgebra", "Random", "Statistics"] git-tree-sha1 = "42378d3bab8b4f57aa1ca443821b752850592668" uuid = "90137ffa-7385-5640-81b9-e52037218182" version = "1.2.2" [[Statistics]] deps = ["LinearAlgebra", "SparseArrays"] uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" [[StatsAPI]] git-tree-sha1 = "1958272568dc176a1d881acb797beb909c785510" uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" version = "1.0.0" [[StatsBase]] deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] git-tree-sha1 = "2f6792d523d7448bbe2fec99eca9218f06cc746d" uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" version = "0.33.8" [[StatsFuns]] deps = ["LogExpFunctions", "Rmath", "SpecialFunctions"] git-tree-sha1 = "30cd8c360c54081f806b1ee14d2eecbef3c04c49" uuid = "4c63d2b9-4356-54db-8cca-17b64c39e42c" version = "0.9.8" [[SuiteSparse]] deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"] uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9" [[TOML]] deps = ["Dates"] uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" [[Tar]] deps = ["ArgTools", "SHA"] uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" [[TensorCore]] deps = ["LinearAlgebra"] git-tree-sha1 = "1feb45f88d133a655e001435632f019a9a1bcdb6" uuid = "62fd8b95-f654-4bbd-a8a5-9c27f68ccd50" version = "0.1.1" [[Test]] deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" [[ThreadPools]] deps = ["Printf", "RecipesBase", "Statistics"] git-tree-sha1 = "705ccc29d575b87cceb359dfea19f4653d06df8f" uuid = "b189fb0b-2eb5-4ed4-bc0c-d34c51242431" version = "1.2.1" [[TiffImages]] deps = ["ColorTypes", "DocStringExtensions", "FileIO", "FixedPointNumbers", "IndirectArrays", "OffsetArrays", "OrderedCollections", "PkgVersion", "ProgressMeter"] git-tree-sha1 = "af30d22cc2cdda1848821135bfb04ebd657005a2" uuid = "731e570b-9d59-4bfa-96dc-6df516fadf69" version = "0.3.2" [[TiledIteration]] deps = ["OffsetArrays"] git-tree-sha1 = "52c5f816857bfb3291c7d25420b1f4aca0a74d18" uuid = "06e1c1a7-607b-532d-9fad-de7d9aa2abac" version = "0.3.0" [[TimeZones]] deps = ["Dates", "EzXML", "LazyArtifacts", "Mocking", "Pkg", "Printf", "RecipesBase", "Serialization", "Unicode"] git-tree-sha1 = "960099aed321e05ac649c90d583d59c9309faee1" uuid = "f269a46b-ccf7-5d73-abea-4c690281aa53" version = "1.5.5" [[TimerOutputs]] deps = ["ExprTools", "Printf"] git-tree-sha1 = "bf8aacc899a1bd16522d0350e1e2310510d77236" uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f" version = "0.5.9" [[TranscodingStreams]] deps = ["Random", "Test"] git-tree-sha1 = "7c53c35547de1c5b9d46a4797cf6d8253807108c" uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa" version = "0.9.5" [[UUIDs]] deps = ["Random", "SHA"] uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" [[UnPack]] git-tree-sha1 = "387c1f73762231e86e0c9c5443ce3b4a0a9a0c2b" uuid = "3a884ed6-31ef-47d7-9d2a-63182c4928ed" version = "1.0.2" [[Unicode]] uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" [[WoodburyMatrices]] deps = ["LinearAlgebra", "SparseArrays"] git-tree-sha1 = "59e2ad8fd1591ea019a5259bd012d7aee15f995c" uuid = "efce3f68-66dc-5838-9240-27a6d6f5f9b6" version = "0.5.3" [[XML2_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "Zlib_jll"] git-tree-sha1 = "1acf5bdf07aa0907e0a37d3718bb88d4b687b74a" uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" version = "2.9.12+0" [[ZipFile]] deps = ["Libdl", "Printf", "Zlib_jll"] git-tree-sha1 = "c3a5637e27e914a7a445b8d0ad063d701931e9f7" uuid = "a5390f91-8eb1-5f08-bee0-b1d1ffed6cea" version = "0.9.3" [[Zlib_jll]] deps = ["Libdl"] uuid = "83775a58-1f1d-513f-b197-d71354ab007a" [[Zstd_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "cc4bf3fdde8b7e3e9fa0351bdeedba1cf3b7f6e6" uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" version = "1.5.0+0" [[Zygote]] deps = ["AbstractFFTs", "ChainRules", "ChainRulesCore", "DiffRules", "Distributed", "FillArrays", "ForwardDiff", "IRTools", "InteractiveUtils", "LinearAlgebra", "MacroTools", "NaNMath", "Random", "Requires", "SpecialFunctions", "Statistics", "ZygoteRules"] git-tree-sha1 = "b1d95edd4e693066c38c13a10aab0a8f6a6e2f65" uuid = "e88e6eb3-aa80-5325-afca-941959d7151f" version = "0.6.12" [[ZygoteRules]] deps = ["MacroTools"] git-tree-sha1 = "9e7a1e8ca60b742e508a315c17eef5211e7fbfd7" uuid = "700de1a5-db45-46bc-99cf-38207098b444" version = "0.2.1" [[libpng_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"] git-tree-sha1 = "94d180a6d2b5e55e447e2d27a29ed04fe79eb30c" uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" version = "1.6.38+0" [[nghttp2_jll]] deps = ["Artifacts", "Libdl"] uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" [[p7zip_jll]] deps = ["Artifacts", "Libdl"] uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" ```

Expected behavior

No OOMs since I am not on a device that is currently being used.

Version info

Details on Julia:

Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_NUM_THREADS = 4

Details on CUDA:

CUDA toolkit 11.3.1, artifact installation
NVIDIA driver 465.27.0

Libraries: 
- CUBLAS: 11.5.1
- CURAND: 10.2.4
- CUFFT: 10.4.2
- CUSOLVER: 11.1.2
- CUSPARSE: 11.6.0
- CUPTI: 14.0.0
- NVML: 11.0.0+465.27
- CUDNN: 8.20.0 (for CUDA 11.3.0)
  Downloaded artifact: CUTENSOR
- CUTENSOR: 1.3.0 (for CUDA 11.2.0)

Toolchain:
- Julia: 1.6.1
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

9 devices:
  0: NVIDIA Tesla V100-PCIE-32GB (sm_70, 10.500 MiB / 31.749 GiB available)
  1: NVIDIA Tesla V100-PCIE-32GB (sm_70, 25.500 MiB / 31.749 GiB available)
  2: NVIDIA Tesla V100-PCIE-32GB (sm_70, 185.500 MiB / 31.749 GiB available)
  3: NVIDIA Tesla V100-PCIE-32GB (sm_70, 25.500 MiB / 31.749 GiB available)
  4: NVIDIA Tesla V100-PCIE-16GB (sm_70, 10.500 MiB / 15.782 GiB available)
  5: NVIDIA Tesla P100-PCIE-16GB (sm_60, 15.372 GiB / 15.899 GiB available)
  6: NVIDIA Tesla P100-PCIE-16GB (sm_60, 15.650 GiB / 15.899 GiB available)
  7: NVIDIA GeForce GTX 1080 Ti (sm_61, 10.526 GiB / 10.917 GiB available)
  8: NVIDIA GeForce GTX 1080 Ti (sm_61, 10.783 GiB / 10.917 GiB available)
darsnack commented 3 years ago
The error message:

``` ERROR: LoadError: InitError: Out of GPU memory Stacktrace: [1] throw_api_error(res::CUDA.cudaError_enum) @ CUDA ~/.julia/packages/CUDA/mVgLI/lib/cudadrv/error.jl:103 [2] macro expansion @ ~/.julia/packages/CUDA/mVgLI/lib/cudadrv/error.jl:115 [inlined] [3] cuDevicePrimaryCtxRetain(pctx::Base.RefValue{Ptr{Nothing}}, dev::CUDA.CuDevice) @ CUDA ~/.julia/packages/CUDA/mVgLI/lib/utils/call.jl:26 [4] CuContext @ ~/.julia/packages/CUDA/mVgLI/lib/cudadrv/context.jl:64 [inlined] [5] context(dev::CUDA.CuDevice) @ CUDA ~/.julia/packages/CUDA/mVgLI/src/state.jl:248 [6] TaskLocalState (repeats 2 times) @ ~/.julia/packages/CUDA/mVgLI/src/state.jl:50 [inlined] [7] task_local_state!() @ CUDA ~/.julia/packages/CUDA/mVgLI/src/state.jl:73 [8] prepare_cuda_state @ ~/.julia/packages/CUDA/mVgLI/src/state.jl:88 [inlined] [9] initialize_api @ ~/.julia/packages/CUDA/mVgLI/lib/cudnn/error.jl:27 [inlined] [10] macro expansion @ ~/.julia/packages/CUDA/mVgLI/lib/cudnn/libcudnn.jl:2129 [inlined] [11] macro expansion @ ~/.julia/packages/CUDA/mVgLI/src/pool.jl:402 [inlined] [12] macro expansion @ ~/.julia/packages/CUDA/mVgLI/lib/cudnn/error.jl:37 [inlined] [13] cudnnSetCallback(mask::UInt32, udata::Base.AsyncCondition, fptr::Ptr{Nothing}) @ CUDA.CUDNN ~/.julia/packages/CUDA/mVgLI/lib/utils/call.jl:26 [14] __runtime_init__() @ CUDA.CUDNN ~/.julia/packages/CUDA/mVgLI/lib/cudnn/CUDNN.jl:148 [15] macro expansion @ ~/.julia/packages/CUDA/mVgLI/deps/bindeps.jl:496 [inlined] [16] macro expansion @ lock.jl:209 [inlined] [17] libcudnn(; throw_error::Bool) @ CUDA.Deps ~/.julia/packages/CUDA/mVgLI/deps/bindeps.jl:14 [18] has_cudnn() @ CUDA.Deps ~/.julia/packages/CUDA/mVgLI/deps/bindeps.jl:504 [19] __init__() @ Flux ~/.julia/packages/Flux/0c9kI/src/Flux.jl:62 [20] _include_from_serialized(path::String, depmods::Vector{Any}) @ Base ./loading.jl:674 [21] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String) @ Base ./loading.jl:760 [22] _require(pkg::Base.PkgId) @ Base ./loading.jl:998 [23] require(uuidkey::Base.PkgId) @ Base ./loading.jl:914 [24] require(into::Module, mod::Symbol) @ Base ./loading.jl:901 [25] include @ ./Base.jl:386 [inlined] [26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String) @ Base ./loading.jl:1213 [27] top-level scope @ none:1 [28] eval @ ./boot.jl:360 [inlined] [29] eval(x::Expr) @ Base.MainInclude ./client.jl:446 [30] top-level scope @ none:1fatal: error thrown and no exception handler available. CUDA.OutOfGPUMemoryError(sz=0) throw_api_error at /home/darsnack/.julia/packages/CUDA/mVgLI/lib/cudadrv/error.jl:103 unknown function (ip: 0x7f4dc4f2d9ff) macro expansion at /home/darsnack/.julia/packages/CUDA/mVgLI/lib/cudadrv/error.jl:115 [inlined] cuDevicePrimaryCtxRetain at /home/darsnack/.julia/packages/CUDA/mVgLI/lib/utils/call.jl:26 CuContext at /home/darsnack/.julia/packages/CUDA/mVgLI/lib/cudadrv/context.jl:64 [inlined] context at /home/darsnack/.julia/packages/CUDA/mVgLI/src/state.jl:248 TaskLocalState at /home/darsnack/.julia/packages/CUDA/mVgLI/src/state.jl:50 [inlined] TaskLocalState at /home/darsnack/.julia/packages/CUDA/mVgLI/src/state.jl:50 [inlined] task_local_state! at /home/darsnack/.julia/packages/CUDA/mVgLI/src/state.jl:73 device at /home/darsnack/.julia/packages/CUDA/mVgLI/src/state.jl:207 [inlined] memory_status at /home/darsnack/.julia/packages/CUDA/mVgLI/src/pool.jl:584 showerror at /home/darsnack/.julia/packages/CUDA/mVgLI/src/pool.jl:247 #showerror#760 at ./errorshow.jl:88 showerror##kw at ./errorshow.jl:87 unknown function (ip: 0x7f4dc4f3d3fb) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419 #showerror#762 at ./errorshow.jl:103 showerror##kw at ./errorshow.jl:102 unknown function (ip: 0x7f4dc4f30bdb) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419 #showerror#761 at ./errorshow.jl:96 showerror##kw at ./errorshow.jl:95 unknown function (ip: 0x7f4dc4f307eb) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419 show_exception_stack at ./errorshow.jl:877 display_error at ./client.jl:104 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419 display_error at ./client.jl:107 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined] jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:714 #invokelatest#2 at ./essentials.jl:708 [inlined] invokelatest at ./essentials.jl:706 [inlined] _start at ./client.jl:487 jfptr__start_34289.clone_1 at /home/darsnack/julia-1.6.1/lib/julia/sys.so (unknown line) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined] true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560 repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702 main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:51 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) _start at /home/darsnack/julia-1.6.1/bin/julia (unknown line) ERROR: LoadError: Failed to precompile Metalhead [dbeba491-748d-5e0e-a39e-b530a07fa0cc] to /home/darsnack/.julia/compiled/v1.6/Metalhead/jl_feilmD. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY) @ Base ./loading.jl:1360 [3] compilecache(pkg::Base.PkgId, path::String) @ Base ./loading.jl:1306 [4] _require(pkg::Base.PkgId) @ Base ./loading.jl:1021 [5] require(uuidkey::Base.PkgId) @ Base ./loading.jl:914 [6] require(into::Module, mod::Symbol) @ Base ./loading.jl:901 [7] include(fname::String) @ Base.MainInclude ./client.jl:444 [8] top-level scope @ REPL[2]:1 in expression starting at /home/darsnack/.julia/dev/Metalhead/training/run.jl:5 ```

darsnack commented 3 years ago

Here is the PR I am working on: https://github.com/FluxML/Metalhead.jl/pull/70

An alternate MWE is to checkout the repo, switch to the darsnack/training branch, then

julia>] activate training
julia> include("training/run.jl")
maleadt commented 3 years ago

I can't reproduce this. Could it be that the OOM happened in the precompilation process (when importing Flux), because that also triggers __init__ but without device! having been called before?

Regardless, Flux really shouldn't be calling this from __init__, because it forces download and initialization of CUDA even if you won't use it.

Also, this bug won't happen on master anymore, because (1) after https://github.com/JuliaGPU/CUDA.jl/pull/992 the log configuration won't trigger initialization, and (2) with https://github.com/JuliaGPU/CUDA.jl/pull/987 the loggers aren't installed unless JULIA_DEBUG is set to CUBLAS or CUDNN.

darsnack commented 3 years ago

Yes, actually upon further investigation, I discovered last night that I had one process on which using Flux was being called without device! before. We can close this; it is really a Flux issue cause __init__ triggers CUDA related code on a process where I needed Flux but not CUDA.