search

FluxML / Torch.jl

Sensible extensions for exposing torch in Julia.
Other
211 stars 14 forks source link

Example in README does not work correctly? #21

Closed DilumAluthge closed 4 years ago

DilumAluthge commented 4 years ago

I'm getting a weird result when I run the example in the README. Specifically, the output of tresnet(tip) produces all ones, while the output of resnet(ip) seems to produce the correct output.

Click to show output: ```julia julia> using Metalhead, Metalhead.Flux, Torch julia> resnet = ResNet() ResNet() julia> tresnet = Flux.fmap(Torch.to_tensor, resnet.layers) Chain(Conv((7, 7), 3=>64), MaxPool((3, 3), pad = (1, 1), stride = (2, 2)), Metalhead.ResidualBlock((Conv((1, 1), 64=>64), Conv((3, 3), 64=>64), Conv((1, 1), 64=>256)), (BatchNorm(64), BatchNorm(64), BatchNorm(256)), Chain(Conv((1, 1), 64=>256), BatchNorm(256))), Metalhead.ResidualBlock((Conv((1, 1), 256=>64), Conv((3, 3), 64=>64), Conv((1, 1), 64=>256)), (BatchNorm(64), BatchNorm(64), BatchNorm(256)), identity), Metalhead.ResidualBlock((Conv((1, 1), 256=>64), Conv((3, 3), 64=>64), Conv((1, 1), 64=>256)), (BatchNorm(64), BatchNorm(64), BatchNorm(256)), identity), Metalhead.ResidualBlock((Conv((1, 1), 256=>128), Conv((3, 3), 128=>128), Conv((1, 1), 128=>512)), (BatchNorm(128), BatchNorm(128), BatchNorm(512)), Chain(Conv((1, 1), 256=>512), BatchNorm(512))), Metalhead.ResidualBlock((Conv((1, 1), 512=>128), Conv((3, 3), 128=>128), Conv((1, 1), 128=>512)), (BatchNorm(128), BatchNorm(128), BatchNorm(512)), identity), Metalhead.ResidualBlock((Conv((1, 1), 512=>128), Conv((3, 3), 128=>128), Conv((1, 1), 128=>512)), (BatchNorm(128), BatchNorm(128), BatchNorm(512)), identity), Metalhead.ResidualBlock((Conv((1, 1), 512=>128), Conv((3, 3), 128=>128), Conv((1, 1), 128=>512)), (BatchNorm(128), BatchNorm(128), BatchNorm(512)), identity), Metalhead.ResidualBlock((Conv((1, 1), 512=>256), Conv((3, 3), 256=>256), Conv((1, 1), 256=>1024)), (BatchNorm(256), BatchNorm(256), BatchNorm(1024)), Chain(Conv((1, 1), 512=>1024), BatchNorm(1024))), Metalhead.ResidualBlock((Conv((1, 1), 1024=>256), Conv((3, 3), 256=>256), Conv((1, 1), 256=>1024)), (BatchNorm(256), BatchNorm(256), BatchNorm(1024)), identity), Metalhead.ResidualBlock((Conv((1, 1), 1024=>256), Conv((3, 3), 256=>256), Conv((1, 1), 256=>1024)), (BatchNorm(256), BatchNorm(256), BatchNorm(1024)), identity), Metalhead.ResidualBlock((Conv((1, 1), 1024=>256), Conv((3, 3), 256=>256), Conv((1, 1), 256=>1024)), (BatchNorm(256), BatchNorm(256), BatchNorm(1024)), identity), Metalhead.ResidualBlock((Conv((1, 1), 1024=>256), Conv((3, 3), 256=>256), Conv((1, 1), 256=>1024)), (BatchNorm(256), BatchNorm(256), BatchNorm(1024)), identity), Metalhead.ResidualBlock((Conv((1, 1), 1024=>256), Conv((3, 3), 256=>256), Conv((1, 1), 256=>1024)), (BatchNorm(256), BatchNorm(256), BatchNorm(1024)), identity), Metalhead.ResidualBlock((Conv((1, 1), 1024=>512), Conv((3, 3), 512=>512), Conv((1, 1), 512=>2048)), (BatchNorm(512), BatchNorm(512), BatchNorm(2048)), Chain(Conv((1, 1), 1024=>2048), BatchNorm(2048))), Metalhead.ResidualBlock((Conv((1, 1), 2048=>512), Conv((3, 3), 512=>512), Conv((1, 1), 512=>2048)), (BatchNorm(512), BatchNorm(512), BatchNorm(2048)), identity), Metalhead.ResidualBlock((Conv((1, 1), 2048=>512), Conv((3, 3), 512=>512), Conv((1, 1), 512=>2048)), (BatchNorm(512), BatchNorm(512), BatchNorm(2048)), identity), MeanPool((7, 7), pad = (0, 0, 0, 0), stride = (7, 7)), #103, Dense(2048, 1000), softmax) julia> ip = rand(Float32, 224, 224, 3, 1) 224×224×3×1 Array{Float32,4}: [:, :, 1, 1] = 0.788864 0.0058502 0.890958 0.549588 … 0.235041 0.603126 0.674397 0.999311 0.985704 0.799772 0.817769 0.351721 0.891382 0.11919 0.213889 0.912991 0.21946 0.198282 0.921315 0.652733 0.398499 0.0130252 0.218439 0.166911 0.913552 0.0332044 0.450518 0.264798 0.486238 0.0451902 0.00599992 0.389088 0.418433 0.925302 0.29043 0.589826 0.941069 0.118208 0.26286 … 0.285611 0.375672 0.72936 0.0216198 0.421899 0.46768 0.2434 0.734571 0.787353 0.187157 ⋮ ⋱ 0.617242 0.994453 0.417432 0.033021 0.287248 0.0748078 0.450582 0.826242 0.431456 0.615832 0.723868 0.877138 0.425455 0.331793 0.798118 0.904967 0.443702 0.220164 0.240615 0.498799 0.640266 0.728871 0.60007 0.44531 0.922485 … 0.0371425 0.618428 0.768098 0.517763 0.416089 0.681586 0.944767 0.753302 0.349325 0.896748 0.0506959 0.0892867 0.841178 0.988841 0.561589 0.532282 0.988862 0.166737 0.807801 0.156191 0.388135 0.310189 0.995949 0.141318 [:, :, 2, 1] = 0.807599 0.497326 0.69302 0.0967755 0.415334 … 0.0921698 0.725908 0.440099 0.958504 0.846646 0.587914 0.699806 0.889055 0.808845 0.307114 0.895731 0.033898 0.379116 0.284471 0.31531 0.686093 0.835778 0.28961 0.428077 0.59477 0.736368 0.0927321 0.260409 0.920969 0.873114 0.303251 0.66144 0.81445 0.731043 0.80972 0.283928 0.168506 0.441939 0.239633 0.127432 0.372218 0.587221 0.237972 0.46188 0.805081 … 0.433773 0.564939 0.475007 0.985872 0.212017 0.551498 0.727321 0.762514 0.0967743 0.186588 0.515469 ⋮ ⋱ 0.460097 0.300087 0.104347 0.765258 0.93407 0.518985 0.765961 0.993958 0.325239 0.0369684 0.454788 0.969992 0.540689 0.581669 0.479602 0.219732 0.388296 0.962162 0.0700796 0.250489 0.357813 0.624863 0.260669 0.66425 0.174929 0.0928007 0.00651288 0.378307 0.390218 … 0.620427 0.236085 0.84049 0.771641 0.900496 0.207622 0.264957 0.948786 0.69516 0.238745 0.34208 0.522268 0.918121 0.848074 0.248188 0.966213 0.664656 0.382861 0.553808 0.986438 0.0881685 0.435936 0.0266194 0.385546 0.30548 0.220695 0.260787 [:, :, 3, 1] = 0.218241 0.884007 0.809854 0.412883 … 0.116144 0.181158 0.997082 0.587093 0.739195 0.690466 0.438511 0.4009 0.634617 0.958583 0.289274 0.593044 0.455332 0.902924 0.554916 0.109829 0.232312 0.303047 0.11677 0.841859 0.284147 0.135607 0.40055 0.719077 0.248166 0.906829 0.407552 0.585733 0.910292 0.830691 0.878018 0.148402 0.0317982 0.918101 0.197592 … 0.591174 0.586109 0.265249 0.922237 0.156719 0.126242 0.427481 0.143738 0.0413144 0.00661194 ⋮ ⋱ 0.914771 0.521671 0.289128 0.320104 0.677098 0.939435 0.62052 0.375757 0.734876 0.482571 0.795617 0.626005 0.399592 0.226236 0.0812535 0.675505 0.135298 0.208895 0.873382 0.335736 0.149947 0.489477 0.527099 0.0601703 0.927838 … 0.0823067 0.0535049 0.70316 0.979313 0.255928 0.907132 0.453926 0.779097 0.499022 0.122352 0.958179 0.261293 0.340035 0.131347 0.335443 0.773852 0.70852 0.0084312 0.452387 0.029972 0.159452 0.906282 0.84314 0.197022 julia> tip = tensor(ip, dev = 0) 224×224×3×1 Tensor{Float32,4}: [:, :, 1, 1] = 0.788864 0.0058502 0.890958 0.549588 … 0.235041 0.603126 0.674397 0.999311 0.985704 0.799772 0.817769 0.351721 0.891382 0.11919 0.213889 0.912991 0.21946 0.198282 0.921315 0.652733 0.398499 0.0130252 0.218439 0.166911 0.913552 0.0332044 0.450518 0.264798 0.486238 0.0451902 0.00599992 0.389088 0.418433 0.925302 0.29043 0.589826 0.941069 0.118208 0.26286 … 0.285611 0.375672 0.72936 0.0216198 0.421899 0.46768 0.2434 0.734571 0.787353 0.187157 ⋮ ⋱ 0.617242 0.994453 0.417432 0.033021 0.287248 0.0748078 0.450582 0.826242 0.431456 0.615832 0.723868 0.877138 0.425455 0.331793 0.798118 0.904967 0.443702 0.220164 0.240615 0.498799 0.640266 0.728871 0.60007 0.44531 0.922485 … 0.0371425 0.618428 0.768098 0.517763 0.416089 0.681586 0.944767 0.753302 0.349325 0.896748 0.0506959 0.0892867 0.841178 0.988841 0.561589 0.532282 0.988862 0.166737 0.807801 0.156191 0.388135 0.310189 0.995949 0.141318 [:, :, 2, 1] = 0.807599 0.497326 0.69302 0.0967755 0.415334 … 0.0921698 0.725908 0.440099 0.958504 0.846646 0.587914 0.699806 0.889055 0.808845 0.307114 0.895731 0.033898 0.379116 0.284471 0.31531 0.686093 0.835778 0.28961 0.428077 0.59477 0.736368 0.0927321 0.260409 0.920969 0.873114 0.303251 0.66144 0.81445 0.731043 0.80972 0.283928 0.168506 0.441939 0.239633 0.127432 0.372218 0.587221 0.237972 0.46188 0.805081 … 0.433773 0.564939 0.475007 0.985872 0.212017 0.551498 0.727321 0.762514 0.0967743 0.186588 0.515469 ⋮ ⋱ 0.460097 0.300087 0.104347 0.765258 0.93407 0.518985 0.765961 0.993958 0.325239 0.0369684 0.454788 0.969992 0.540689 0.581669 0.479602 0.219732 0.388296 0.962162 0.0700796 0.250489 0.357813 0.624863 0.260669 0.66425 0.174929 0.0928007 0.00651288 0.378307 0.390218 … 0.620427 0.236085 0.84049 0.771641 0.900496 0.207622 0.264957 0.948786 0.69516 0.238745 0.34208 0.522268 0.918121 0.848074 0.248188 0.966213 0.664656 0.382861 0.553808 0.986438 0.0881685 0.435936 0.0266194 0.385546 0.30548 0.220695 0.260787 [:, :, 3, 1] = 0.218241 0.884007 0.809854 0.412883 … 0.116144 0.181158 0.997082 0.587093 0.739195 0.690466 0.438511 0.4009 0.634617 0.958583 0.289274 0.593044 0.455332 0.902924 0.554916 0.109829 0.232312 0.303047 0.11677 0.841859 0.284147 0.135607 0.40055 0.719077 0.248166 0.906829 0.407552 0.585733 0.910292 0.830691 0.878018 0.148402 0.0317982 0.918101 0.197592 … 0.591174 0.586109 0.265249 0.922237 0.156719 0.126242 0.427481 0.143738 0.0413144 0.00661194 ⋮ ⋱ 0.914771 0.521671 0.289128 0.320104 0.677098 0.939435 0.62052 0.375757 0.734876 0.482571 0.795617 0.626005 0.399592 0.226236 0.0812535 0.675505 0.135298 0.208895 0.873382 0.335736 0.149947 0.489477 0.527099 0.0601703 0.927838 … 0.0823067 0.0535049 0.70316 0.979313 0.255928 0.907132 0.453926 0.779097 0.499022 0.122352 0.958179 0.261293 0.340035 0.131347 0.335443 0.773852 0.70852 0.0084312 0.452387 0.029972 0.159452 0.906282 0.84314 0.197022 julia> resnet(ip) 1000×1 Array{Float32,2}: 0.00056996325 0.0009985788 0.0011034772 0.0009276171 0.00097143865 0.0009628762 0.0007680057 ⋮ 0.0004784039 0.00051957194 0.0007831323 0.0010429893 0.00080520276 0.0014603696 0.0017016815 julia> tresnet(tip) 1000×1 Tensor{Float32,2}: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 ⋮ 1.0 1.0 1.0 1.0 1.0 1.0 1.0 julia> ip == convert(Array, tip) true julia> versioninfo(; verbose = true) Julia Version 1.4.1 Commit 381693d3df* (2020-04-14 17:20 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) Ubuntu 18.04.4 LTS uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 CPU: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz: speed user nice sys idle irq #1-24 2601 MHz 36322372 s 953 s 8051028 s 694276948 s 0 s Memory: 187.5776710510254 GB (173123.50390625 MB free) Uptime: 308098.0 sec Load Avg: 2.6279296875 1.83740234375 1.54638671875 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-8.0.1 (ORCJIT, skylake) Environment: LD_LIBRARY_PATH = /gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/libcutensor/10.2/lib/10.2:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/cudnn/7.6.5/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/src/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64:/.singularity.d/libs C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/cudnn/7.6.5/include JAVA_HOME = /gpfs/runtime/opt/java/8u111 USER_PATH = /users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/cuda/10.2/cuda/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME = /users/daluthge CUDA_HOME = /gpfs/runtime/opt/cuda/10.2/cuda CPATH = /gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/libcutensor/10.2/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/cudnn/7.6.5/include:/gpfs/runtime/opt/cuda/10.2/cuda/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US LIBRARY_PATH = /gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/libcutensor/10.2/lib/10.2:/gpfs/runtime/opt/cudnn/7.6.5/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 MODULEHOME = /gpfs/runtime/pymodules QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins TERM = xterm-256color LD_RUN_PATH = /gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/libcutensor/10.2/lib/10.2:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/cudnn/7.6.5/lib64:/gpfs/runtime/opt/cudnn/7.6.5/lib:/gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1: MODULEPATH = /gpfs/runtime/modulefiles IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include PATH = /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig INCLUDE_PATH = /gpfs/runtime/opt/libcutensor/10.2/include:/gpfs/runtime/opt/cudnn/7.6.5/include julia> import Pkg julia> Pkg.status() Status `~/.julia/environments/v1.4/Project.toml` [dbeba491] Metalhead v0.5.0 [6a2ea274] Torch v0.1.0 julia> Pkg.status(; mode = Pkg.PKGMODE_MANIFEST) Status `~/.julia/environments/v1.4/Manifest.toml` [621f4979] AbstractFFTs v0.5.0 [1520ce14] AbstractTrees v0.3.3 [79e6a3ab] Adapt v1.0.1 [4c555306] ArrayLayouts v0.2.6 [13072b0f] AxisAlgorithms v1.0.0 [39de3d68] AxisArrays v0.4.3 [fbb218c0] BSON v0.2.6 [b99e7846] BinaryProvider v0.5.9 [fa961155] CEnum v0.3.0 [3895d2a7] CUDAapi v4.0.0 [c5f51814] CUDAdrv v6.3.0 [be33ccc6] CUDAnative v3.1.0 [aafaddc9] CatIndices v0.2.1 [da1fd8a2] CodeTracking v0.5.11 [944b1d66] CodecZlib v0.7.0 [19ecbf4d] Codecs v0.5.0 [3da002f7] ColorTypes v0.9.1 [c3611d14] ColorVectorSpace v0.8.5 [5ae59095] Colors v0.11.2 [bbf7d656] CommonSubexpressions v0.2.0 [34da2185] Compat v3.9.1 [e66e0078] CompilerSupportLibraries_jll v0.3.3+0 [ed09eef8] ComputationalResources v0.3.2 [150eb455] CoordinateTransformations v0.5.1 [f68482b8] Cthulhu v1.0.2 [3a865a2d] CuArrays v2.2.0 [dc8bdbbb] CustomUnitRanges v1.0.0 [9a962f9c] DataAPI v1.3.0 [864edb3b] DataStructures v0.17.15 [163ba53b] DiffResults v1.0.2 [b552c78f] DiffRules v1.0.1 [b4f34e82] Distances v0.8.2 [da5c29d0] EllipsisNotation v0.4.0 [e2ba6199] ExprTools v0.1.1 [4f61f5a4] FFTViews v0.3.1 [7a1cc6ca] FFTW v1.2.1 [f5851436] FFTW_jll v3.3.9+5 [5789e2e9] FileIO v1.3.0 [1a297f60] FillArrays v0.8.9 [53c48c17] FixedPointNumbers v0.7.1 [587475ba] Flux v0.10.4 [f6369f11] ForwardDiff v0.10.10 [0c68f7d7] GPUArrays v3.3.0 [61eb1bfa] GPUCompiler v0.2.0 [a2bd30eb] Graphics v1.0.2 [7869d1d1] IRTools v0.3.2 [bbac6d45] IdentityRanges v0.3.1 [2803e5a7] ImageAxes v0.6.4 [f332f351] ImageContrastAdjustment v0.3.5 [a09fc81d] ImageCore v0.8.14 [51556ac3] ImageDistances v0.2.7 [6a3955dd] ImageFiltering v0.6.11 [bc367c6b] ImageMetadata v0.9.1 [787d08f9] ImageMorphology v0.2.5 [2996bd0c] ImageQualityIndexes v0.1.4 [4e3cecfd] ImageShow v0.2.3 [02fcd773] ImageTransformations v0.8.4 [916415d5] Images v0.22.2 [9b13fd28] IndirectArrays v0.5.1 [1d5cc7b8] IntelOpenMP_jll v2018.0.3+0 [a98d9a8b] Interpolations v0.12.9 [8197267c] IntervalSets v0.5.0 [c8e1da08] IterTools v1.3.0 [e5e0dc1b] Juno v0.8.1 [929cbde3] LLVM v1.4.1 [856f044c] MKL_jll v2019.0.117+2 [1914dd2f] MacroTools v0.5.5 [dbb5928d] MappedArrays v0.2.2 [e89f7d12] Media v0.5.0 [dbeba491] Metalhead v0.5.0 [e1d29d7a] Missings v0.4.3 [e94cdb99] MosaicViews v0.2.2 [872c559c] NNlib v0.6.6 [77ba4419] NaNMath v0.3.3 [6fe1bfb0] OffsetArrays v1.0.4 [efe28fd5] OpenSpecFun_jll v0.5.3+3 [bac558e1] OrderedCollections v1.2.0 [5432bcbf] PaddedViews v0.5.5 [d96e819e] Parameters v0.12.1 [b3c3ace0] RangeArrays v0.3.2 [c84ed2f1] Ratios v0.4.0 [189a3867] Reexport v0.2.0 [ae029012] Requires v1.0.1 [6038ab10] Rotations v0.13.0 [699a6c99] SimpleTraits v0.9.2 [a2af1166] SortingAlgorithms v0.3.1 [276daf66] SpecialFunctions v0.10.0 [90137ffa] StaticArrays v0.12.3 [2913bbd2] StatsBase v0.33.0 [06e1c1a7] TiledIteration v0.2.4 [a759f4b9] TimerOutputs v0.5.5 [6a2ea274] Torch v0.1.0 [c12fb04c] Torch_jll v1.4.0+0 [3bb67fe8] TranscodingStreams v0.9.5 [3a884ed6] UnPack v1.0.0 [efce3f68] WoodburyMatrices v0.5.2 [ddb6d928] YAML v0.3.2 [a5390f91] ZipFile v0.9.1 [83775a58] Zlib_jll v1.2.11+9 [e88e6eb3] Zygote v0.4.20 [700de1a5] ZygoteRules v0.2.0 [2a0f44e3] Base64 [ade2ca70] Dates [8bb1440f] DelimitedFiles [8ba89e20] Distributed [9fa8497b] Future [b77e0a4c] InteractiveUtils [76f85450] LibGit2 [8f399da3] Libdl [37e2e46d] LinearAlgebra [56ddb016] Logging [d6f4376e] Markdown [a63ad114] Mmap [44cfe95a] Pkg [de0858da] Printf [9abbd945] Profile [3fa0cd96] REPL [9a3f8284] Random [ea8e919c] SHA [9e88b42a] Serialization [1a1011a3] SharedArrays [6462fe0b] Sockets [2f01184e] SparseArrays [10745b16] Statistics [8dfed614] Test [cf7118a7] UUIDs [4ec0a83e] Unicode ```
DhairyaLGandhi commented 4 years ago

Oh weird. Taking a look now

DilumAluthge commented 4 years ago

Same problem with VGG19:

Click to show output: ```julia julia> using Metalhead, Metalhead.Flux, Torch julia> vgg = VGG19() VGG19() julia> tvgg = Flux.fmap(Torch.to_tensor, vgg) VGG19() julia> ip = rand(Float32, 224, 224, 3, 1) 224×224×3×1 Array{Float32,4}: [:, :, 1, 1] = 0.498491 0.407769 0.780371 0.981334 … 0.414955 0.571731 0.666906 0.50192 0.838953 0.524446 0.604871 0.295441 0.993992 0.764641 0.326346 0.378683 0.0117905 0.444573 0.885431 0.190656 0.314631 0.061934 0.748433 0.285714 0.998341 0.951807 0.299708 0.836227 0.879701 0.93355 0.332404 0.961252 0.330685 0.826398 0.240245 0.583209 0.3256 0.135303 0.963617 … 0.0623926 0.219541 0.204681 0.112235 0.390018 0.375054 0.208431 0.0290717 0.558152 0.0957671 ⋮ ⋱ 0.0434773 0.994421 0.541314 0.245272 0.192545 0.433185 0.155571 0.893189 0.232502 0.0487102 0.481045 0.48532 0.836903 0.630077 0.170602 0.231163 0.166599 0.106811 0.48701 0.0837305 0.0131342 0.766904 0.0659106 0.920495 0.854426 … 0.867676 0.433635 0.62029 0.568807 0.0801383 0.06183 0.740401 0.780359 0.950305 0.0280939 0.0124248 0.998498 0.622203 0.576715 0.149641 0.558667 0.493478 0.294769 0.182792 0.0179323 0.225979 0.26827 0.101056 0.0529317 [:, :, 2, 1] = 0.861143 0.497205 0.863553 0.207238 … 0.0977297 0.0136046 0.606403 0.297435 0.300496 0.557261 0.386956 0.387526 0.872755 0.787488 0.964823 0.00495851 0.525722 0.806817 0.664591 0.705213 0.135782 0.683599 0.817782 0.138909 0.0763923 0.0851334 0.629516 0.338233 0.80776 0.0761968 0.538631 0.190833 0.192458 0.473393 0.349492 0.337013 0.172399 0.650229 0.825923 … 0.659838 0.135936 0.0633227 0.330748 0.774329 0.52036 0.356899 0.85589 0.58197 0.86568 ⋮ ⋱ 0.555386 0.990589 0.494161 0.649997 0.629331 0.691182 0.649388 0.905025 0.153197 0.395096 0.104524 0.461473 0.117742 0.791419 0.296676 0.147374 0.274984 0.896856 0.250116 0.82962 0.436349 0.937658 0.119136 0.94655 0.530961 … 0.138572 0.66508 0.196571 0.679097 0.164798 0.726963 0.545807 0.425591 0.42864 0.627767 0.719939 0.031505 0.912476 0.0288508 0.0972805 0.936565 0.0500817 0.677535 0.0545363 0.705661 0.513347 0.384348 0.202991 0.58372 [:, :, 3, 1] = 0.377408 0.435672 0.204086 … 0.192947 0.679851 0.793043 0.875468 0.709507 0.669047 0.637288 0.245621 0.151128 0.750164 0.356844 0.559717 0.973876 0.997191 0.0237378 0.233072 0.192858 0.742437 0.638844 0.941153 0.545791 0.609829 0.865038 0.103989 0.7745 0.0810506 0.719562 0.179504 0.487095 0.783669 0.757483 0.25499 0.257945 0.127502 0.416734 … 0.480713 0.419331 0.892056 0.978448 0.250991 0.80993 0.767973 0.737972 0.429088 0.517631 0.553251 ⋮ ⋱ ⋮ 0.687455 0.640948 0.717372 0.0440218 0.340633 0.858809 0.663277 0.34785 0.755911 0.998059 0.450204 0.319235 0.162414 0.890301 0.322903 0.511888 0.253829 0.454762 0.868327 0.0928278 0.300161 0.92249 0.246458 0.641859 … 0.469886 0.310537 0.268333 0.611962 0.041773 0.487632 0.260817 0.15195 0.229596 0.790443 0.433225 0.216284 0.527045 0.0919229 0.697979 0.835644 0.474768 0.924436 0.830612 0.686848 0.208704 0.377592 0.17417 0.596311 0.44356 julia> tip = tensor(ip, dev = 0) 224×224×3×1 Tensor{Float32,4}: [:, :, 1, 1] = 0.498491 0.407769 0.780371 0.981334 … 0.414955 0.571731 0.666906 0.50192 0.838953 0.524446 0.604871 0.295441 0.993992 0.764641 0.326346 0.378683 0.0117905 0.444573 0.885431 0.190656 0.314631 0.061934 0.748433 0.285714 0.998341 0.951807 0.299708 0.836227 0.879701 0.93355 0.332404 0.961252 0.330685 0.826398 0.240245 0.583209 0.3256 0.135303 0.963617 … 0.0623926 0.219541 0.204681 0.112235 0.390018 0.375054 0.208431 0.0290717 0.558152 0.0957671 ⋮ ⋱ 0.0434773 0.994421 0.541314 0.245272 0.192545 0.433185 0.155571 0.893189 0.232502 0.0487102 0.481045 0.48532 0.836903 0.630077 0.170602 0.231163 0.166599 0.106811 0.48701 0.0837305 0.0131342 0.766904 0.0659106 0.920495 0.854426 … 0.867676 0.433635 0.62029 0.568807 0.0801383 0.06183 0.740401 0.780359 0.950305 0.0280939 0.0124248 0.998498 0.622203 0.576715 0.149641 0.558667 0.493478 0.294769 0.182792 0.0179323 0.225979 0.26827 0.101056 0.0529317 [:, :, 2, 1] = 0.861143 0.497205 0.863553 0.207238 … 0.0977297 0.0136046 0.606403 0.297435 0.300496 0.557261 0.386956 0.387526 0.872755 0.787488 0.964823 0.00495851 0.525722 0.806817 0.664591 0.705213 0.135782 0.683599 0.817782 0.138909 0.0763923 0.0851334 0.629516 0.338233 0.80776 0.0761968 0.538631 0.190833 0.192458 0.473393 0.349492 0.337013 0.172399 0.650229 0.825923 … 0.659838 0.135936 0.0633227 0.330748 0.774329 0.52036 0.356899 0.85589 0.58197 0.86568 ⋮ ⋱ 0.555386 0.990589 0.494161 0.649997 0.629331 0.691182 0.649388 0.905025 0.153197 0.395096 0.104524 0.461473 0.117742 0.791419 0.296676 0.147374 0.274984 0.896856 0.250116 0.82962 0.436349 0.937658 0.119136 0.94655 0.530961 … 0.138572 0.66508 0.196571 0.679097 0.164798 0.726963 0.545807 0.425591 0.42864 0.627767 0.719939 0.031505 0.912476 0.0288508 0.0972805 0.936565 0.0500817 0.677535 0.0545363 0.705661 0.513347 0.384348 0.202991 0.58372 [:, :, 3, 1] = 0.377408 0.435672 0.204086 … 0.192947 0.679851 0.793043 0.875468 0.709507 0.669047 0.637288 0.245621 0.151128 0.750164 0.356844 0.559717 0.973876 0.997191 0.0237378 0.233072 0.192858 0.742437 0.638844 0.941153 0.545791 0.609829 0.865038 0.103989 0.7745 0.0810506 0.719562 0.179504 0.487095 0.783669 0.757483 0.25499 0.257945 0.127502 0.416734 … 0.480713 0.419331 0.892056 0.978448 0.250991 0.80993 0.767973 0.737972 0.429088 0.517631 0.553251 ⋮ ⋱ ⋮ 0.687455 0.640948 0.717372 0.0440218 0.340633 0.858809 0.663277 0.34785 0.755911 0.998059 0.450204 0.319235 0.162414 0.890301 0.322903 0.511888 0.253829 0.454762 0.868327 0.0928278 0.300161 0.92249 0.246458 0.641859 … 0.469886 0.310537 0.268333 0.611962 0.041773 0.487632 0.260817 0.15195 0.229596 0.790443 0.433225 0.216284 0.527045 0.0919229 0.697979 0.835644 0.474768 0.924436 0.830612 0.686848 0.208704 0.377592 0.17417 0.596311 0.44356 julia> vgg(ip) 1000×1 Array{Float32,2}: 0.00015968765 0.001315148 0.0005566572 0.0014204705 0.0019615644 0.0017357047 0.006765445 ⋮ 2.6461603f-5 0.00013513313 0.00019892212 0.0001468909 0.00011562854 0.00018346732 0.019745728 julia> tvgg(tip) 1000×1 Tensor{Float32,2}: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 ⋮ 1.0 1.0 1.0 1.0 1.0 1.0 1.0 julia> ip == convert(Array, tip) true julia> versioninfo(; verbose = true) Julia Version 1.4.1 Commit 381693d3df* (2020-04-14 17:20 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) Ubuntu 18.04.4 LTS uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 CPU: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz: speed user nice sys idle irq #1-24 2601 MHz 36399409 s 953 s 8075124 s 696277046 s 0 s Memory: 187.5776710510254 GB (172788.265625 MB free) Uptime: 308974.0 sec Load Avg: 1.55615234375 1.5478515625 1.50830078125 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-8.0.1 (ORCJIT, skylake) Environment: LD_LIBRARY_PATH = /gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/libcutensor/10.2/lib/10.2:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/cudnn/7.6.5/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/src/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64:/.singularity.d/libs C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/cudnn/7.6.5/include JAVA_HOME = /gpfs/runtime/opt/java/8u111 USER_PATH = /users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/cuda/10.2/cuda/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin HOME = /users/daluthge CUDA_HOME = /gpfs/runtime/opt/cuda/10.2/cuda CPATH = /gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/libcutensor/10.2/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/cudnn/7.6.5/include:/gpfs/runtime/opt/cuda/10.2/cuda/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US LIBRARY_PATH = /gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/libcutensor/10.2/lib/10.2:/gpfs/runtime/opt/cudnn/7.6.5/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 MODULEHOME = /gpfs/runtime/pymodules QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins TERM = xterm-256color LD_RUN_PATH = /gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/libcutensor/10.2/lib/10.2:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/cudnn/7.6.5/lib64:/gpfs/runtime/opt/cudnn/7.6.5/lib:/gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1: MODULEPATH = /gpfs/runtime/modulefiles IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include PATH = /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig INCLUDE_PATH = /gpfs/runtime/opt/libcutensor/10.2/include:/gpfs/runtime/opt/cudnn/7.6.5/include julia> import Pkg julia> Pkg.status() Status `~/.julia/environments/v1.4/Project.toml` [dbeba491] Metalhead v0.5.0 [6a2ea274] Torch v0.1.0 julia> Pkg.status(; mode = Pkg.PKGMODE_MANIFEST) Status `~/.julia/environments/v1.4/Manifest.toml` [621f4979] AbstractFFTs v0.5.0 [1520ce14] AbstractTrees v0.3.3 [79e6a3ab] Adapt v1.0.1 [4c555306] ArrayLayouts v0.2.6 [13072b0f] AxisAlgorithms v1.0.0 [39de3d68] AxisArrays v0.4.3 [fbb218c0] BSON v0.2.6 [b99e7846] BinaryProvider v0.5.9 [fa961155] CEnum v0.3.0 [3895d2a7] CUDAapi v4.0.0 [c5f51814] CUDAdrv v6.3.0 [be33ccc6] CUDAnative v3.1.0 [aafaddc9] CatIndices v0.2.1 [da1fd8a2] CodeTracking v0.5.11 [944b1d66] CodecZlib v0.7.0 [19ecbf4d] Codecs v0.5.0 [3da002f7] ColorTypes v0.9.1 [c3611d14] ColorVectorSpace v0.8.5 [5ae59095] Colors v0.11.2 [bbf7d656] CommonSubexpressions v0.2.0 [34da2185] Compat v3.9.1 [e66e0078] CompilerSupportLibraries_jll v0.3.3+0 [ed09eef8] ComputationalResources v0.3.2 [150eb455] CoordinateTransformations v0.5.1 [f68482b8] Cthulhu v1.0.2 [3a865a2d] CuArrays v2.2.0 [dc8bdbbb] CustomUnitRanges v1.0.0 [9a962f9c] DataAPI v1.3.0 [864edb3b] DataStructures v0.17.15 [163ba53b] DiffResults v1.0.2 [b552c78f] DiffRules v1.0.1 [b4f34e82] Distances v0.8.2 [da5c29d0] EllipsisNotation v0.4.0 [e2ba6199] ExprTools v0.1.1 [4f61f5a4] FFTViews v0.3.1 [7a1cc6ca] FFTW v1.2.1 [f5851436] FFTW_jll v3.3.9+5 [5789e2e9] FileIO v1.3.0 [1a297f60] FillArrays v0.8.9 [53c48c17] FixedPointNumbers v0.7.1 [587475ba] Flux v0.10.4 [f6369f11] ForwardDiff v0.10.10 [0c68f7d7] GPUArrays v3.3.0 [61eb1bfa] GPUCompiler v0.2.0 [a2bd30eb] Graphics v1.0.2 [7869d1d1] IRTools v0.3.2 [bbac6d45] IdentityRanges v0.3.1 [2803e5a7] ImageAxes v0.6.4 [f332f351] ImageContrastAdjustment v0.3.5 [a09fc81d] ImageCore v0.8.14 [51556ac3] ImageDistances v0.2.7 [6a3955dd] ImageFiltering v0.6.11 [bc367c6b] ImageMetadata v0.9.1 [787d08f9] ImageMorphology v0.2.5 [2996bd0c] ImageQualityIndexes v0.1.4 [4e3cecfd] ImageShow v0.2.3 [02fcd773] ImageTransformations v0.8.4 [916415d5] Images v0.22.2 [9b13fd28] IndirectArrays v0.5.1 [1d5cc7b8] IntelOpenMP_jll v2018.0.3+0 [a98d9a8b] Interpolations v0.12.9 [8197267c] IntervalSets v0.5.0 [c8e1da08] IterTools v1.3.0 [e5e0dc1b] Juno v0.8.1 [929cbde3] LLVM v1.4.1 [856f044c] MKL_jll v2019.0.117+2 [1914dd2f] MacroTools v0.5.5 [dbb5928d] MappedArrays v0.2.2 [e89f7d12] Media v0.5.0 [dbeba491] Metalhead v0.5.0 [e1d29d7a] Missings v0.4.3 [e94cdb99] MosaicViews v0.2.2 [872c559c] NNlib v0.6.6 [77ba4419] NaNMath v0.3.3 [6fe1bfb0] OffsetArrays v1.0.4 [efe28fd5] OpenSpecFun_jll v0.5.3+3 [bac558e1] OrderedCollections v1.2.0 [5432bcbf] PaddedViews v0.5.5 [d96e819e] Parameters v0.12.1 [b3c3ace0] RangeArrays v0.3.2 [c84ed2f1] Ratios v0.4.0 [189a3867] Reexport v0.2.0 [ae029012] Requires v1.0.1 [6038ab10] Rotations v0.13.0 [699a6c99] SimpleTraits v0.9.2 [a2af1166] SortingAlgorithms v0.3.1 [276daf66] SpecialFunctions v0.10.0 [90137ffa] StaticArrays v0.12.3 [2913bbd2] StatsBase v0.33.0 [06e1c1a7] TiledIteration v0.2.4 [a759f4b9] TimerOutputs v0.5.5 [6a2ea274] Torch v0.1.0 [c12fb04c] Torch_jll v1.4.0+0 [3bb67fe8] TranscodingStreams v0.9.5 [3a884ed6] UnPack v1.0.0 [efce3f68] WoodburyMatrices v0.5.2 [ddb6d928] YAML v0.3.2 [a5390f91] ZipFile v0.9.1 [83775a58] Zlib_jll v1.2.11+9 [e88e6eb3] Zygote v0.4.20 [700de1a5] ZygoteRules v0.2.0 [2a0f44e3] Base64 [ade2ca70] Dates [8bb1440f] DelimitedFiles [8ba89e20] Distributed [9fa8497b] Future [b77e0a4c] InteractiveUtils [76f85450] LibGit2 [8f399da3] Libdl [37e2e46d] LinearAlgebra [56ddb016] Logging [d6f4376e] Markdown [a63ad114] Mmap [44cfe95a] Pkg [de0858da] Printf [9abbd945] Profile [3fa0cd96] REPL [9a3f8284] Random [ea8e919c] SHA [9e88b42a] Serialization [1a1011a3] SharedArrays [6462fe0b] Sockets [2f01184e] SparseArrays [10745b16] Statistics [8dfed614] Test [cf7118a7] UUIDs [4ec0a83e] Unicode ```
DhairyaLGandhi commented 4 years ago

Interesting, could you run specifically with the Chain object instead of the struct?

DilumAluthge commented 4 years ago

Interesting, could you run specifically with the Chain object instead of the struct?

Could you explain how to do that?

DhairyaLGandhi commented 4 years ago

tresnet = Flux.fmap(Torch.to_tensor, resnet.layers)

Note the .layers field

DhairyaLGandhi commented 4 years ago

Hmm, not able to reproduce.

julia> tresnet2(tip)
1000×5 Tensor{Float32,2}:
 0.000490387  2.8881e-18   …  9.43812e-11  2.06312e-6
 2.59334e-14  1.0             1.0          0.0783882
 0.684608     5.45478e-13     1.51907e-9   5.58125e-5
 0.314902     6.37972e-18     1.99655e-8   0.000203497
 2.20994e-15  1.11982e-15     4.3811e-22   0.92135
 5.03003e-17  0.0756458    …  0.000474865  3.06938e-8
DilumAluthge commented 4 years ago

I tried both of these:

tresnet = Flux.fmap(Torch.to_tensor, resnet)
tresnet = Flux.fmap(Torch.to_tensor, resnet.layers)

Unfortunately, in both of these cases, tresnet(tip) returns a vector of all ones.

DilumAluthge commented 4 years ago

Also, the dimension of tresnet(tip) seems wrong...

Click to show output: ```julia julia> typeof(ip) Array{Float32,4} julia> ndims(ip) 4 julia> size(ip) (224, 224, 3, 1) julia> typeof(tip) Tensor{Float32,4} julia> ndims(tip) 4 julia> size(tip) (224, 224, 3, 1) julia> typeof(resnet(ip)) Array{Float32,2} julia> ndims(resnet(ip)) 2 julia> size(resnet(ip)) (1000, 1) julia> typeof(tresnet(tip)) Tensor{Float32,2} julia> ndims(tresnet(tip)) 2 julia> size(tresnet(tip)) (1000, 1) ```
DhairyaLGandhi commented 4 years ago

Does converting it back to an Array also yield ones? I just want to rule out that it's just display that's causing this, however, the input seems alright

DhairyaLGandhi commented 4 years ago

Ah, yes, that should be fixed on master.

DilumAluthge commented 4 years ago 
julia> tresnet(tip)
1000×1 Tensor{Float32,2}:
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 ⋮
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0

julia> convert(Array, tresnet(tip))
1000×1 Array{Float32,2}:
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 ⋮
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
DilumAluthge commented 4 years ago

Ah, yes, that should be fixed on master.

Let me do ] add Torch#master and see if I can reproduce.

DilumAluthge commented 4 years ago

So... on Torch#master, I get a MethodError:

julia> using Metalhead, Metalhead.Flux, Torch

julia> resnet = ResNet();

julia> tresnet = Flux.fmap(Torch.to_tensor, resnet.layers);

julia> ip = rand(Float32, 224, 224, 3, 1);

julia> tip = tensor(ip, dev = 0);

julia> resnet(ip)
1000×1 Array{Float32,2}:
 0.00087634224
 0.000810427
 0.0006268254
 0.0007971713
 0.001212693
 0.0013046135
 0.0007224348
 ⋮
 0.00033712923
 0.0006899684
 0.00080008985
 0.0007252848
 0.0007919692
 0.0016419729
 0.001108708

julia> tresnet(tip)
ERROR: MethodError: no method matching conv(::Tensor{Float32,4}, ::Tensor{Float32,4}, ::Tensor{Float32,1}, ::DenseConvDims{2,(1, 1),64,64,(1, 1),(0, 0, 0, 0),(1, 1),false}; stride=1, pad=0, dilation=1)
Closest candidates are:
  conv(::Tensor{xT,N}, ::Tensor{T,N}, ::Tensor{T,N} where N, ::DenseConvDims{M,K,C_in,C_out,S,P,D,F}; stride, pad, dilation) where {T, N, xT, M, K, C_in, C_out, S, P, D, F} at /users/daluthge/.julia/packages/Torch/W539X/src/nnlib.jl:10
  conv(::Tensor, ::Tensor, ::DenseConvDims; stride, pad, dilation) at /users/daluthge/.julia/packages/Torch/W539X/src/nnlib.jl:15
  conv(::Any, ::AbstractArray{T,N}; stride, pad, dilation, flipped) where {T, N} at /users/daluthge/.julia/packages/NNlib/FAI3o/src/conv.jl:174
  ...
Stacktrace:
 [1] conv(::Tensor{Float32,4}, ::Tensor{Float32,4}, ::DenseConvDims{2,(1, 1),64,64,(1, 1),(0, 0, 0, 0),(1, 1),false}; stride::Int64, pad::Int64, dilation::Int64) at /users/daluthge/.julia/packages/Torch/W539X/src/nnlib.jl:16
 [2] conv(::Tensor{Float32,4}, ::Tensor{Float32,4}, ::DenseConvDims{2,(1, 1),64,64,(1, 1),(0, 0, 0, 0),(1, 1),false}) at /users/daluthge/.julia/packages/Torch/W539X/src/nnlib.jl:15
 [3] (::Conv{2,2,typeof(identity),Tensor{Float32,4},Tensor{Float32,1}})(::Tensor{Float32,4}) at /users/daluthge/.julia/packages/Flux/Fj3bt/src/layers/conv.jl:61
 [4] (::Metalhead.ResidualBlock)(::Tensor{Float32,4}) at /users/daluthge/.julia/packages/Metalhead/RZn9O/src/resnet.jl:26
 [5] applychain(::Tuple{Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,MeanPool{2,4},Metalhead.var"#103#104",Dense{typeof(identity),Tensor{Float32,2},Tensor{Float32,1}},typeof(softmax)}, ::Tensor{Float32,4}) at /users/daluthge/.julia/packages/Flux/Fj3bt/src/layers/basic.jl:36 (repeats 3 times)
 [6] (::Chain{Tuple{Conv{2,2,typeof(identity),Tensor{Float32,4},Tensor{Float32,1}},MaxPool{2,2},Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,Metalhead.ResidualBlock,MeanPool{2,4},Metalhead.var"#103#104",Dense{typeof(identity),Tensor{Float32,2},Tensor{Float32,1}},typeof(softmax)}})(::Tensor{Float32,4}) at /users/daluthge/.julia/packages/Flux/Fj3bt/src/layers/basic.jl:38
 [7] top-level scope at REPL[6]:1
DhairyaLGandhi commented 4 years ago

Oh oops, let me check that out. Sorry about that!

DhairyaLGandhi commented 4 years ago

Seems to have broken in #19

DilumAluthge commented 4 years ago

I tried on Torch#eb8f25c, which is before #19 was merged, and I still get the same problem:

julia> tresnet(tip)
1000×1 Tensor{Float32,2}:
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 ⋮
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
 1.0
DhairyaLGandhi commented 4 years ago

Let me try and reproduce on a different machine.

ToucheSir commented 4 years ago

I'm able to reproduce this issue as well, MWE follows:

using NNlib
using Torch

ip = permutedims(Float32[1 2 3])
tip = Torch.to_tensor(ip) # 0 => GPU:0 in Torch

println("Default")
@show op = softmax(ip)

println("Torch")
# This line fails because dims=1
# @show top = softmax(tip) |> collect
@show top = softmax(tip, dims=0) |> collect

@assert op ≈ top

In short, torch dimensions start at 0, so the default dim of 1 was incorrectly calculating softmax along the batch dimension instead of the label dim. This also applies to mean, sum and any other method using dimension indices. @dhairyagandhi96 should this indexing match julia's 1-based dims? If so, I can put in a PR for the aforementioned methods.

DhairyaLGandhi commented 4 years ago

It should, I'd appreciate a PR

It had seemed handled, in the tests I had run earlier. Do add this to the test suite.

DhairyaLGandhi commented 4 years ago

@DilumAluthge could you confirm that omitting the softmax layer produces expected output for you?

DilumAluthge commented 4 years ago

@DilumAluthge could you confirm that omitting the softmax layer produces expected output for you?

Sorry, I'm very new to everything related to Flux, Metalhead, etc.

In the README example, how would I modify the example to omit the softmax layer?

DilumAluthge commented 4 years ago

In short, torch dimensions start at 0, so the default dim of 1 was incorrectly calculating softmax along the batch dimension instead of the label dim. This also applies to mean, sum and any other method using dimension indices. @dhairyagandhi96 should this indexing match julia's 1-based dims? If so, I can put in a PR for the aforementioned methods.

Instead of just patching a certain set of methods, would it be better to have a robust way of automatically converting between Julia's 1-based indexing and Python's/C++'s 0-based indexing whenever you convert a Julia (Flux, Metalhead, etc.) model to a PyTorch model?

ToucheSir commented 4 years ago

Instead of just patching a certain set of methods, would it be better to have a robust way of automatically converting between Julia's 1-based indexing and Python's/C++'s 0-based indexing whenever you convert a Julia (Flux, Metalhead, etc.) model to a PyTorch model?

So, I had a crack at doing just this while trying to make sum and mean work. Turns out it's a much deeper rabbit hole than expected!

To elaborate, the current Tensor implementation does a whole bunch of shape/dimension gymnastics in order to accommodate memory layout differences between Julia and Torch. I ran into this when torch.mean and torch.sum wanted dimensions in reverse order :exploding_head:. This discrepancy appears to be a result of Julia and Torch using different strides, so the most elegant solution would be to use something like Torch.from_blob by default when constructing tensors. Doing so would probably also get rid of all the if t isa TensorMatrix ... reverse(size(t)) and such.

On a more general note, maybe it would help to deduplicate some of the work here with ThArrays.jl? For example, I noticed that they already have a stride-aware tensor constructor (see https://github.com/TuringLang/ThArrays.jl/blob/05bd5ceb2d358dee7be1fe3242d796805313b7e5/csrc/torch_capi_tensor.cpp#L40), a pretty complete device API, etc.

DhairyaLGandhi commented 4 years ago

We can check the API with from_blob, I had intended its use to be more widespread too.

For the sum and so on, see https://github.com/FluxML/Torch.jl/pull/13/commits/e658fa2e3e9f1d743db4629dbb1b88884b92cf76

What do we currently lack from our device API? Some choices to turn off getindex and setindex are really only temporary so we don't falsely see passes when things should in fact fail just for going to fallback methods. Happy to turn those on if helpful.

DhairyaLGandhi commented 4 years ago

In the README example, how would I modify the example to omit the softmax layer?

you just need to call the forward pass with tresnet[1:end-1](...)

DhairyaLGandhi commented 4 years ago

Using from_blob more might be cleaner

ToucheSir commented 4 years ago

For the sum and so on, see e658fa2

I guess the question is whether or not possible to avoid lines like https://github.com/FluxML/Torch.jl/blob/e658fa2e3e9f1d743db4629dbb1b88884b92cf76/src/statistics.jl#L25 and to use the dims as-is (i.e. -1 instead of reversed). Assuming this can be achieved using from_blob, I was able to get pretty close to Base semantics locally: 

# Performs error checking and converts singleton dimensions to tuples
function _normalize_dims(t, dims)
    inds = collect(reduced_indices(t, dims))
    eachindex(inds)[length.(inds) .== 1] .- 1
end

function Statistics.mean(t::Tensor{T,N}; dims = :) where {T,N}
  ptr = Ref(Ptr{Cvoid}())

  if dims isa Colon
    atg_mean(ptr, t.ptr, options[T])
    Tensor{T,0}(ptr[], on(t))
  else
    # To match Julia's behaviour, we always keep dimensions when dims != :
    atg_mean1(ptr, t.ptr, dims, length(dims), true, options[T])
    Tensor{T,N-length(dims)}(ptr[], on(t))
  end
end

function Statistics.sum(t::Tensor{T,N}; dims = :) where {T,N}
  ptr = Ref(Ptr{Cvoid}())

  if dims isa Colon
    atg_sum(ptr, t.ptr, options[T])
    Tensor{T,0}(ptr[], on(t))
  else
    @show dims = _normalize_dims(t, dims)
    atg_sum1(ptr, t.ptr, dims, length(dims), true, options[T])
    Tensor{T,N-length(dims)}(ptr[], on(t))
  end
end
DhairyaLGandhi commented 4 years ago

guess the question is whether or not possible to avoid lines like

Absolutely! Reversing the dimensions every time seems error prone. Ideally we'd have a more automatic solution that can interface with the binary with minimal handling on that side since changes there would be harder to debug.

DhairyaLGandhi commented 4 years ago

Maybe PR the dims change?

ToucheSir commented 4 years ago

Will see if I can find some time later tomorrow, will probably hack the from_blob/tensor constructor changes first so that size reversal can be completely excised from the implementation (no more if isa TensorMatrix..., hopefully...)

DhairyaLGandhi commented 4 years ago

I have a naive from_blob based tensor conversion here https://github.com/dhairyagandhi96/TorchGPU.jl/blob/33878b0dcbbe41dae7c32ce5ab4a645be0292db7/src/TorchGPU.jl#L9 which works pretty well, it catches the dims for CuArrays well, and that should be good as a starting point

ToucheSir commented 4 years ago

Add a look at the c++ -> c wrapper and I think at_from_blob might need to be modified. More specifically, torch complains when a normal Julia array is passed in because the device type is hard-coded to torch::kCUDA. If it's possible to modify at_from_blob so that one can pass a device id as well (and choose the correct type based on said ID), that would be great!

DhairyaLGandhi commented 4 years ago

Yeah, that would be fine.

DhairyaLGandhi commented 4 years ago

should be fixed on master