Open hpoit opened 8 years ago
@tenthdimension Are you running on some big endian architectures? Or maybe your downloaded data gets corrupted?
Hi Chiyuan. No, running on a little endian, x86 processor, and downloading the four files from http://yann.lecun.com/exdb/mnist/
Chiyuan, where are these files supposed to be in for convert.jl to read them?
datasets = @compat(Dict("train" => ["train-labels-idx1-ubyte","train-images-idx3-ubyte"], "test" => ["t10k-labels-idx1-ubyte","t10k-images-idx3-ubyte"]))
@hpoit You need to run get-mnist.sh
which will download the MNIST data and then run convert.jl to convert them into HDF5 format.
I tried doing that. Let me finish a course and I'll get back to this shortly.
I'm having trouble making convert.jl locate t10k-labels-idx1-ubyte.gz file (which is in my Downloads folder):
julia> include("/Users/Corvus/Google Drive/convert.jl") ERROR: LoadError: SystemError: opening file t10k-labels-idx1-ubyte: No such file or directory in open at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib in open at iostream.jl:99 [inlined code] from /Users/Corvus/Google Drive/convert.jl:13 in anonymous at no file:0 in include at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib in include_from_node1 at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib while loading /Users/Corvus/Google Drive/convert.jl, in expression starting on line 11
I think it worked-- I just changed the directory on convert.jl to my Downloads folder
include("/Users/Corvus/Google Drive/convert.jl") Exporting 10000 digits of size 28 x 28 1:10000... Exporting 60000 digits of size 28 x 28 1:60000...
Now I got this
julia> include("/Users/Corvus/Google Drive/mnist.jl") WARNING: Method definition info(Any...) in module Base at util.jl:334 overwritten in module Logging at /Users/Corvus/.julia/v0.4/Logging/src/Logging.jl:61. WARNING: Method definition warn(Any...) in module Base at util.jl:364 overwritten in module Logging at /Users/Corvus/.julia/v0.4/Logging/src/Logging.jl:61. Configuring Mocha...
- CUDA disabled by default
- Native Ext disabled by default Mocha configured, continue loading module... DefaultBackend = Mocha.CPUBackend 19-Apr 18:11:24:INFO:root:Constructing net MNIST-train on Mocha.CPUBackend... 19-Apr 18:11:24:INFO:root:Topological sorting 8 layers... 19-Apr 18:11:24:INFO:root:Setup layers... HDF5-DIAG: Error detected in HDF5 (1.8.15-patch1) thread 0:
000: H5F.c line 439 in H5Fis_hdf5(): unable open file
major: File accessibilty minor: Not an HDF5 file
001: H5Fint.c line 554 in H5F_is_hdf5(): unable to open file
major: Low-level I/O minor: Unable to initialize object
002: H5FD.c line 993 in H5FD_open(): open failed
major: Virtual File Layer minor: Unable to initialize object
003: H5FDsec2.c line 343 in H5FD_sec2_open(): unable to open file: name = 'data/train.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessibilty minor: Unable to open file ERROR: LoadError: Cannot access file data/train.hdf5 in h5open at /Users/Corvus/.julia/v0.4/HDF5/src/plain.jl:547 in h5open at /Users/Corvus/.julia/v0.4/HDF5/src/plain.jl:557 in setup at /Users/Corvus/.julia/v0.4/Mocha/src/layers/async-hdf5-data.jl:50 in setup at /Users/Corvus/.julia/v0.4/Mocha/src/layers.jl:112 in call at /Users/Corvus/.julia/v0.4/Mocha/src/net.jl:227 in include at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib in include_from_node1 at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib while loading /Users/Corvus/Google Drive/mnist.jl, in expression starting on line 18
line 18 being
net = Net("MNIST-train", backend, [data_layer, common_layers..., loss_layer])
Is "MNIST-train"
supposed to be .../train.hdf5
? I tried changing it for this file directory but it didn't work, so I guess not.
On http://mochajl.readthedocs.org/en/latest/tutorial/mnist.html
Then we will define a data layer, which reads the HDF5 file and provides input for the network:
data_layer = HDF5DataLayer(name="train-data", source="data/train.txt", batch_size=64, shuffle=true)
Why read from train.txt
instead of train.hdf5
? @staticfloat @uschmidt83
@pluskid I know you have a handful so attempting to distribute it a little.
Hi, in the docs it says "Note the source is a simple text file that contains a list of real data files (in this case data/train.hdf5)"
From my brief reading of the above, you haven't used get-mnist.sh
so you may have the hdf5 in directories where Mocha isn't expecting them. I think if you change the path(s) in that txt file to point to your hdf5 file(s), I expect it should work.
I ran get-mnist.sh
and generated train.hdf5
and test.hdf5
with convert.jl
. It was all a directory issue.
Network constructed!
Objective completed. Thanks.
Hi! Need assistance...
include("/.../convert.jl") ERROR: LoadError: AssertionError: ntoh(label_header[1]) == 2049 [inlined code] from /.../convert.jl:17 in anonymous at no file:0 in include at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib in include_from_node1 at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib while loading /.../convert.jl, in expression starting on line 11