Trouble Training on Tiny ImageNet

I'm attempting to train a network on the Tiny Imagenet, a subset of ImageNet created for use in the Stanford cs231n course.

I've created a large network (code below), and have tried tweaking its structure and network hyperparameters quite a bit, but I haven't been able to get the network's classification error significantly lower than random (there are 200 image categories in the Tiny ImageNet dataset; random guessing would yield about 99.5% error rate).

Does anyone have any insight into why might this be happening, or some example code that trains a network successfully on the Tiny ImageNet dataset? I hope I am not overlooking something silly.

function [net, info] = train_tiny_imagenet_network(varargin)
% TRAIN_TINY_IMAGENET_NETWORK Train a deep neural network to classify the 
% Tiny Imagenet dataset.
%
% You must run generate_TINY_IMAGENET in the datasets folder with the
% default options in order to prepare the images for this network.

% default optional argument settings
opts.numEpochs = -1;
opts.imdb = 'imdb.mat';

% parsing optional arguments
opts = vl_argparse(opts, varargin);

% load settings to get imdb path
SystemSettings;

% pass execution to the 'train_helper' function to set up network training
[net, info] = train.train_helper(@init_net, ...
    'tiny_imagenet_network', fullfile(datasets_path, 'tiny_imagenet', opts.imdb), ...
    'numEpochs', opts.numEpochs);

function net = init_net()

lr = [.1 2] ;

% Define network
net.layers = {} ;

% Block 1
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(3,3,3,128, 'single'), zeros(1,128,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 1) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(3,3,128,128, 'single'), zeros(1,128,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 1) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
                           'method', 'max', ...
                           'pool', [2 2], ...
                           'stride', 1, ...
                           'pad', [0 1 0 1]) ;

% Block 2
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(3,3,128,128, 'single'), zeros(1,128,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
               'pad', 1) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(3,3,128,128, 'single'), zeros(1,128,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 1) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
                           'method', 'max', ...
                           'pool', [2 2], ...
                           'stride', 1, ...
                           'pad', [0 1 0 1]) ;
% Block 3
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(3,3,128,128,'single'), zeros(1,128,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 1) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
                           'method', 'max', ...
                           'pool', [2 2], ...
                           'stride', 1, ...
                           'pad', [0 1 0 1]) ;

% Block 4
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(3,3,128,128,'single'), zeros(1,128,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 1) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
                           'method', 'max', ...                           
               'pool', [2 2], ...
                           'stride', 1, ...
                           'pad', [0 1 0 1]) ;

% Block 7
net.layers{end+1} = struct('type', 'conv', ...
               'weights', {{0.01*randn(64,64,128,400,'single'), zeros(1,400,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 0) ;
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(1,1,400,400,'single'), zeros(1,400,'single')}}, ...
                           'learningRate', lr, ...                           
                           'stride', 1, ...
                           'pad', 0) ;
net.layers{end+1} = struct('type', 'conv', ...
                           'weights', {{0.01*randn(1,1,400,200,'single'), zeros(1,200,'single')}}, ...
                           'learningRate', lr, ...
                           'stride', 1, ...
                           'pad', 0) ;

% Loss layer
net.layers{end+1} = struct('type', 'softmaxloss') ;

% Meta parameters
net.meta.inputSize = [64 64 3] ;
net.meta.trainOpts.learningRate = 0.000001 * ones(1, 100);
net.meta.trainOpts.weightDecay = 0.000001 ;
net.meta.trainOpts.batchSize = 100 ;
net.meta.trainOpts.numEpochs = numel(net.meta.trainOpts.learningRate) ;

% Fill in default values
net = vl_simplenn_tidy(net) ;

vlfeat / matconvnet

Trouble Training on Tiny ImageNet #1007