Load Experiment

[kadarakos]

I save out a network like so:

e = Experiment(Regressor, layers=(input_layer, hidden1, output_layer), optimize='hf', num_updates=30, verbose='True') e.run(dataset, dataset) e.save('network.dat')

Then when I'm trying to load it back in: network = theanets.Experiment(theanets.Network).load('network.dat')

I get the following error message, and I'm not sure what am I doing wrong. Traceback (most recent call last): File "test.py", line 10, in network = theanets.Experiment(theanets.Network).load('network.dat') File "/usr/local/lib/python2.7/dist-packages/theanets/main.py", line 90, in init self.network = self._build_network(network_class, _kw) File "/usr/local/lib/python2.7/dist-packages/theanets/main.py", line 103, in _build_network return network_class(activation=activation, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theanets/feedforward.py", line 107, in init self.x.tag.test_value = np.random.randn(DEBUG_BATCH_SIZE, layers[0]) TypeError: 'NoneType' object has no attribute 'getitem'

[lmjohns3]

When you create your Experiment class to load in the saved model, you need to pass the same layers=... value as before. The model serialization code could use a little work in this area, but at the moment you have to tell theanets how many layers are in your model before you load it. (The loading should really be renamed to something like load_weights or load_params because that's really what's happening.)

[kadarakos]

Oh, ok thanks. Just as a feedback I would like to add that it would be nice if the save() function would store the layout of the network as well. I now write it out to a dictionary and then read it back in, which is not that big of a hassle of course, just saying that it would be like more "aesthetically" pleasing to me personally. I hope you can understand what I'm trying to say :D. Like the interface would be more "smooth".

2014-04-06 16:24 GMT-04:00 Leif Johnson notifications@github.com:

Closed #18 https://github.com/lmjohns3/theano-nets/issues/18.

Reply to this email directly or view it on GitHubhttps://github.com/lmjohns3/theano-nets/issues/18 .

[lmjohns3]

Ok, I'll reopen this as a feature request.

It's a bit tricky to do, because currently theanets constructs the computation graph before the loading takes place, and then loads the parameters into the computation graph. I think the experiment saving/loading could just store all of the kwargs along with the parameters though.

[lmjohns3]

I think this is working now! If you get a chance to test and it works, please close the issue.

[lmjohns3]

Closing after a month, please reopen if needed.

[varghesealex90]

I have a array which is 1024,100(cols,rows), I want to use Autoencoder with 3 layers (Input,Hidden, Output layer). Should the size of neural n etwork be 1024,N,1024??

[lmjohns3]

Yes. The rows in a dataset are treated as training examples, and the columns are treated as variables.

[varghesealex90]

Thank you for the prompt reply. I am interested in Autoencoders. Is there a in built command to compare the input and the output generated by the network., In other words is there a command to figure out the reconstructional error

[lmjohns3]

There's not a command for that, but it's easy to do yourself:

# ... train up an Autoencoder ...
diff = exp.network.predict(data) - data
euclidean = np.sqrt((diff * diff).sum(axis=1))

Then euclidean will be a vector of the same length as data giving the euclidean distance from the original data to their image in the Autoencoder.

[varghesealex90]

I get such a statement at the end of the training of autoencoder. theanets.trainer:163 patience elapsed, bailing out

Is it something I should worry about

Regards Varghese Alex

On Sat, Dec 20, 2014 at 12:10 AM, Leif Johnson notifications@github.com wrote:

There's not a command for that, but it's easy to do yourself:

... train up an Autoencoder ...

diff = exp.network.predict(data) - data euclidean = np.sqrt((diff * diff).sum(axis=1))

Then euclidean will be a vector of the same length as data giving the euclidean distance from the original data to their image in the Autoencoder.

— Reply to this email directly or view it on GitHub https://github.com/lmjohns3/theano-nets/issues/18#issuecomment-67678496.

[lmjohns3]

No, this is how training ends using any of the SGD-based trainers (NAG, Rprop, RmsProp). It indicates that less than min-improvement relative progress has been made on the validation set for patience parameter updates. You can train for longer by increasing the patience value, or by reducing the min-improvement value.

[varghesealex90]

Thank you Mr. John's for the reply and assistance you have offered

On Mon, Dec 22, 2014 at 10:37 AM, Leif Johnson notifications@github.com wrote:

No, this is how training ends using any of the SGD-based trainers (NAG, Rprop, RmsProp). It indicates that less than min-improvement relative progress has been made on the validation set for patience parameter updates. You can train for longer by increasing the patience value, or by reducing the min-improvement value.

— Reply to this email directly or view it on GitHub https://github.com/lmjohns3/theanets/issues/18#issuecomment-67802708.

[varghesealex90]

I am interested in autoencoders and I prefer to use Layerwise optimization technique. The layerwise technique uses NAG technique at each layer. In NAG we do have an option of setting the learning rate and momentum too. What is the default learning rate and momentum values.

It would be good to know like how small can the learning rate go.

Thanks for the help in advance

Varghese

On Mon, Dec 22, 2014 at 11:26 AM, varghese alex varghesealex90@gmail.com wrote:

Thank you Mr. John's for the reply and assistance you have offered

On Mon, Dec 22, 2014 at 10:37 AM, Leif Johnson notifications@github.com wrote:

No, this is how training ends using any of the SGD-based trainers (NAG, Rprop, RmsProp). It indicates that less than min-improvement relative progress has been made on the validation set for patience parameter updates. You can train for longer by increasing the patience value, or by reducing the min-improvement value.

— Reply to this email directly or view it on GitHub https://github.com/lmjohns3/theanets/issues/18#issuecomment-67802708.

[varghesealex90]

During training are the weights and biases updated one row of training data at a time or one mini batch or after seeing the entire training data . We would like to parallelize the training procedure so as to make training faster. Is there any existing technique that can be implemented in theanets to speed up training.

In case of autoencoders the "cost function" that is minimized is just the difference between the reconstruction and input data ?

Regards Varghese Alex

On Tue, Dec 30, 2014 at 11:17 AM, varghese alex varghesealex90@gmail.com wrote:

I am interested in autoencoders and I prefer to use Layerwise optimization technique. The layerwise technique uses NAG technique at each layer. In NAG we do have an option of setting the learning rate and momentum too. What is the default learning rate and momentum values.

It would be good to know like how small can the learning rate go.

Thanks for the help in advance

Varghese

On Mon, Dec 22, 2014 at 11:26 AM, varghese alex varghesealex90@gmail.com wrote:

Thank you Mr. John's for the reply and assistance you have offered

On Mon, Dec 22, 2014 at 10:37 AM, Leif Johnson notifications@github.com wrote:

No, this is how training ends using any of the SGD-based trainers (NAG, Rprop, RmsProp). It indicates that less than min-improvement relative progress has been made on the validation set for patience parameter updates. You can train for longer by increasing the patience value, or by reducing the min-improvement value.

— Reply to this email directly or view it on GitHub https://github.com/lmjohns3/theanets/issues/18#issuecomment-67802708.

[lmjohns3]

Hi -

I'll answer these questions here, but in the future would you mind posting usage questions like this on the mailing list for the project? You can sign up at https://groups.google.com/forum/#!forum/theanets.

For the learning rate and momentum: the default learning rate is 1e-4, and the default momentum is 0.9. You can see the default values by invoking your training script with the --help flag on the command-line, or by looking at the file theanets/flags.py. I've never seen a learning rate set to lower than 1e-7 in the literature, but you can set it that low if you like. Probably it's not too useful to set it lower than 1e-7, since if you're using 32-bit floats (e.g. for running on a GPU) you'll be near the machine epsilon. http://en.wikipedia.org/wiki/Machine_epsilon

For the training question: the weights and biases are updated after every minibatch. If you are using the logging output during training, then one line gets printed out during training after --train-batches minibatches and parameter updates are computed. For an autoencoder, the cost that is minimized is the squared error between the reconstruction and input, plus any regularizers that are being applied to the model (e.g., --hidden-l1 for a sparse autoencoder, etc.).