Closed hedgy123 closed 7 years ago
Hi there, Thanks for reaching out! I need to be clearer about this, I haven't had time to join together the two scripts yet. I'll get back to you ASAP with an updated answer but for now: init_alpha: -785.866918162 is an error. (<0)
Note that for big magnitutes of alpha mean of tte is same as the complex estimate using log etc
Furthermore
OBS NOT TESTED:
def epoch():
for i in xrange(n_samples):
model.fit(x_train[i,:seq_length[i],:], y_train[i,:seq_length[i],:],
epochs=1,
batch_size=1,
verbose=2
)
But even better debug-mode initially is to simply transform the data to [n_non_masked_samples,1,n_features] (feed in only seen timesteps) to a simple ANN and when that works test the RNN.
Would love to see forks!
There's multiple reasons for NANs to show up but just found a very important:
shift_discrete_padded_features
is currently broken which is supposed to hide target but apparently doesn't. This means that if input is "event" then it's possible to make a perfect prediction, causing exploding gradient
I'm trying to fix it asap
Hi Egil,
Thanks for the update! Here's a fork with the notebook Combined_data_pipeline_and_analysis in examples/keras.
https://github.com/NataliaVConnolly/wtte-rnn-1
The last cell shows an example of training with just one input sequence. It does result in a non-NaN loss, although a very large one (but I didn't optimize the initial alpha or the network config much).
Cheers, Natalia (aka hedgy123 :))
@NataliaVConnolly Sorry for the wait. It took me some time to figure out what was wrong!
shift_discrete_padded_features
that would also lead to NaN (perfect fit) after some training.Check out the new data_pipeline and let me know if you have more questions! :)
Hi Egil,
Thank you so much for making your code available! This is really great stuff.
So in trying to understand better how it all works I tried using the tensorflow.log-extracted data (as in your data_pipeline notebook) as inputs to the network (same config as in your simple_example). Unfortunately I got all nan's as losses:
Model summary:
Results of running model.fit:
I was wondering if you've tried doing the same experiment and if so, whether it worked for you? Thanks so much!