Closed TeaPearce closed 7 years ago
I'm with you. Batch_size >1 is inshallah coming tomorrow in the updated data_pipeline. If you can't wait, the tests gives you a hint: https://github.com/ragulpr/wtte-rnn/blob/master/python/tests/test_keras.py#L93
There's allready support for it. You need to do masking and use sample weights 👌
thanks! think I got it.
Note that if you use zero-mask then you may not learn anything from the nonevents as the input vec may be zero for those steps. Ping @daynebatten
Not sure I'm following, @ragulpr. Can you give a little more detail as to when this would occur?
@daynebatten I actually just noticed that you use model.add(Masking(mask_value=0., input_shape=(max_time, 24)))
link
But it was there all along and sorry, actually it shouldn't be a problem for the Turbofan dataset since sensors measure . The problem would occur in data [data_pipeline}(https://github.com/ragulpr/wtte-rnn/blob/master/examples/data_pipeline/data_pipeline.ipynb)-example where we have long periods of nothingness, but we still want to propagate state. Here data could be set to 0. If we'd have mask_value=0
we wouldn't predict or learn anything here.
If you want to apply to other dataset I suggest using some highly unlikely mask-value instead.
Also, we never said hi awesome work @daynebatten! 👏🙌
That's a great point. It's probably a best practice to use a very large mask value. If you've normalized your data, the mask value should never come up by accident, and certainly not for all variables.
And I think most of the thanks go to you for doing all the heavy lifting here!
Hi Egil, I'm not sure I understand why do you suggest to use sample weights if we are already masking. From what I read in stackoverflow
If there's a mask in your model, it'll be propagated layer-by-layer and eventually applied to the loss. So if you're padding and masking the sequences in a correct way, the loss on the padding placeholders would be ignored.
If the loss on the padding placeholders is ignored, why do we need sample weights?
@Manelmc Good catch. While one would expect masking
to propagate to the loss function, there was some inconsistencies in Keras at the time of implementation w.r.t how _weighted_masked_objective
worked back then with custom loss function. It might have been fixed now. If that's the case, then sample_weights
are not necessary if you want equal weights.
I like to use sample_weights
anyway since I usually don't use equal weights for every sample.
I'm experimenting with using the wtte model on teh CMAPPS dataset. I note you recommend using batch size = 1:
However this is resulting in very slow training (around 200 secs per epoch with GPU).
Are there any workarounds for this? I've tried modifying the loss function so it's divided by the number of timesteps of interest (which I'd hoped would normalise it), but this doesn't produce good results.