Closed aaronfderybel closed 5 years ago
Can you repeat the same without Talos i.e. just running the model both multi_gpu and no as stand alone Keras model and see what happens.
My guess is that whatever Keras is doing to handle multi_gpu situation, results in the kind of output you are seeing.
Hello @mikkokotila ,
thank you for your swift response.
NOT using the talos.Scan(..)
function and directly using build(hyperpar)
with the hyperparameters:
params = {'opt': adam, \
'lr': 10 ** (-3), \
'n_layer': 2, \
'loss_func': 'mse'
}
When using multi_gpu(..)
the output remains the same.
When not using multi_gpu
i see expected output.
I could also remake build(self, hyperpar)
into build(self)
but this should not make a difference since i'm
sending one set of hyperparameters to the function not using talos scan.
I think this confirms that multi_gpu(..)
is responsible for the output.
Update
I also looked at the loss functions of both putting multi_gpu on/off , keep in mind that they will not produce exactly same results:
They look as similar as multiple runs without multi_gpu.
@mikkokotila Did you have time to look into this ?
It's not a pressing issue if multi_gpu()
only produces a wrong model.summary()
printout.
If this is the case, I suggest the issue is closed
Thanks for the clarification and sorry for delay in replying. As you had suggested, does not seem that anything is wrong with Talos, so closing here.
Thanks again.
Thanks so much for coming here to raise an issue. Please take a moment to 'check' the below boxes:
[x ] I'm up-to-date with the latest release:
pip install -U talos
[x ] I've confirmed that my Keras model works outside of Talos.
Currently facing a weird/critical bug When running the script below and setting hyperpar to:
I pass the dictionary to the build model function ...
when I use![image](https://user-images.githubusercontent.com/43145159/57216669-ce91b300-6ff0-11e9-86c4-ede7108ab77b.png)
multi_gpu(..)
I receive the following output formodel.summary()
:However my model uses LSTM and when removing the![image](https://user-images.githubusercontent.com/43145159/57216787-27614b80-6ff1-11e9-83df-63a0afe5f7b0.png)
multi_gpu(..)
I receive following expected output:The model trains and gives output. didn't check if it gives exactly same results. In short Should I be worried ? Are these Lambda layers, custom layers created by talos for multi_gpu processing or is some nasty bug going on ?
Thanks in advance, Aaron De Rybel