search

deephealthproject / eddl

European Distributed Deep Learning (EDDL) library. A general-purpose library initially developed to cover deep learning needs in healthcare use cases within the DeepHealth project.
https://deephealthproject.github.io/eddl/
MIT License
34 stars 10 forks source link

Selu output problem #269

Closed georgemavrakis-wings closed 3 years ago

georgemavrakis-wings commented 3 years ago

Good afternoon,

I have a problem with the Selu layer in Pyeddl. I have opened an issue in the pyeddl repo, but I was redirected here.

More specifically, it seems that Selu layer does not calculate its output correctly. I have tested other layers instead (e.g. Elu, ReLu) and the output is as expected. All layers have been compared with PyTorch and seem to produce the same output, except for Selu.

I provided the input and output numpy arrays of the selu in pyeddl and pytorch in the above pyeddl repo issue. I use Linux OS, with the latest version of pyeddl 0.13.0 (EDDL 0.9.1b).

Thank you in advance.

salvacarrion commented 3 years ago

Thank you! We'll fix it asap

chavicoski commented 3 years ago

Hello, I have done tests and get the correct results. But I have noticed that in the tensors you have sent, the result of the eddl is the equivalent of applying the SELU twice to the input. Could it be that you are applying the operation twice?

georgemavrakis-wings commented 3 years ago

Hello,

No, I have the Selu layer initialized only once. I have constructed the following network:

Xin_t = Tensor.fromarray(Xin) # Xin: Selu_input.npy array

in_ = eddl.Input([128, Xin_t.shape[-1]])
layer = in_
layer2 = eddl.Selu(parent=layer)
layer3 = eddl.GlobalAveragePool1D(parent=layer2)
layer4 = eddl.Squeeze(parent=layer3)
out = eddl.Constant(l=eddl.Dense(parent=layer4, ndim=2), v=0.03)

net = eddl.Model(in_=[in_], out=[out])
eddl.build(net, eddl.adam(0.001), ["soft_cross_entropy"], ["categorical_accuracy"], eddl.CS_CPU())

eddl.train_batch(net, [Xin_t], [y_train], indices=np.linspace(0, 9, 10, dtype=int).tolist())  # y_train: (10x2) label array

Then, I use the following code to take the output of the Selu layer from the above net:

OUT = net.layers[1].output
OUT = OUT.getdata()

The result is equal with the pyeddl_SELU_output.npy array, which is different from the pytorch_SELU_output.npy.

chavicoski commented 3 years ago

Ok, your are right. There is a bug in the backward function of the Selu. In the inital tests that I made I used the predict function so I didn't see the error because the backward is not executed. Now it is fixed. Thank you!