search

hfawaz / bigdata18

Transfer learning for time series classification
http://germain-forestier.info/src/bigdata2018/
GNU General Public License v3.0
378 stars 68 forks source link

Accuracy: Train or Val? #3

Closed tushar-semwal closed 5 years ago

tushar-semwal commented 5 years ago

Is the accuracy reported in the results on the paper, training or validation (test)? For e.g., Fig 9, Meat target dataset, the best accuracy without transfer is reported to be around 81%, however, I ran the train_fcn_scratch on Meat and the best val_acc I could get was 43%. This difference is stark!

I found the same for other datasets. Am I missing something?

What I did: just executed this command,

python3.py main train_fcn_scratch

hfawaz commented 5 years ago

Hello,

Yeah in fact you should change the number of epochs.

https://github.com/hfawaz/bigdata18/blob/c5cf4037c72c2ad98593cd28cbfe08ab7d8e3c10/main.py#L160

It was set to 10 by mistake, it should be equal to 2000.

Thakns for sharing the problem

tushar-semwal commented 5 years ago

Sorry I did not mention that I have already changed the nb_epoch = 200 (not 2000). However, just in case, I will again try but this time for 2000 epochs and will report you the results.

hfawaz commented 5 years ago

Okay good to know.

Indeed 200 in the paper is used only for visualization purposes, but the models were trained for 2000 epochs.

Thanks again

tushar-semwal commented 5 years ago

Hello!

Thank you for the response!

I tried running the experiments with epochs = 2000. The maximum validation accuracy I got was 0.53. Attached are the files for the Meat dataset.

meat_best_model_trained_scratch.zip

epochs_loss

hfawaz commented 5 years ago

Are you failing to reproduce only on the Meat dataset ?

hfawaz commented 5 years ago

In the file you have sent me, the maximum validation accuracy is much more than 0.53

For example if you look at the validation accuracy on epoch 2000 you will see an accuracy of 0.95

tushar-semwal commented 5 years ago

That is the confusion. The model loss graph is very fluctuating as you can see also the val_acc fluctuates and jumps even reached to 0.95.

But, if you check the precision, accuracy, recall file you will find the accuracy to be 0.53.

Sorry for the drag, but I tried for other datasets too, for e.g., BeetleFly graph in the repo shows an accuracy of 0.9 but I got 0.5 for the best_model_val_acc metric.

tushar-semwal commented 5 years ago

I think the reasons are:

hfawaz commented 5 years ago

Yes indeed this is confusing.

I will take a look and get back to you ASAP.

tushar-semwal commented 5 years ago

Great! Thanks!

hfawaz commented 5 years ago

I ran the code several times to make sure.

I obtained these accuracies:

I pushed the code with some corrections regarding the visualization etc.

Let me know what results you will have once you rerun.

tushar-semwal commented 5 years ago

Hey Hi! Thanks for the efforts.

Unfortunately, I am still not able to get that level of accuracy.

However, after changing the monitor option from 'loss' to 'val_loss', I obtained similar accuracies as shown in the paper. These lines of code: reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=50, min_lr=0.0001) model_checkpoint = keras.callbacks.ModelCheckpoint(filepath=file_path, monitor='val_loss', save_best_only=True)

Check the monitor option. I think the model checkpoint should be based on validation loss and not on the training loss. If you check the reference manual of Keras, it points to the use of val_loss as the monitoring metric.

What's your opinion?

hfawaz commented 5 years ago

Hello,

In fact, if you set the monitor to val_loss, you will be tuning your model based on the test loss, which is considered over-fitting the test set. Therefore I advise leaving it set to loss. The ideal thing would be to have a separate validation set (which is a split from the training set). However here we did not perform this validation split.

So to sum up, the results in the paper are not based on monitoring the val_loss.

Maybe there are other factors that are playing a role here, (version of Keras, Tensorflow, CUDA, CUDNN.. etc).

Are you using a GPU ?

tushar-semwal commented 5 years ago

Ah, I see. Yes, you are right. There could be other factors, and I am using a CPU which could be the reason for the loss of accuracy. There are certain messages which are thrown when I run the script. Snapshot attached.

q

The last message is interesting.

hfawaz commented 5 years ago

I think your version of tensorflow is very old, try updating your packages such as Keras, numpy and tensorflow.

Usually the newest version are the best and should work with the code in this repository.

Try that before testing on GPU.

tushar-semwal commented 5 years ago

Hi Fawaz, I am now getting the accuracies as reported. I tried reinstalling the Tensorflow, though I already had the latest versions. It seems this worked!

Thanks for the help.