Error with TennisTable dataset

[dimstudio]

Hi @HansBambel when I try to train a model using the TableTennis500 dataset I get this error, is the same for you?

Traceback (most recent call last):
  File "/Users/daniele/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-d8916166bc74>", line 1, in <module>
    runfile('/Users/daniele/Documents/research/SharpFlow/model_training_pytorch.py', wdir='/Users/daniele/Documents/research/SharpFlow')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/daniele/Documents/research/SharpFlow/model_training_pytorch.py", line 387, in <module>
    train_test_model()
  File "/Users/daniele/Documents/research/SharpFlow/model_training_pytorch.py", line 383, in train_test_model
    batchsize=batch_size)
  File "/Users/daniele/Documents/research/SharpFlow/model_training_pytorch.py", line 336, in test_model
    f"Target-class: {tar_class} Accuracy: {acc[i]:.5f} Precision: {precision[i]:.5f} Recall: {recall[i]:.5f} F1-Score: {f1[i]}")
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

[HansBambel]

The error occurs after training when testing the model. I am guessing it is because you only have one class. That for-loop is made for multiple classes. I dont have the dataset, but I think you can fix it by adding the following condition before the for-loop:

if len(target_classes) > 1:
        for i, tar_class in enumerate(target_classes):
             print(f"Target-class: {tar_class} Accuracy: {acc[i]:.5f} Precision: {precision[i]:.5f} Recall: {recall[i]:.5f} F1-Score: {f1[i]}")
else:
    print(f"Accuracy: {acc:.5f} Precision: {precision:.5f} Recall: {recall:.5f} F1-Score: {f1}")

[dimstudio]

Hey @HansBambel, @khaleel needs to report the results of the model training in his thesis.

Can you check if you get better results? Running multiple times I get different results, meaning the split between training and validation plays a big role. I assume because the dataset is small (look at the shape of the Tensor)

These are my results"

With Kinect+Android ignore_files = []

runfile('/Users/***/Documents/research/SharpFlow/model_training_pytorch.py', wdir='/Users/***/Documents/research/SharpFlow')
Device: cpu
Shape tensor_data (433, 51, 40)
Shape annotations (433, 6)
Epoch:     0, Time: 0.016 min, Train_loss: 0.0115705815, Val_loss: 0.6929198430, Accuracy: 0.58462, Precision: 0.56962, Recall: 0.69231, F1-Score: 0.625
Epoch:     1, Time: 0.033 min, Train_loss: 0.0114874132, Val_loss: 0.6928086207, Accuracy: 0.50000, Precision: 0.50000, Recall: 1.00000, F1-Score: 0.6666666865348816
Epoch:     2, Time: 0.051 min, Train_loss: 0.0114238644, Val_loss: 0.6660660716, Accuracy: 0.62308, Precision: 0.60256, Recall: 0.72308, F1-Score: 0.6573426723480225
Epoch:     3, Time: 0.069 min, Train_loss: 0.0111531273, Val_loss: 0.6672933047, Accuracy: 0.63077, Precision: 0.86957, Recall: 0.30769, F1-Score: 0.45454543828964233
Epoch:     4, Time: 0.088 min, Train_loss: 0.0109126818, Val_loss: 0.6720546282, Accuracy: 0.61538, Precision: 0.60563, Recall: 0.66154, F1-Score: 0.632352888584137
Epoch:     5, Time: 0.107 min, Train_loss: 0.0113479047, Val_loss: 0.6660644036, Accuracy: 0.66923, Precision: 0.65278, Recall: 0.72308, F1-Score: 0.6861313581466675
Epoch:     6, Time: 0.125 min, Train_loss: 0.0112717919, Val_loss: 0.6775083102, Accuracy: 0.50000, Precision: 0.50000, Recall: 0.01538, F1-Score: 0.02985074557363987
Epoch:     7, Time: 0.144 min, Train_loss: 0.0112679115, Val_loss: 0.6567938676, Accuracy: 0.64615, Precision: 0.62667, Recall: 0.72308, F1-Score: 0.6714285612106323
Epoch:     8, Time: 0.162 min, Train_loss: 0.0106667349, Val_loss: 0.6320748393, Accuracy: 0.66154, Precision: 0.70588, Recall: 0.55385, F1-Score: 0.6206896901130676
Epoch:     9, Time: 0.178 min, Train_loss: 0.0116059613, Val_loss: 0.7103443907, Accuracy: 0.56923, Precision: 0.54286, Recall: 0.87692, F1-Score: 0.6705882549285889
Epoch:    10, Time: 0.195 min, Train_loss: 0.0114113643, Val_loss: 0.6854615221, Accuracy: 0.50000, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    11, Time: 0.211 min, Train_loss: 0.0113487073, Val_loss: 0.6979358921, Accuracy: 0.50000, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    12, Time: 0.227 min, Train_loss: 0.0114526056, Val_loss: 0.6911495722, Accuracy: 0.58462, Precision: 0.76190, Recall: 0.24615, F1-Score: 0.3720930218696594
Epoch:    13, Time: 0.243 min, Train_loss: 0.0113705412, Val_loss: 0.6850616281, Accuracy: 0.54615, Precision: 0.52679, Recall: 0.90769, F1-Score: 0.6666666865348816
Epoch:    14, Time: 0.259 min, Train_loss: 0.0112032072, Val_loss: 0.6901133812, Accuracy: 0.51538, Precision: 0.50794, Recall: 0.98462, F1-Score: 0.6701570749282837
Epoch:    15, Time: 0.275 min, Train_loss: 0.0113789085, Val_loss: 0.6899820419, Accuracy: 0.50769, Precision: 0.50388, Recall: 1.00000, F1-Score: 0.6701030731201172
Epoch:    16, Time: 0.291 min, Train_loss: 0.0113352329, Val_loss: 0.6873491315, Accuracy: 0.59231, Precision: 0.75000, Recall: 0.27692, F1-Score: 0.40449443459510803
Epoch:    17, Time: 0.306 min, Train_loss: 0.0112699717, Val_loss: 0.6781673890, Accuracy: 0.65385, Precision: 0.77778, Recall: 0.43077, F1-Score: 0.5544554591178894
Epoch:    18, Time: 0.322 min, Train_loss: 0.0110257834, Val_loss: 0.6689952896, Accuracy: 0.62308, Precision: 0.64286, Recall: 0.55385, F1-Score: 0.5950413346290588
Epoch:    19, Time: 0.337 min, Train_loss: 0.0112914624, Val_loss: 0.6812486704, Accuracy: 0.51538, Precision: 0.62500, Recall: 0.07692, F1-Score: 0.13698630034923553
Epoch:    20, Time: 0.353 min, Train_loss: 0.0112004976, Val_loss: 0.6765117022, Accuracy: 0.50000, Precision: 0.50000, Recall: 0.98462, F1-Score: 0.6632124781608582
Epoch:    21, Time: 0.368 min, Train_loss: 0.0110635376, Val_loss: 0.6423410948, Accuracy: 0.69231, Precision: 0.72727, Recall: 0.61538, F1-Score: 0.6666666865348816
Epoch:    22, Time: 0.384 min, Train_loss: 0.0104235405, Val_loss: 0.7060903036, Accuracy: 0.52308, Precision: 0.71429, Recall: 0.07692, F1-Score: 0.1388888955116272
Epoch:    23, Time: 0.400 min, Train_loss: 0.0110754190, Val_loss: 0.6406689268, Accuracy: 0.64615, Precision: 0.62338, Recall: 0.73846, F1-Score: 0.6760563850402832
Epoch:    24, Time: 0.417 min, Train_loss: 0.0103346849, Val_loss: 0.6338408947, Accuracy: 0.67692, Precision: 0.78049, Recall: 0.49231, F1-Score: 0.6037735939025879
Epoch:    25, Time: 0.433 min, Train_loss: 0.0099023352, Val_loss: 0.6375842544, Accuracy: 0.63846, Precision: 0.78125, Recall: 0.38462, F1-Score: 0.5154639482498169
Epoch:    26, Time: 0.449 min, Train_loss: 0.0100923133, Val_loss: 0.6281467759, Accuracy: 0.68462, Precision: 0.70690, Recall: 0.63077, F1-Score: 0.6666666865348816
Epoch:    27, Time: 0.465 min, Train_loss: 0.0097990799, Val_loss: 0.6448610122, Accuracy: 0.60769, Precision: 0.75000, Recall: 0.32308, F1-Score: 0.4516128897666931
Epoch:    28, Time: 0.480 min, Train_loss: 0.0100232400, Val_loss: 0.6297016731, Accuracy: 0.64615, Precision: 0.64179, Recall: 0.66154, F1-Score: 0.6515151858329773
Epoch:    29, Time: 0.497 min, Train_loss: 0.0104895626, Val_loss: 0.6175874527, Accuracy: 0.66923, Precision: 0.80556, Recall: 0.44615, F1-Score: 0.5742574334144592
/Users/daniele/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type MyLSTM. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/Users/daniele/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LSTM. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/Users/daniele/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Linear. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
Device: cpu
Accuracy: 0.68831 Precision: 0.83333 Recall: 0.56818 F1-Score: 0.6756756901741028

without Kinect ignore_files = ['kinect']

runfile('/Users/daniele/Documents/research/SharpFlow/model_training_pytorch.py', wdir='/Users/daniele/Documents/research/SharpFlow')
Device: cpu
Shape tensor_data (433, 51, 6)
Shape annotations (433, 6)
Epoch:     0, Time: 0.011 min, Train_loss: 0.0120834771, Val_loss: 0.6930300713, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:     1, Time: 0.023 min, Train_loss: 0.0114761469, Val_loss: 0.6942599682, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:     2, Time: 0.034 min, Train_loss: 0.0115645044, Val_loss: 0.6938430401, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:     3, Time: 0.046 min, Train_loss: 0.0114645246, Val_loss: 0.6937604216, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:     4, Time: 0.058 min, Train_loss: 0.0114950843, Val_loss: 0.6938754971, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:     5, Time: 0.070 min, Train_loss: 0.0114667364, Val_loss: 0.6930258109, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:     6, Time: 0.083 min, Train_loss: 0.0114481963, Val_loss: 0.6931612079, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:     7, Time: 0.096 min, Train_loss: 0.0114784949, Val_loss: 0.6945463254, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:     8, Time: 0.109 min, Train_loss: 0.0114460041, Val_loss: 0.6934814765, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:     9, Time: 0.122 min, Train_loss: 0.0114375496, Val_loss: 0.6930559654, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    10, Time: 0.136 min, Train_loss: 0.0114555489, Val_loss: 0.6930088557, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    11, Time: 0.150 min, Train_loss: 0.0114458129, Val_loss: 0.6929933062, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    12, Time: 0.165 min, Train_loss: 0.0114443731, Val_loss: 0.6932804053, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    13, Time: 0.179 min, Train_loss: 0.0114542771, Val_loss: 0.6936185204, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    14, Time: 0.193 min, Train_loss: 0.0114383762, Val_loss: 0.6929770470, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    15, Time: 0.207 min, Train_loss: 0.0114451775, Val_loss: 0.6929643393, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    16, Time: 0.222 min, Train_loss: 0.0114534074, Val_loss: 0.6929639770, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    17, Time: 0.236 min, Train_loss: 0.0114540552, Val_loss: 0.6939796283, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    18, Time: 0.250 min, Train_loss: 0.0114741044, Val_loss: 0.6942989248, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    19, Time: 0.265 min, Train_loss: 0.0114592255, Val_loss: 0.6931585642, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    20, Time: 0.279 min, Train_loss: 0.0114664898, Val_loss: 0.6929251597, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    21, Time: 0.294 min, Train_loss: 0.0114446182, Val_loss: 0.6931015950, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    22, Time: 0.308 min, Train_loss: 0.0114771641, Val_loss: 0.6933614107, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    23, Time: 0.322 min, Train_loss: 0.0114545397, Val_loss: 0.6941220339, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    24, Time: 0.337 min, Train_loss: 0.0114567087, Val_loss: 0.6930190398, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    25, Time: 0.351 min, Train_loss: 0.0114476458, Val_loss: 0.6934582967, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    26, Time: 0.365 min, Train_loss: 0.0114461062, Val_loss: 0.6929372347, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
Epoch:    27, Time: 0.379 min, Train_loss: 0.0114513152, Val_loss: 0.6926062676, Accuracy: 0.50769, Precision: 0.00000, Recall: 0.00000, F1-Score: nan
Epoch:    28, Time: 0.393 min, Train_loss: 0.0114472994, Val_loss: 0.6929746756, Accuracy: 0.48462, Precision: 0.48800, Recall: 0.95312, F1-Score: 0.6455026268959045
Epoch:    29, Time: 0.407 min, Train_loss: 0.0114433716, Val_loss: 0.6929546705, Accuracy: 0.49231, Precision: 0.49231, Recall: 1.00000, F1-Score: 0.6597938537597656
/Users/daniele/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type MyLSTM. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/Users/daniele/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LSTM. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
/Users/daniele/anaconda3/lib/python3.7/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Linear. It won't be checked for correctness upon loading.
  "type " + obj.__name__ + ". It won't be checked "
Device: cpu
Accuracy: 0.57143 Precision: 0.57143 Recall: 1.00000 F1-Score: 0.7272727489471436

[HansBambel]

To get a very nice and valid metric for this dataset we could do a n-fold cross-validation. But for this we need to add the sklearn.model_selection.KFold for splitting instead of the random splitting of the validation.

[dimstudio]

Hey @HansBambel I am trying to apply Kfold this is the current code:

    # Split into train, validation
    perm_img_ind = np.random.permutation(range(len(tensor_data)))

    train_ind = perm_img_ind[:int(len(perm_img_ind) * train_valid_split)]
    valid_ind = perm_img_ind[int(len(perm_img_ind) * train_valid_split):]

    x_train = np.array([tensor_data[i] for i in train_ind])
    y_train = np.array([targets[i] for i in train_ind])
    x_valid = np.array([tensor_data[i] for i in valid_ind])
    y_valid = np.array([targets[i] for i in valid_ind])

I tried with the following, but not really sure how to continue:

    from sklearn.model_selection import KFold
    kf = KFold(n_splits=10)
    kf.get_n_splits(tensor_data)
    KFold(n_splits=2, random_state=None, shuffle=False)
    for train_index, test_index in kf.split(tensor_data):
        print("TRAIN:", train_index, "TEST:", test_index)
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]

[HansBambel]

This is done in the load_train_data function, right? When we use KFold we can not really use that function then, because this needs to be done k times. We need to modify the train_model function for this. Or better: create a train_model_kfold function which then saves all k trained models to later run the test_model on each model.

[dimstudio]

OK got it, I try to implement train_model_kfold. Do you think we should also create test_model_kfold or we can use the same test_model function?

[HansBambel]

I think I just implemented that function. I haven't tested it yet though. I don't think we need a separate test model function. I have to check.

[dimstudio]

I tried and it works, you need to call the right function train_model_kfold

It only works if I do not exclude any file or any attribute, if I exclude e.g. Kinect and Ankles, it gives me shape errors. I think the to_exclude clause needs to be checked.

I sent you again the tabletennis500 dataset on Skype ;-)

[HansBambel]

Thanks for the data!

I just tested it and it works for me. Also with excluding things.

[HansBambel]

I will write a function for training and testing with kfold.