Testing results in 100% accuracy always, overfitting possibility?

[RahulB117]

Good evening ohjay, Thank you so much for clarifying the previous error that occurred during execution. I have run into this new problem here as seen from the attached screenshots.

I seem to be getting a 100% accuracy during testing for boxing yet 0% for anything else. I have tried this with the other activities but only boxing is recognised, but even then a 100% ( 1.0) test accuracy is too high.

Output : config = yaml.load(open(args.config, 'r'))

c:\users\rahul\hmm_activity_recognition\main.py(42)classify_activity() -> ef_params = config['extract_features'] (Pdb) c [o] NOT normalizing model output --- 0 [o] Video path: C:\Users\Rahul\hmm_activity_recognition\data\kth\test\boxing\person06_boxing_d4_uncomp.avi [o] including feature "edge" [o] including feature "centroid" [o] NOT including feature "optical_flow" [o] NOT including feature "freq_optical_flow" [o] NOT including feature "dense_optical_flow" [o] including feature "freq_dense_optical_flow" [o] including feature "divergence" [o] including feature "curl" [o] including feature "avg_velocity" [+] Processed 392 frames. Shape of video feature matrices: [(94, 80), (97, 80), (91, 80), (102, 80)] [o] Classified 1 / 2 in the boxing evaluation set. [o] Current accuracy: 1.00 --- 1 [o] Video path: C:\Users\Rahul\hmm_activity_recognition\data\kth\test\boxing\person07_boxing_d1_uncomp.avi [o] including feature "edge" [o] including feature "centroid" [o] NOT including feature "optical_flow" [o] NOT including feature "freq_optical_flow" [o] NOT including feature "dense_optical_flow" [o] including feature "freq_dense_optical_flow" [o] including feature "divergence" [o] including feature "curl" [o] including feature "avg_velocity" [+] Processed 385 frames. Shape of video feature matrices: [(106, 80), (98, 80), (80, 80), (93, 80)] [o] Classified 2 / 2 in the boxing evaluation set. [o] Current accuracy: 1.00 --- 2 [o] Video path: C:\Users\Rahul\hmm_activity_recognition\data\kth\test\walking\person04_walking_d4_uncomp.avi [o] including feature "edge" [o] including feature "centroid" [o] NOT including feature "optical_flow" [o] NOT including feature "freq_optical_flow" [o] NOT including feature "dense_optical_flow" [o] including feature "freq_dense_optical_flow" [o] including feature "divergence" [o] including feature "curl" [o] including feature "avg_velocity" [+] Processed 418 frames. Shape of video feature matrices: [(98, 80), (92, 80), (91, 80), (89, 80)] [o] Classified 1 / 2 in the walking evaluation set. [o] Current accuracy: 0.00 --- 3 [o] Video path: C:\Users\Rahul\hmm_activity_recognition\data\kth\test\walking\person07_walking_d3_uncomp.avi [o] including feature "edge" [o] including feature "centroid" [o] NOT including feature "optical_flow" [o] NOT including feature "freq_optical_flow" [o] NOT including feature "dense_optical_flow" [o] including feature "freq_dense_optical_flow" [o] including feature "divergence" [o] including feature "curl" [o] including feature "avg_velocity" [+] Processed 495 frames. Shape of video feature matrices: [(86, 80), (81, 80), (87, 80), (80, 80)] [o] Classified 2 / 2 in the walking evaluation set. [o] Current accuracy: 0.00 {'boxing': 1.0, 'walking': 0.0} total: 1.00

Could you please help me. Thank you in advance.

[RahulB117]

Please do help, this is the only step I'm not fully understanding

[ohjay]

Is your test set always 2-3 videos? With that small of a sample size, it's hard to judge how meaningful these results are. For my experiments, I used the videos for subjects 1-16 as training data (in total 64 videos/activity) and the videos for subjects 17-25 as test data (in total 36 videos/activity). I suggest you re-run the code with this split and the owen.yaml config file (just change the filepaths, nothing else). You should get results which are ~58-83% classification accuracy for any activity. After confirming that this works, you can make adjustments of your own.

You might also want to normalize the features, which is theoretically helpful for stuff like this. I notice from your output that the normalization stats aren't being loaded, which is mostly my fault since the process for generating the stats file is very counterintuitive. In order to generate said file you need to run an "extract" pass with extract_features > normalize set to False in the config file (the reason being that we want to compute stats over the whole unnormalized training dataset). After generating the stats file, you'd then have to do everything over again: re-extract the features, build models, etc.

But for convenience I'll include my own norm_stats.pkl file, which you ought to be able to use if you divide up the data as I've outlined above. Just put it in the directory somewhere and set the value of stats_path as appropriate in the config file.

norm_stats.zip

In summary

From a clean pull of the repository and KTH data, the full steps I'm suggesting are:

redo your train/test splits for the KTH data
unzip the file above to get `norm_stats.pkl`
edit filepaths as appropriate in `config/owen.yaml`
python main.py extract config/owen.yaml
python main.py build config/owen.yaml
python main.py classify config/owen.yaml

[RahulB117]

Thank you so much, I’ll make the following changes and get back to you as soon as I can.

Sent from my iPhone

On 03-Apr-2019, at 6:32 AM, Owen Jow notifications@github.com wrote:

Is your test set always 2-3 videos? With that small of a sample size, it's hard to judge how meaningful these results are. For my experiments, I used the videos for subjects 1-16 as training data (in total 64 videos/activity) and the videos for subjects 17-25 as test data (in total 36 videos/activity). I suggest you re-run the code with this split and the owen.yaml config file (just change the filepaths, nothing else). You should get results which are ~58-83% classification accuracy for any activity. After confirming that this works, you can make adjustments of your own.

You might also want to normalize the features, which is theoretically helpful for stuff like this. I notice from your output that the normalization stats aren't being loaded, which is mostly my fault since the process for generating the stats file is very counterintuitive. In order to generate said file you need to run an "extract" pass with extract features > normalize set to False in the config file (the reason being that we want to compute stats over the whole unnormalized training dataset). After generating the stats file, you'd then have to do everything over again: re-extract the features, build models, etc.

But for convenience I'll include my own norm_stats.pkl file, which you ought to be able to use if you divide up the data as I've outlined above. Just put it in the directory somewhere and set the value of stats_path as appropriate in the config file.

norm_stats.zip

In summary

From a clean pull of the repository and KTH data, the full steps I'm suggesting are:

redo your train/test splits for the KTH data unzip the file above to get norm_stats.pkl edit filepaths as appropriate in config/owen.yaml python main.py extract config/owen.yaml python main.py build config/owen.yaml python main.py classify config/owen.yaml — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[RahulB117]

Greetings ohjay, After following the above steps with n_components = 4, I got the following accuracies,

Boxing : 0.3333 Handclapping : 0.1111 Handwaving : 1.00 Walking : 0 Running : 0

I'm not sure as to why the accuracies for the last three activities were as above. After modifying n_components to 3 ( n_components = 3) in the build_model section, I got the following accuracies ,

Boxing : 0.25 Handclapping : 0.22 Handwaving : 0.8611 Walking : 0.8055 Running : 0.9422

Could you please let me know if I should change the value of another component? Apart from that, is it possible to use another dataset apart from Kth as I want to try the one here ( https://www3.cs.stonybrook.edu/~kyun/research/kinect_interaction/index.html ) and also what is the functionality of sequences.txt ?

Thank you in advance

[ohjay]

Is there a reason you're not using the jogging class? I could imagine a scenario in which its absence would cause trouble, although it's hard to say for sure. If you really have everything set up the same way I do, there shouldn't be that much variance in the results.

I updated the setup script and added a new config file. Just to make sure everything is the same, could you re-clone the repository somewhere and run the new setup script+config file?

git clone git@github.com:ohjay/hmm_activity_recognition.git
cd hmm_activity_recognition
./get_data.sh
python main.py extract config/quickstart.yaml
python main.py build config/quickstart.yaml
python main.py classify config/quickstart.yaml

You might also want to check that your versions are the same, particularly for OpenCV and hmmlearn.

As far as I'm aware, there's no need to change any other parameters if you just want to replicate my results. But of course the parameters can make a big difference on results in general, so it's true you'll ultimately want to tune them.

---

You should be able to switch out the dataset pretty easily if you're able to set up the same directory structure as data (basically with videos in activity-labeled folders). You might need to change the reading process (see process_video in scripts/extract_features.py) if the new videos aren't AVI files.

sequences.txt is specific to the KTH dataset. It just gives the frame ranges corresponding to different "sequences" within each video. If you're using a new dataset, you should be able to delete sequences_path from the config file without any repercussions.

[RahulB117]

Thank you for getting back to me on this. I ran the above specifications with the running dataset and got your results. A colleague within my group mentioned that I should try doing a cross fold validation of this. Is it possible to do so, if so how must I proceed?

On Sun, Apr 14, 2019 at 6:10 PM Owen Jow notifications@github.com wrote:

Is there a reason you're not using the jogging class? I could imagine a scenario in which its absence would cause trouble, although it's hard to say for sure. If you really have everything set up the same way I do, there shouldn't be that much variance in the results.

I updated the setup script and added a new config file. Just to make sure everything is the same, could you re-clone the repository somewhere and run the new setup script+config file?

git clone git@github.com:ohjay/hmm_activity_recognition.git cd hmm_activity_recognition ./get_data.sh python main.py extract config/quickstart.yaml python main.py build config/quickstart.yaml python main.py classify config/quickstart.yaml

You might also want to check that your versions are the same https://github.com/ohjay/hmm_activity_recognition/blob/master/requirements.txt, particularly for OpenCV and hmmlearn.

As far as I'm aware, there's no need to change any other parameters if you just want to replicate my results. But of course the parameters can make a big difference on results in general, so it's true you'll ultimately want to tune them.

You should be able to switch out the dataset pretty easily if you're able to set up the same directory structure as data (basically with videos in activity-labeled folders). You might need to change the reading process (see process_video in scripts/extract_features.py) if the new videos aren't AVI files.

sequences.txt is specific to the KTH dataset. It just gives the frame ranges corresponding to different "sequences" within each video. If you're using a new dataset, you should be able to delete sequences_path from the config file without any repercussions.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ohjay/hmm_activity_recognition/issues/3#issuecomment-482966656, or mute the thread https://github.com/notifications/unsubscribe-auth/AuiwhWKzxR4UJS1hdF7ZuIwlAHz097nbks5vgyG_gaJpZM4cVTlY .

[ohjay]

Sure it's possible. Given the current infrastructure, the simplest approach would probably be to write a shell script akin to get_data.sh that would continually (a) rearrange the train/validation splits (switching up the validation split each time), (b) run the usual set of commands, and (c) send the output to a file.

[RahulB117]

I've begun work on that shell script but I've run into a few problems regarding the usage of another dataset. I'm unable to build any models. I get this error.

exchanging.h5

model 0: Initializing gmm model with args: {'init_params': 't', 'verbose': True, 'n_components': 4, 'transmat_prior': array([[0.33333333, 0.33333333, 0.33333333, 0. ], [0. , 0.33333333, 0.33333333, 0.33333333], [0. , 0. , 0.5 , 0.5 ], [0. , 0. , 0. , 1. ]]), 'n_iter': 20} subsample p: 0.50 [o] Loaded feature matrix: (1026, 80) [o] Loaded n_sequences: 20 [o] Fitting to feature matrix of shape (492, 80). 1 9200.2093 +nan 2 15859.2034 +6658.9941 3 15859.3046 +0.1013 4 15913.4253 +54.1206 5 18364.1924 +2450.7672 6 23488.2184 +5124.0260 Traceback (most recent call last): File "main.py", line 103, in eval(command)(config=config) File "main.py", line 60, in build_models bm.populate_model_dir(h5_dir, model_dir, all_model_args, n_features, compute_stats) File "C:\Users\Rahul\hmm_activity_recognition\scripts\build_models.py", line 172, in populate_model_dir model_args, n_features) File "C:\Users\Rahul\hmm_activity_recognition\scripts\build_models.py", line 118, in learn_params model.fit(feature_matrix, seq_lengths) File "C:\Users\Rahul\HMM2\lib\site-packages\hmmlearn\base.py", line 436, in fit bwdlattice) File "C:\Users\Rahul\HMM2\lib\site-packages\hmmlearn\hmm.py", line 607, in _accumulate_sufficient_statistics raise ValueError ValueError

I'm not sure how to fix this. Could you please help me out? Thanks in advance

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Mon, Apr 22, 2019 at 7:57 PM Owen Jow notifications@github.com wrote:

Sure it's possible. Given the current infrastructure, the simplest approach would probably be to write a shell script akin to get_data.sh https://github.com/ohjay/hmm_activity_recognition/blob/154c3c8e97d0c82bbee4bd3efded2765cbe450b4/get_data.sh#L32 that would continually (a) rearrange the train/validation splits (switching up the validation split each time), (b) run the usual set of commands, and (c) send the output to a file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ohjay/hmm_activity_recognition/issues/3#issuecomment-485432694, or mute the thread https://github.com/notifications/unsubscribe-auth/ALULBBIKXE62ODQJR5B64Y3PRXDLVANCNFSM4HCVHFMA .

[ohjay]

I'm not too sure either. Based on a quick internet search, you might want to try upgrading hmmlearn to version 0.2.1+, as they apparently rewrote the GMMHMM class to avoid this kind of problem. You can also try switching m_type to gaussian in the config file. I assume you've already checked that the features are sensible.