Prediction on new videos

[antortjim]

Dear deepethogram devs,

I was wondering what would be the minimum steps required for, given a set of flow generator + feature extractor + sequence models already trained on my specific dataset, getting new predictions for new videos. It is not clear to me from the Colab notebook and the documentation. I only got the workflows there to work if the video is labeled and I train all 3 models on the video.

For example, I added a new video, and if I open it in the GUI and click on sequence > infer (which is the last step of the above workflows), I can only choose among the videos I used for training, but not the new one. Ideally, I would do this with the CLI though, with something like the one-liner you provide in the documentation:

python -m deepethogram.flow_generator.train project.config_file=path/to/config/file.yaml reload.weights=path/to/flow/weights.pt split.file=path/to/split.yaml preset=deg_s

Maybe I only need to train the flow generator (which needs no user labels) on the new video, but not the feature extractor? What about the sequence model?

Thank you!

Antonio

[antortjim]

Answering my own question:

when training deepethogram on any video, this is the sequence of steps that need to be taken (as explained in the Colab notebook )

flow_generator.train feature_extractor.train feature_extractor.inference sequence.train

i.e. the sequence model cannot be trained before the videos used for training have been passed through the feature extractor inference module.

Once all 3 models (flow generator, feature extractor and sequence) are trained, then one needs to do the following to predict on unseen videos

feature_extractor.inference sequence.inference

Moreover, when calling the inference modules from the CLI, one can pass the parameter inference.directory_list like so

inference.directory_list=[${PROJECT_PATH}/DATA/foo,${PROJECT_PATH}/DATA/bar]

to select which videos will be run through inference (vs all)

[antortjim]

Finally, I have made a Github repository where the .pbs scripts I wrote to run deepethogram in a cluster are available for anyone to consult

[jbwallace123]

@antortjim I am attempting to reuse my trained models on new, unseen videos, however, I keep encountering problems along the way. Were you able to successfully run feature_extractor.inference on new videos or did you have to retrain the feature_extractor first with the unseen videos? Could you please briefly describe your workflow (e.g. do you start a new project or add video to existing project - do you need to retrain the flow generator and feature extractor before running inference) for using pretrained models to create inferences on new video? Thank you in advance!

[antortjim]

Hi @jbwallace123 when I reuse the trained models on new data I only run deepethogram.feature_extractor.inference and deepethogram.sequence.inference. So no retraining is needed! I add the videos to the existing project and specify which videos I want to infer using the option inference.directory_list in the config.

[antortjim]

I may be able to help if you share a log of the error, and also make sure you are using the last version of deepethogram

[jbwallace123]

@antortjim Thank you so much for your help! Updating deepethogram completely remedied the issue and now it's working like a dream. Thank you for the advice and the quick reply!