search

hcmlab / nova

NOVA is a tool for annotating and analyzing behaviours in social interactions. It supports Annotators using Machine Learning already during the coding process. Further it features both, discrete labels and continuous scores and a visuzalization of streams recorded with the SSI Framework.
http://openssi.net/
GNU General Public License v3.0
174 stars 37 forks source link

Stream Files missing from Feature Extraction window #446

Open jepa88 opened 3 years ago

jepa88 commented 3 years ago

Hello and thank you for this tool! I am currently learning how to use the tool so I am using some example videos. When I am using the annotation ability of the tool, I can see that I have two videos. Please see attached image Smiling_database However, when I open the feature extraction tab I no longer see these videos Extract_Features The same when opening the training tab train_models The two videos are located in a folder /data/name_of_database/name_of_session/ and they are named as role.video_name.mp4 Could you please explain how I should fix this?

tobiasbaur commented 3 years ago

Hi! Thanks for raising this.

Indeed it is not yet described well in the documentation and we are hosting only a limited amount of features on github for now. We plan to add additional ones to the interface, but at this time you need to add them manually.

Make sure you point your cml directory in the database settings also to the cml folder. If you only downloaded the exe you will have to add the CML folder from here https://github.com/hcmlab/nova/tree/master/bin/cml

As you are intending to recognise smiles, i recommend using a featureset for facial features, such as openface. We are currently not hosting this publicly on github but I added a nova-ready-to-use openface2 (https://github.com/TadasBaltrusaitis/OpenFace) component here:

https://mediastore.rz.uni-augsburg.de/get/N1IsN68cRx/ (link might expire at some point)

Please see Licence instructions in the Openface Git.

Make sure you extract the content of this folder so you have the structure nova/cml/chains/video/openface2.

In the first screenshot you posted, under "streams" -> edit make sure the files have the type "video" (needs to match the cml/chains/video folder structure). and that the samplerate of the video is correct (e.g 25 or 30 frames per second)

Now you should see the video and the openface2 chain. When you extract the features for the first time, it will download all necessary files. It should then create the stream files automatically and add them to the database. (might take some time depending on the length of your video and the performance of your pc).

For the training process, make sure you have the the cml/models/templates/.. folder checked out. also note that you will need at least one annotation in the database to train the model. (You can try to start with linsvm which is very fast)

Please let me know if this helps, or if you need further assistance.

Best, Tobias Baur

jepa88 commented 3 years ago

Thank you very much for this detailed answer. It helped a lot I was able to proceed and extract features but when I tried to train the model the nova application crushes I have created some annotations for the videos shown in the following figure. image

Could you please point at some directions to what to look at to solve this issue? I am using the latest release NOVA 1.0.7.1 on windows 10. Thank you in advance

tobiasbaur commented 3 years ago

Hi, does this also happen on the 30hz stream? I could image it may be a problem with the 29.99 and 29.98 streams. (try to add it as 30hz and extract the features again on these).

If that is not the problem: Do you get any error before the crash? did you download the cml tools (via ? -> update cml tools)

jepa88 commented 3 years ago

This issue happens also with the 30hz video. I changed the video settings from 29.99 and 29.98 to 30hz, I extracted features again but the tool crashes. Unfortunately, I do not get any error: When I select the stream with the extracted features and the session and I click on 'train', the mouse point changes into 'loading' and the program is killed without any warning message to debug it. I clicked on 'update cml tools' and I got the message that the tools are already up to date. Could you let me know, what else I should look for? Thank you very much for your help.

tobiasbaur commented 3 years ago

hi, depending on how long you have been using NOVA you might want to update the cml tools (?-> Update CML tools) and, let the dialog delete the dlls, so they are redownloaded. In case there is no update option, please delete the cmltrain.exe and perform this step again. Please let me know if that helps.

jepa88 commented 3 years ago

Thank you very much for your feedback. I tried your suggestions but unfortunately nova keeps exiting without any message. Attached I upload a video of the procedure I follow to train the classifier after I have extracted features. Each session contains a video with annotations. I tried linsvm, svm, sklearn decision tree and sklearn svm. https://user-images.githubusercontent.com/78227691/107977305-27aabb00-6fbb-11eb-8e6f-cd09412d9635.mp4

jepa88 commented 3 years ago

Dear Dr Baur, I checked the event viewer on windows and I attach the message I get. From my understanding, a folder is missing.

Application: nova.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.IO.DirectoryNotFoundException at System.IO.__Error.WinIOError(Int32, System.String) at System.IO.Directory.InternalCreateDirectory(System.String, System.String, System.Object, Boolean) at System.IO.Directory.InternalCreateDirectoryHelper(System.String, Boolean) at ssi.DatabaseCMLTrainAndPredictWindow.Apply_Click(System.Object, System.Windows.RoutedEventArgs) at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean) at System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject, System.Windows.RoutedEventArgs) at System.Windows.Controls.Primitives.ButtonBase.OnClick() at System.Windows.Controls.Button.OnClick() at System.Windows.Controls.Primitives.ButtonBase.OnMouseLeftButtonUp(System.Windows.Input.MouseButtonEventArgs) at System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate, System.Object) at System.Windows.RoutedEventHandlerInfo.InvokeHandler(System.Object, System.Windows.RoutedEventArgs) at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean) at System.Windows.UIElement.ReRaiseEventAs(System.Windows.DependencyObject, System.Windows.RoutedEventArgs, System.Windows.RoutedEvent) at System.Windows.UIElement.OnMouseUpThunk(System.Object, System.Windows.Input.MouseButtonEventArgs) at System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate, System.Object) at System.Windows.RoutedEventHandlerInfo.InvokeHandler(System.Object, System.Windows.RoutedEventArgs) at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean) at System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject, System.Windows.RoutedEventArgs) at System.Windows.UIElement.RaiseTrustedEvent(System.Windows.RoutedEventArgs) at System.Windows.Input.InputManager.ProcessStagingArea() at System.Windows.Input.InputManager.ProcessInput(System.Windows.Input.InputEventArgs) at System.Windows.Input.InputProviderSite.ReportInput(System.Windows.Input.InputReport) at System.Windows.Interop.HwndMouseInputProvider.ReportInput(IntPtr, System.Windows.Input.InputMode, Int32, System.Windows.Input.RawMouseActions, Int32, Int32, Int32) at System.Windows.Interop.HwndMouseInputProvider.FilterMessage(IntPtr, MS.Internal.Interop.WindowMessage, IntPtr, IntPtr, Boolean ByRef) at System.Windows.Interop.HwndSource.InputFilterMessage(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef) at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef) at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object) at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32) at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate) at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32) at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr) at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef) at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame) at System.Windows.Window.ShowHelper(System.Object) at System.Windows.Window.ShowDialog() at ssi.MainHandler.showDialogClearWorkspace(System.Windows.Window) at System.Windows.EventRoute.InvokeHandlersImpl(System.Object, System.Windows.RoutedEventArgs, Boolean) at System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject, System.Windows.RoutedEventArgs) at System.Windows.Controls.MenuItem.InvokeClickAfterRender(System.Object) at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32) at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate) at System.Windows.Threading.DispatcherOperation.InvokeImpl() at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext, System.Threading.ContextCallback, System.Object) at System.Windows.Threading.DispatcherOperation.Invoke() at System.Windows.Threading.Dispatcher.ProcessQueue() at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef) at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef) at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object) at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32) at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate) at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32) at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr) at MS.Win32.UnsafeNativeMethods.DispatchMessage(System.Windows.Interop.MSG ByRef) at System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame) at System.Windows.Application.RunDispatcher(System.Object) at System.Windows.Application.RunInternal(System.Windows.Window) at ssi.App.Main()

tobiasbaur commented 3 years ago

Hello, yes

Description: The process was terminated due to an unhandled exception. Exception Info: System.IO.DirectoryNotFoundException at System.IO.__Error.WinIOError(Int32, System.String) at System.IO.Directory.InternalCreateDirectory(System.String, System.String, System.Object, Boolean) at System.IO.Directory.InternalCreateDirectoryHelper(System.String, Boolean)

this looks like NOVA tries to create a new folder (probably in cml/models/trainer/....) but maybe it does not have the rights to do so. Is the cml path in some restricted folder? (E.g. C:Programs) Did you try to run it as admin? (Depending on the configuration of your machine, there might be some problem with writing rights)

I can have a look at the code as well, just never ran into this before, so might be system specific.

jepa88 commented 3 years ago

Thank you very much for your reply. I had tried to run it as admin and it does not work either. The path is not in a restricted folder, I also moved it to another location to make sure that it is not a path issue. I really appreciate all your help