search

opendr-eu / opendr

A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning
Apache License 2.0
614 stars 95 forks source link

Intent recognition tool #443

Closed katerynaCh closed 11 months ago

katerynaCh commented 1 year ago

This PR implements an intent recognition tool that can classify text data into the following intents: [ 'Complain', 'Praise', 'Apologise', 'Thank', 'Criticize', 'Agree', 'Taunt', 'Flaunt', 'Joke', 'Oppose', 'Comfort', 'Care', 'Inform', 'Advise', 'Arrange', 'Introduce', 'Leave', 'Prevent', 'Greet', 'Ask for help' ]. This tool is meant to be used together with the speech transcription tool for speech-to-text conversion and hence the ROS nodes are expected to subscribe to output of speech transcription ROS node. Similarly, python demo is provided with integration of the speech transcription tool.

The model implements an architecture that is trained in a multimodal manner (audio, visual, text) with the goal of improving inference on one of the modalities, in our case, text. The reference to the paper describing the approach will be added in the places where it is missing soon, once the paper/preprint is published. We provide several pretrained models of different sizes, from bert-tiny to bert-base.

This tool can be fully tested once the speech transcription (whisper) tool (#433) is integrated (or by just reusing its code) but I would rather start the initial review sooner if possible.

I am not sure why some tools' tests are failing, but seems unrelated to this PR?

tsampazk commented 1 year ago

I've been in the process of reviewing this PR for the past few days now, i am still hung up on the gpu installation issues while trying to test the demos of this PR. In the meantime i added the required init file for this tool's tests to run and it seems we get an out of memory error as far as i can tell.

katerynaCh commented 12 months ago

Thank you for the review @tsampazk ! I have addressed most of the things mentioned, however I am not able to reproduce the error in demo_text.py that you mentioned. Could you provide more details, e.g. if you have made any changes in the config or elsewhere? Are you using the provided model from the fileserver (i.e., the one retrieved with .download()) or maybe you trained your own checkpoint?

As i mentioned in the comment before, I have ran into various issues with my local installation, and since currently the gpu installation of the toolkit is broken i went off on a huge tangent trying to test this tool. I tried running demo_text.py and got the following error:

tsampazk commented 12 months ago

Thank you for the review @tsampazk ! I have addressed most of the things mentioned, however I am not able to reproduce the error in demo_text.py that you mentioned. Could you provide more details, e.g. if you have made any changes in the config or elsewhere? Are you using the provided model from the fileserver (i.e., the one retrieved with .download()) or maybe you trained your own checkpoint?

As i mentioned in the comment before, I have ran into various issues with my local installation, and since currently the gpu installation of the toolkit is broken i went off on a huge tangent trying to test this tool. I tried running demo_text.py and got the following error:

Sorry for the late reply @katerynaCh, thanks for trying it out yourself. I am in the process of trying to figure out the installation issues (#461) and fix them before retrying to test your code. The error i mentioned has a good chance originating from my hacked installation. Nevertheless, i ran it pretty much as is. Just python demo_text.py, no arguments, config changes or custom models etc, just a messed up installation :laughing:.