search

OpenVoiceOS / ovos-core

OpenVoiceOS Core, the FOSS Artificial Intelligence platform.
https://openvoiceos.org
Apache License 2.0
147 stars 17 forks source link

ocp pipeline #396

Closed NeonJarbas closed 9 months ago

NeonJarbas commented 10 months ago

docs https://github.com/OpenVoiceOS/ovos-technical-manual/pull/14 training https://github.com/OpenVoiceOS/ovos-classifiers/tree/dev/scripts/training/ocp

Layer 1 - Unambiguous

Before regular intent stage, taking into account current OCP state (media ready to play / playing)

Only matches if user unambiguously wants to trigger OCP

uses padacioso for exact matches

from ocp_nlp.intents import OCPPipelineMatcher

ocp = OCPPipelineMatcher()
print(ocp.match_high("play metallica", "en-us"))
# IntentMatch(intent_service='OCP_intents',
#   intent_type='ocp:play',
#   intent_data={'media_type': <MediaType.MUSIC: 2>, 'query': 'metallica',
#                'entities': {'album_name': 'Metallica', 'artist_name': 'Metallica'},
#                'conf': 0.96, 'lang': 'en-us'},
#   skill_id='ovos.common_play', utterance='play metallica')

Layer 2 - Semi-Ambiguous

uses a binary classifier to detect if a query is about media playback

from ocp_nlp.intents import OCPPipelineMatcher

ocp = OCPPipelineMatcher()

print(ocp.match_high("put on some metallica", "en-us"))
# None

print(ocp.match_medium("put on some metallica", "en-us"))
# IntentMatch(intent_service='OCP_media',
#   intent_type='ocp:play',
#   intent_data={'media_type': <MediaType.MUSIC: 2>,
#                'entities': {'album_name': 'Metallica', 'artist_name': 'Metallica', 'movie_name': 'Some'},
#                'query': 'put on some metallica',
#                'conf': 0.9578441098114333},
#   skill_id='ovos.common_play', utterance='put on some metallica')

Layer 3 - Ambiguous

Uses keyword matching and requires at least 1 keyword

OCP skills can provide these keywords at runtime, additional keywords for things such as media_genre were collected via SPARQL queries to wikidata

from ocp_nlp.intents import OCPPipelineMatcher

ocp = OCPPipelineMatcher()

print(ocp.match_medium("i wanna hear metallica", "en-us"))
# None

print(ocp.match_fallback("i wanna hear metallica", "en-us"))
#  IntentMatch(intent_service='OCP_fallback',
#    intent_type='ocp:play',
#    intent_data={'media_type': <MediaType.MUSIC: 2>,
#                 'entities': {'album_name': 'Metallica', 'artist_name': 'Metallica'},
#                 'query': 'i wanna hear metallica',
#                 'conf': 0.5027561091821287},
#    skill_id='ovos.common_play', utterance='i wanna hear metallica')

Config

companion PR https://github.com/OpenVoiceOS/ovos-config/pull/96

{
  "OCP": {
     // ignore entries that report a score below this value
     // this is the way for the skills to say they are not confident
     // 0 - 100
     "min_score": 50,

    // minimum required confidence of classifier,
    // if below this the query uses MediaType.GENERIC
    // 0.0 - 1.0
    "classifier_threshold": 0.5,

    // ignore results that dont match the classifier's media type
    // does not apply to MediaType.GENERIC
    "filter_media": true

  }
}

LOGS

2024-01-09 23:46:23.880 - skills - ovos_core.transformers:transform:60 - DEBUG - ovos-utterance-normalizer: {'session': {'active_skills': [], 'utterance_states': {}, 'session_id': 'default', 'lang': 'en-us', 'context': {'timeout': 120, 'frame_stack': []}, 'site_id': 'unknown', 'pipeline': ['converse', 'ocp_high', 'padatious_high', 'adapt', 'common_qa', 'ocp_medium', 'fallback_high', 'padatious_medium', 'fallback_medium', 'ocp_fallback', 'fallback_low'], 'stt': {'plugin_id': 'ovos-stt-plugin-server', 'config': {}}, 'tts': {'plugin_id': 'ovos-tts-plugin-server', 'config': {}}}, 'lang': 'en-us'}
2024-01-09 23:46:23.992 - skills - padacioso:calc_intent:255 - DEBUG - {'entities': {'query': 'lovecraft'}, 'conf': 0.96, 'name': 'play'}
2024-01-09 23:46:27.150 - skills - ovos_core.intent_services.ocp_service:classify_media:552 - INFO - OVOSCommonPlay MediaType prediction: movie confidence: 0.3032108745919962
2024-01-09 23:46:27.150 - skills - ovos_core.intent_services.ocp_service:classify_media:553 - DEBUG -      utterance: lovecraft
2024-01-09 23:46:27.150 - skills - ovos_core.intent_services.ocp_service:classify_media:555 - INFO - ignoring MediaType classifier, low confidence prediction
2024-01-09 23:46:27.153 - skills - ovos_core.intent_services:handle_utterance:318 - DEBUG - intent matching took: 3.2681217193603516
2024-01-09 23:46:27.162 - skills - ovos_bus_client.session:update:546 - DEBUG - replacing default session with: {'active_skills': [['ovos.common_play', 1704843987.1537983]], 'utterance_states': {}, 'session_id': 'default', 'lang': 'en-us', 'context': {'timeout': 120, 'frame_stack': []}, 'site_id': 'unknown', 'pipeline': ['converse', 'ocp_high', 'padatious_high', 'adapt', 'common_qa', 'ocp_medium', 'fallback_high', 'padatious_medium', 'fallback_medium', 'ocp_fallback', 'fallback_low'], 'stt': {'plugin_id': 'ovos-stt-plugin-server', 'config': {}}, 'tts': {'plugin_id': 'ovos-tts-plugin-server', 'config': {}}}
2024-01-09 23:46:27.162 - skills - ovos_bus_client.client.client:on_default_session_update:161 - DEBUG - synced default_session
2024-01-09 23:46:27.162 - skills - ovos_bus_client.apis.ocp:__init__:828 - DEBUG - Created GENERIC query: lovecraft
2024-01-09 23:46:27.315 - skills - ovos_bus_client.apis.ocp:register_events:874 - DEBUG - Registering Search Bus Events
2024-01-09 23:46:27.330 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-lovecraft-comics.jarbasai is searching
2024-01-09 23:46:27.332 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-librivoxx.jarbasai is searching
2024-01-09 23:46:27.332 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-sovietwave.jarbasai is searching
2024-01-09 23:46:27.334 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-underrated-albums.jarbasai is searching
2024-01-09 23:46:27.335 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-horrorbabble.jarbasai is searching
2024-01-09 23:46:27.337 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-reddit-cartoons.jarbasai is searching
2024-01-09 23:46:27.338 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-smod.jarbasai is searching
2024-01-09 23:46:27.338 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-ccc.jarbasai
2024-01-09 23:46:27.339 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-WTvFDigitalGrindhouseDriveIn.jarbasai is searching
2024-01-09 23:46:27.339 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-ccc.jarbasai is searching
2024-01-09 23:46:27.341 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-lovecraft-comics.jarbasai
2024-01-09 23:46:27.341 - skills - ovos_bus_client.apis.ocp:handle_skill_search_start:887 - DEBUG - skill-wayne-june-lovecraft.jarbasai is searching
2024-01-09 23:46:27.342 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-ccc.jarbasai finished search
2024-01-09 23:46:27.342 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-lovecraft-comics.jarbasai finished search
2024-01-09 23:46:27.386 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-WTvFDigitalGrindhouseDriveIn.jarbasai
2024-01-09 23:46:27.386 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-sovietwave.jarbasai
2024-01-09 23:46:27.386 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-sovietwave.jarbasai finished search
2024-01-09 23:46:27.392 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.396 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 45 results from skill-horrorbabble.jarbasai
2024-01-09 23:46:27.397 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-WTvFDigitalGrindhouseDriveIn.jarbasai finished search
2024-01-09 23:46:27.397 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-underrated-albums.jarbasai
2024-01-09 23:46:27.398 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-horrorbabble.jarbasai finished search
2024-01-09 23:46:27.400 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 1 results from skill-smod.jarbasai
2024-01-09 23:46:27.400 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-underrated-albums.jarbasai finished search
2024-01-09 23:46:27.401 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 1 results from skill-smod.jarbasai
2024-01-09 23:46:27.402 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.405 - skills - ovos_workshop.resource_files:_locate:344 - ERROR - Could not find resource file librivox.voc
2024-01-09 23:46:27.407 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-reddit-cartoons.jarbasai
2024-01-09 23:46:27.407 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-smod.jarbasai finished search
2024-01-09 23:46:27.413 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-reddit-cartoons.jarbasai finished search
2024-01-09 23:46:27.414 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 1 results from skill-wayne-june-lovecraft.jarbasai
2024-01-09 23:46:27.414 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 10 results from skill-wayne-june-lovecraft.jarbasai
2024-01-09 23:46:27.414 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.415 - skills - ovos_workshop.resource_files:resolve_resource_file:124 - WARNING - This method has moved to `ovos_utils.file_utils` and will beremoved in a future release.
2024-01-09 23:46:27.415 - skills - ovos_workshop.resource_files:_locate:344 - ERROR - Could not find resource file librivox.voc
2024-01-09 23:46:27.452 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-wayne-june-lovecraft.jarbasai finished search
2024-01-09 23:46:28.480 - skills - ovos_bus_client.apis.ocp:handle_skill_response:921 - DEBUG - got 0 results from skill-librivoxx.jarbasai
2024-01-09 23:46:28.481 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:954 - DEBUG - skill-librivoxx.jarbasai finished search
2024-01-09 23:46:28.982 - skills - ovos_bus_client.apis.ocp:handle_skill_search_end:968 - INFO - Received search responses from all skills!
2024-01-09 23:46:29.021 - skills - ovos_bus_client.apis.ocp:remove_events:880 - DEBUG - Removing Search Bus Events
2024-01-09 23:46:29.022 - skills - ovos_core.intent_services.ocp_service:_execute_query:676 - DEBUG - Returning 5 search results
2024-01-09 23:46:29.022 - skills - ovos_core.intent_services.ocp_service:_search:650 - DEBUG - Got 58 results
2024-01-09 23:46:29.022 - skills - ovos_core.intent_services.ocp_service:filter_results:604 - DEBUG - filtered -2 low confidence results
2024-01-09 23:46:29.027 - skills - ovos_core.intent_services.ocp_service:_search:652 - DEBUG - Got 56 usable results
2024-01-09 23:46:29.027 - skills - ovos_core.intent_services.ocp_service:handle_play_intent:422 - DEBUG - Playing 56 results for: lovecraft
2024-01-09 23:46:29.028 - skills - ovos_core.intent_services.ocp_service:select_best:724 - INFO - OVOSCommonPlay selected: skill-horrorbabble.jarbasai - 60
2024-01-09 23:46:29.028 - skills - ovos_core.intent_services.ocp_service:select_best:725 - DEBUG - {'title': '"The Man of Stone" by H. P. Lovecraft / A HorrorBabble Production', 'author': 'Lovecraft', 'match_confidence': 60, 'media_type': 4, 'uri': 'youtube//https://youtube.com/watch?v=I5W3nk2Yrfs', 'playback': 2, 'skill_icon': 'https://github.com/OpenVoiceOS/ovos-ocp-audio-plugin/raw/master/ovos_plugin_common_play/ocp/res/ui/images/ocp.png', 'skill_id': 'skill-horrorbabble.jarbasai', 'image': 'https://i.ytimg.com/vi/I5W3nk2Yrfs/sddefault.jpg', 'bg_image': '/home/miro/.venvs/OCP/lib/python3.9/site-packages/skill_horrorbabble/res/bg.png'}

Issues

closes https://github.com/OpenVoiceOS/ovos-ocp-audio-plugin/issues/73

codecov[bot] commented 10 months ago

Codecov Report

Attention: 346 lines in your changes are missing coverage. Please review.

Comparison is base (972b319) 63.41% compared to head (6c9b852) 56.79%. Report is 2 commits behind head on dev.

Files Patch % Lines
ovos_core/intent_services/ocp_service.py 27.84% 342 Missing :warning:
ovos_core/intent_services/__init__.py 75.00% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## dev #396 +/- ## ========================================== - Coverage 63.41% 56.79% -6.62% ========================================== Files 14 15 +1 Lines 2069 2553 +484 ========================================== + Hits 1312 1450 +138 - Misses 757 1103 +346 ``` | [Flag](https://app.codecov.io/gh/OpenVoiceOS/ovos-core/pull/396/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenVoiceOS) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/OpenVoiceOS/ovos-core/pull/396/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenVoiceOS) | `56.79% <29.67%> (-6.62%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenVoiceOS#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.