Order fallback

[corus87]

Hi, I just saw the STT fallback issue and thought it would be time to suggestion an order fallback I have implemented in my kalliope a while ago. If an order not getting matched then there will be execute a fallback order. My idea was to trigger the answer_of_everything neuron if no other order is matched the order analyser would look for a fallback order where the question starts with something like where, which, when etc. so we can ask kalliope in a natural way of speech a question.

Therefore I had to modify the OrderAnalyser here and add a function here.

This would be the synapse to ask for a question:

  - name: "answer-of-everything-all"
    signals: 
      - order: 
         text: "{{ query }}"
         fallback:
          - "where"
          - "when"
          - "who"
          - "what"
          - "how"
          - "why"
          - "Since"
          - "which"
    neurons:
      - answer_of_everything:
          question: "{{ query }}"
          language: "de"
          engines:  
                wolfram_alpha: 
                    key: "xxxxx-xxxxx"
                    priority: 1
                google:
                    priority: 2
                duckduckgo:
                    priority: 3

          file_template: "templates/answer_of_everything.j2"    

So if I ask a question like “how tall is the eiffel tower” and no other order is matched, the fallback will match. Also like you see I have done some improvements to the neuron, I will release the new version in the next days, just have to make some changes regarding python2 and 3 support.

I think it could be a nice feature, what do you guys think about it?

[Sispheor]

I like the idea, but I'm not sure about the implementation. Would it not be better to update the current "on_order_found" hook to receive the order and so call a new synapse with it?

[JulienSambre]

Hey!

I have made myself a fallback order implementation in a different way :

1. When an order is not match, it triggers the hook "on_order_not_found" and then the called synapse will run a specific neuron I developped.
2. This neuron will try to "fix" the order in order to patch a misunderstanding from the STT (Google STT for example does not understand my girlfriend's name). I could have used the STT correction of Kalliope but I didn't succeed to work with it (tried an hour only) and I wanted to centralize all unmatched orders with different strategies (run a fixed order, execute a synapse, etc.) 3.a. If a fix has been found and the strategy is to patch the order, then the neuron will call a shell script which will call the API with the patched order (didn't find a proper way to submit an order from a neuron) 3.b If no fix has been found, then it will say it had not understood the order and I disable the trigger to allow smooth retry of order (after X retry, the neuron will re-enable the trigger to avoid, well crash when looking the TV for example :)).

For precision, when an order is matched, I enable "de facto" the trigger to avoid endless disabled trigger.

So all in all, I made a much complex things (maybe too complex :)).

For your case, I would use the "on_order_not_found" hook and make a simple neurone wich would search your keywords in the querry (where {{ query }} come from? kalliope_memory['kalliope_last_order']?).

For a more generic approach, the neuron could take in arguments the list of wanted words.

The main concern is mainly the actions to do for the fallback : specific ones (like engines) or generic ones (synapses).

All in all, I think this kind of feature (fallback) is a must have ;). Great idea!

[Sispheor]

Actually, it's even already possible to do it. With the master code, could you try something like this?

Settings:

hooks:
  on_order_not_found: "answer-of-everything-all"

Brain:

  - name: "answer-of-everything-all"
    signals: []
    neurons:
      - answer_of_everything:
          question: "{{ kalliope_memory['kalliope_last_order'] }}"
          language: "de"
          engines:  
                wolfram_alpha: 
                    key: "xxxxx-xxxxx"
                    priority: 1
                google:
                    priority: 2
                duckduckgo:
                    priority: 3

          file_template: "templates/answer_of_everything.j2"  

[corus87]

@Sispheor You are right, it does work, but it has some disadvantages like it will always trigger the answer_of_everything neuron even it is not supposed to, or it does not allow to reask again with the retry-counter.

I like the way more to specify in which way the order acts like a fallback, starting with specific keywords like “where”, “when” or “how”. Also I´m not sure if we should work with the hooks, cause I think it would perfectly fit as a new order parameter, since its a special way the order should be handled.

@JulienSambre Your idea is not bad with the on_order_not_found hook to call a neuron that's search for specific start words in the order and then execute the desire synapse. But it would also got in conflict with the retry-counter, cause I really appreciate the possibility that kalliope automatic trigger the hotword if the order is not found or on a stt-error, so I get my second chance to ask again without calling the hotword.

For me its working really great this way, I have a lot of synapses and almost every questions I ask, which have the fallback keywords in it, get parsed to the answer_of_everything neuron. And all the other orders which get not found, will trigger the on_order_not_found hook.

[JulienSambre]

But it would also got in conflict with the retry-counter, cause I really appreciate the possibility that kalliope automatic trigger the hotword if the order is not found or on a stt-error

-> is this a core feature (retry-counter) or is it an implementation on you side? For letting the user do X retry, I had to do some tweaks to disable the trigger when it is appropriate in order to ask an order again, because of some flaws on how API call is tagged.

Therefore, after thinking about it, I think we could make a generic mecanism in order to satisfy every strategies when an order is not matched. For exemple :

Settings

hooks:
  on_order_not_found: "fallback-synapse"

Brain

  - name: "fallback-synapse"
    signals: []
    neurons:
      - fallback:
            - strategy:
                priority: 1
                conditions: 
                    order-contain:
                        - "a"
                        - "b"
                        - "c"
                    order-startwith:
                        - "thing"
                call_neuron:
                    param1: value1
                    param2: value2
            - strategy:
                priority: 2
                conditions: 
                    order-contain:
                        - "d"
                call_synape: synapse-name
            - strategy:
                retry

I made that synapse as an idea, because it seems a little too complexe in order to match "classic" fallback strategies. All in all,

@corus87 your use case remember me how i loved the idea of "asking a question to the internet". I'll keep your idea, may it be a new neurone/synapse or a fallback strategy as you did. I am curious how you call the different information providers and how you process the answers : is your code on Github?

[corus87]

The retry-counter is a core feature yes, you basically need 4 synapses:

  - name: "set-skip-trigger-max-retry"
    signals: []
    neurons:
      - signals:
          notification: "skip_trigger_max_retry"
          payload:
             max_retry: 2

  - name: "decrease-max-retry-counter"
    signals: []
    neurons:
      - signals:
          notification: "skip_trigger_decrease_max_retry"     

  - name: "start-skip-trigger"
    signals: {}
    neurons:
       - signals:
           notification: "skip_trigger"
           payload:
              status: "True"

  - name: "stop-skip-trigger"
    signals: {}
    neurons:
      - signals:
          notification: "skip_trigger"  
          payload:
            status: "False"

and then you have to assign the hooks:

hooks:
  on_start:
     - "set-skip-trigger-max-retry"
  on_waiting_for_trigger: 
    - "set-skip-trigger-max-retry"
  on_triggered:
    - "start-skip-trigger"
  on_order_found:
    - "set-skip-trigger-max-retry"
    - "stop-skip-trigger" 
  on_order_not_found:
    - "default-synapse-not-found"
    - "decrease-max-retry-counter"
  on_stt_error:
    - "default-synapse-not-understand"
    - "decrease-max-retry-counter"

But it is also mention in the docs.

I think your example neuron could work, but therefor we have to disable the retry-counter what would be possible from the neuron. But I guess it will be a very complex and complicated neuron where we also have to define if/else statements in the yaml config to disable or enable signals etc.

That's one of the reasons why I think adding a new parameter to the order, would be the easiest way.

In your case with the stt fallback it maybe the better solution to work with the hooks.

Yes the code for the answer_of_everything is on github but still in development. The idea was to make one neuron with different search engines. At the moment I have a working neuron which searches in 3 different engines, wolfram alpha, google and duckduckgo, which you can prioritize, if the first one doesn't find an answer, the next one is looking and so on.
I'm almost finished but struggling with python 2 and 3 support for duckduckgo, because the lib for python2 is very outdated (last update 5 years ago). Its possible that I add duckduckgo only for python3.
But anyway most answer you get from wolfram alpha and google.

[JulienSambre]

Thank you for the precision. Your example remind me of that, even if I don't see that explanation in the link you gave, I read about it in an issue. I remember that this mecanism (retry-counter) comes with a problem that I had to manage with : If I send a wrong order via API, it will trigger the skip trigger, which for me make no sense as an API call is not a use case where I need to skip the trigger. And for example, if my girlfriend look at TV at home at the same time, Kalliope could do unwanted things (and it did, matching wrong orders and starting to talk to her, which scares her :D), while I use the API (at work for example).

That's why I had to (and for other reasons) implement a fallback neuron which can handle the difference between API call and voice order, may it concerns the retry counter and the skip trigger.

Thanks for sharing your code, I'll take a close look and will try to make it work on my side too ;).

[corus87]

Hmm I just send a wrong order via the Kalliope app, and it does not affect the skip trigger, maybe you had a wrong configuration in your settings?

But you are right, if I sent a voice order by recording even if I set the mute flag, Kalliope is talking. Kalliope is only mute, if I send the order by text.

@Sispheor I suppose the mute flag should also mute Kalliope if I send the order by voice recording or could it be only an issue with the app?

[JulienSambre]

I tested what you say and it works as you say. Damn it, I think I've implemented an existing thing :).

Maybe should it be added to the doc? I only find the skip/unskip trigger notification, but nothing about "skip_trigger_max_retry" or "skip_trigger_decrease_max_retry".

I'll keep my implementation for the moment (because I included it in a short memory implementation where Kalliope "forget" the counter after X minutes of no false order). But I think I will delete what I did when I will refactore my code in a month or two ;). Thanks for the advice! We can be back on the subject ;).

[Sispheor]

So this one can be closed? The feature is present, right?

[corus87]

In my opinion no. There is a possibility with the hooks to accomplish some kind of order fallback, but my idea was to implement a new order parameter, because it has more advantages over the hook method. It's a bit special for orders where you ask general questions, but it works great, I'm using it now for a long time, if I ask Kalliope --> "when is the next formula 1 race" no other synapse can match my question, so the the order fallback comes in and check if the order starts with one of the fallback words like "when" and if so the order fallback knows it is a question so it triggers the answer_of_everything neuron.

With the hook method, every order which does not match a synapse, will trigger the answer_of_everything neuron, which can be some kind of annoying.

But if you think it couldn't be a feature, then it can be closed.

[Sispheor]

ok, thanks. I keep it then.