mozilla-extensions / firefox-voice

Firefox Voice is an experiment in a voice-controlled web user agent
Mozilla Public License 2.0
287 stars 122 forks source link

Allow other extensions to add intent handlers #68

Open ianb opened 5 years ago

ianb commented 5 years ago

We'd like other extensions to be able to extend the capabilities of this project.

An open issue: how do we extend the intent parser given these extensions?

Two options for extensions that could support Firefox Voice to demonstrate how this works:

(These are good options because I developed them and can make the changes.)

lelouchB commented 4 years ago

@ianb Can you please elaborate on this issue?

ianb commented 4 years ago

Well, for instance, Email Tabs could want to expose its functionality "email this page". The two extensions would have to work together for this to happen, though hopefully the Firefox Voice changes can be generic.

The way I imagine this working is that extensions would register themselves using sendMessage, sending messages between extensions, like:

browser.runtime.sendMessage("firefox-voice@mozilla.org", {
  type: "registerExtension",
  intents: [
    emailThisPage: {
      match: "(email | send) (this |) (page | tab)",
      examples: ["Email this page"],
    }
  ],
});

Email Tabs would have to catch the exception if Firefox Voice wasn't installed, and might have to try to send the message a couple times because we don't know if Email Tabs or Firefox Voice will initialize first.

Then in Firefox Voice we'd have to catch these messages and remember the extension ID that sent the message. Then if someone said "email this page" we'd send a message to Email Tabs with the intent information. We'd want to allow Email Tabs to also respond with some messages to do things like display something in the popup (we don't have to expose everything to Email Tabs, just stuff it can't do on its own to integrate with our interface).

Potentially a way to start this would be to create an example extension in this repository that does something small and simple and integrates with Firefox Voice.

awallin commented 4 years ago

Another example of real world usage would be an ad-blocker extension wanting to implement functionality that let a user "unblock" the specific website the user is viewing with an utterance such as "unblock" or "show ads on this website".

lelouchB commented 4 years ago

@ianb @awallin I would like to work on this issue Though it is hard I still want to try it Please assign this to me

ianb commented 4 years ago

Cool. Please create an example extension in a new examples/ directory (or subdirectory under examples/). You'll probably have to load the second extension using about:debugging#/runtime/this-firefox (Load Temporary Add-on...)