Ensure services are fully ready before reporting readiness

[krisgesling]

This is partially a rubber duck issue to document my own thoughts, but also interested in broader input on the best way to resolve.

Currently on boot the SkillsManager loads all Skills and emits a mycroft.skills.initialized message On detecting this message PadatiousService starts training and if it's the initial train emits mycroft.ready once it's complete.

mycroft.ready is intended to signify that the device is ready to use, however there are some circumstances in which that is not entirely the case.

1. Initial training was for priority_skills only

In this instance only the Volume and Pairing Skill are loaded when Padatious runs it's training. Once the initial training is completed, it declares Mycroft ready and the "Mycroft is now ready for use" dialog is spoken by the device. At this point, asking Mycroft anything not handled by those Skills responds with "Please wait a moment as I finish booting up" because the Unknown Fallback doesn't yet exist.

Expectation: All default Skills for that platform should be loaded and trained prior to the mycroft.ready message being emitted.

2. Significant Skill updates required

If a device is reasonably outdated or there has been significant updates to a number of Mycroft Skills, it can take time for new dependencies to install and updates to be applied.

In this case, the old versions of the Skills are loaded and trained. Mycroft then reports itself ready. Following this, all the Skills start updating. Utterances received during this time may hit the old Skill, or the new Skill prior to intents being trained.

Question: Do we want to run an update of all Skills immediately after boot, before declaring Mycroft is ready? This slows the boot time, but improves the stability and predictability of responses.

3. Precise not downloaded

The Precise executable is a fairly big download and can take some time. This is now shipped as part of the Picroft image but is not part of Mycroft-core and we can't assume it is there.

Question: Should we be waiting for all assets and services to be ready before declaring Mycroft as a whole ready? Eg each service [bus, skills, audio, voice, enclosure] must itself report being ready, before the global mycroft.ready message is sent. Should we be checking that all services report being ready, before declaring that Mycroft as a whole is ready? Eg through the service hooks proposed in #2601

[forslund]

Good write-up, I've below posted some thoughts and some history on these issues, but the main conclusion is that the proposals above are good ideas according to me.

1. is this when no skills are installed or is it always the case? If skills are installed but not loaded before the initialized message is sent there's some bug in the load_new_skills() method.

2. This is the intended behaviour as discussed last summer. Prior to this we had a "2 week check" (and I think a check if all default skills were installed), if no update had been done in the last two weeks the we'd trigger an update before loading skills.

I personally think we could do a combination. The check for all default skills could lead to a blocking skill update while the two weeks check could perhaps indicate (display a message on screen on mark-1/2 devices) when checking for updates the first time after loading.

3. Possibility for querying each service for their ready status (like the skills service) was something Chris, me and David discussed last fall (should be an issue in the backlog somewhere) I do think it's a good idea to have. The hooks can be used if something external monitors this it may be good to have a polling query if the service sending the ready signal isn't the first to be loaded and ready to emit the message.

Due to the pocketsphinx fallback it may be hard to determine if it's fully ready. One way to handle this could be a notification, "I am usable but I'll be better soon when Precise is downloading".

An additional thought, we also send the mycroft.ready signal before settings are fetched due to uploading settingsmeta taking quite a long time. This is generally OK but directly after pairing when no settings are available this may lead to degraded performance. (Spotify won't play until a couple of minutes after pairing, etc)

Precise is something that could also be pre-installed, I've been looking at packaging it up as part of the packaging work I've been doing for the Focal image and making it a dependency so it's always available at the get-go.

[krisgesling]

Thanks Ake,

1. is this when no skills are installed or is it always the case?

No bug that I've seen. I was watching this after running mycroft-wipe on a picroft so it had pairing and volume to load but nothing else. Hence it reports mycroft.ready once these two Skills are loaded and trained, but then needs to download the rest of the default Skills.

2. Yeah I can see this needing some careful consideration for where the right middle ground is. I'm guessing this was part of the cutting down boot time? We don't want to wait forever to load, but I'd also rather wait another few minutes than have the users first query completely fail when it's something that should work. I hadn't thought about the Skill settings in this way too... but it's a good point.

3. I've been thinking about external monitoring, but in the short term maybe a standard Service.is_ready() that we call for each prior to issuing the all clear.

Very much agree on pre-installing Precise on images, this is included in the current Picroft images and it makes the iso slightly larger, but it's data you're going to need to download anyway.

[forslund]

Right after wipe it's sort of expected with the current implementation, ready when all installed skills are loaded no matter how few. Adding a check_and_install_defaults() stage should be relatively easy and shouldn't have much of a speed impact for vanilla images.

You're assumption is correct, it was changed to increase speed until operational status

[j1nx]

To comment on @krisgesling his statement at the pairing skill issue;

To me mycroft.ready means all the default skills are ready to rock an roll. So basically the skills listed in one of the DEFAULT-SKILLS files for the specific device. (https://github.com/MycroftAI/mycroft-skills/blob/20.02/DEFAULT-SKILLS)

The initial training of priority skills is only to be able to control the device duringh the process of getting ready. (Restarting the pairing process or controling the volume).

At least that is my view on the topic, heavily based upon the first run of mycroft. So basic ally fullu agree with point 1 and the expectation of it.

[forslund]

I posted a PR as a suggestion for how to handle the "default skill is missing" case

[krisgesling]

After having many discussions my proposal for mycroft.ready is that this message will indicate that - all five core Services [skills, audio, speech, bus, enclosure] are operational and a user can complete a basic interaction.

More specifically this means:

• Skills
  • Default Skills and any installed Skills (except any blacklisted) have been loaded. This means that the most common utterances can be handled - what time is it, who is ____, etc.
  • Non-default Skills, or Skills requiring settings or other authentication may not be ready at this time. Eg Spotify Skill may not be ready to play music but will report that it needs to sync settings first.
  • Skills may not be updated to the latest version, but pre-loaded Skills should be able to handle requests.
• Audio
  • Sound can be output by the system
  • TTS is operational and dialog is being synthesized (on-device cache used prior to this)
• Speech
  • User can activate device using the wake word selected in their device settings. This should be using Precise shipped with the release but may fallback to PocketSphinx.
  • STT is operational and utterances are transcribed
• Bus
  • Websocket server has been started and other services can connect.
  • Messages are being emitted and detected over the bus.
• Enclosure - Dependent on each platform
  • On Generic Enclosure - only that the Enclosure code is initialized and listening to Messages on the Bus. Implied by the is_device_ready() check being run.

The intention at the moment is to utilise the ProcessStatus class from PR #2645 enabling each Service to define and report its own readiness. In the short term the enclosure will be responsible for conducting the overall is_device_ready() ready check. In the future this might be moved to an independent monitoring service that is external to core.

Any comments, suggestions, or disagreements to the above are most welcome. We really want this in place for the major 20.08 release later this month.

[forslund]

Closing Issue since we're archiving the repo