search

fossasia / susi_linux

Hardware for SUSI AI https://susi.ai
Apache License 2.0
1.6k stars 149 forks source link

Re-design - support for async #500

Closed norbusan closed 5 years ago

norbusan commented 5 years ago

Currently, we have one state machine

idle   --(hotword)-->  recognizing  --(recognized)--> busy   ----> idle
                                |---------(not recognized)--> error ----^

In addition, when in the busy state we have a stop detector that allows to interrupt audio/video playback (in principle).

But other actions, like searching, volume, asking the time etc are not possible while in busy state.

I propose the following:

Concerning music thread

implementation notes

states and transitions on messages

Hotword/recognizing state machine:

Processing state machine

Implementation comments

Implementation can be done via threads, but would suffer from the GIL (Great Interpreter Lock). Threads would be easier to communicate between them, process would needs some IPC. ATM we probably don't need that much computing power so the GIL might not be a problem.

hongquan commented 5 years ago

Agree about threads. Most of the work are done by external program: VLC, calling Google API, so GIL is not a problem.

sansyrox commented 5 years ago

How will this thread work? How will we pass the recognized audio to this thread?

one processing state machine states: - idle - busy

norbusan commented 5 years ago

So many options, just search for python inter thread communication.

For example a queue https://www.bogotobogo.com/python/Multithread/python_multithreading_Synchronization_Producer_Consumer_using_Queue.php

norbusan commented 5 years ago

Controlling VLC via python - it is much better to use the python vlc module, here example code:

import vlc
from time import sleep

player = vlc.MediaPlayer("music/beyond.wav")
player.play()
print("Playing song\n")
sleep(3)
player.pause()
print("Pausing song\n")
sleep(3)
print("Starting song again")
player.play()
sleep(3)
print("Changing song")
player.set_mrl("music/junge.wav")
player.play()
sleep(3)

This can easily be integrated into a receive message system sent from other threads.

norbusan commented 5 years ago

Just for the reference, here is a minimal command line player that supports

play <MRL>
pause
resume
stop

and all this is done via a separate thread

BUT: we actually don't need a separate thread, since the VLC module returns immediately, we can just wrap all that up in more simple calls in the state machine ... working on that now ...

#!/usr/bin/python3

import threading
import queue
import vlc
import sys

cmd_q = queue.Queue()

def vlc_thread():
    player = vlc.MediaPlayer()
    while True:
        s = cmd_q.get()
        if s is None:
            break
        cmd = s.split(" ",1)
        if (cmd[0] == "play"):
            print("Setting mrl to " + cmd[1])
            player.set_mrl(cmd[1])
            player.pause()
            player.play()
        elif (cmd[0] == "pause"):
            player.pause()
        elif (cmd[0] == "resume"):
            player.play()
        elif (cmd[0] == "stop"):
            player.stop()
        else:
            print("\nVLC thread: Got cmd >>" + cmd[0] + "<< and rest " + cmd[1] + "\n")

def main(args):

    vlc = threading.Thread(target=vlc_thread)
    vlc.start()

    running = True
    while running:
        line = input('VLC cmd: ')
        if (line == "quit"):
            cmd_q.put(None)
            vlc.join()
            running = False
        else:
            cmd_q.put(line)

if __name__ == '__main__':
    main(sys.argv[1:])
sansyrox commented 5 years ago

@norbusan , if we can run this through a flask server, we can also integrate the play/pause/stop/restart functionality in the mobile app and control the music via the app as-well.

Also , we were using the python-vlc module before but we removed it because of some issue.

norbusan commented 5 years ago

@stealthanthrax yes, that would be possible. Then the processing would just curl to the url/port to play a song or say something.

Concerning python-vlc, indeed, now I see that it is in the requirements.txt, but nowhere used.

Do you have any further information why it was not used or what was the problem?

sansyrox commented 5 years ago

@norbusan , vlc was removed here(https://github.com/fossasia/susi_linux/pull/450/files) in order to add multiple query support for music player. And cvlc was run as a background process instead.

norbusan commented 5 years ago

@stealthanthrax Yes indeed, but I don't see how this properly deals with async support? And why this couldn't have been realized by having a thread (or the flask server as before) doing audio output?

The discussion states

This PR now uses cvlc instead of vlc module, to allow background process for an asynchronous approach of SUSI.

but I don't see that change actually happening. What has happened is that the StopDetector was added, that allows for partial recognition during playback.

Before I do all that back with python-vlc, I would really like to see how you came to the point of saying that it allows asynchronou access, and why this is not possible using python-vlc. Actually, I think it is much easier with it.

sansyrox commented 5 years ago

@norbusan Now I remember. I was unable to implement a multi-threaded architecture for the complete system and hence I used the cvlc module to play/pause and stop the music using a temporary way as we needed a way to add music controls at that time.

norbusan commented 5 years ago

@stealthanthrax aah, ok, that explains a lot ... but, btw, why do we need a multi threaded approach for the vlc playback? The commands of python-vlc (like .play(), .pause(), ...) return immediately, so one can actually implement playback control, including reduction of volume etc, in the same main thread.

An example (quick and dirty hacked together while watching my child) is in the branch better-vlc-control, please look at it. Basically it imports a VlcPlayer module, and uses its commands:

class VlcPlayer():
    def __init__(self):
        self.instance = vlc.Instance("--no-video")
        self.player = self.instance.media_player_new()
        self.list_player =  self.instance.media_list_player_new()
        self.list_player.set_media_player(self.player)

    def play(self, mrl):
        media = self.instance.media_new(mrl)
        media_list = self.instance.media_list_new([mrl])
        self.player.set_media(media)
        self.list_player.set_media_list(media_list)
        self.list_player.play()

    def pause(self):
        self.list_player.pause()

    def resume(self):
        self.list_player.resume()

    def stop(self):
        self.list_player.stop()

    def wait_till_end(self):
        playing = set([vlc.State.Playing, vlc.State.Buffering])
        time_left = True
        while time_left == True:
            pstate = self.list_player.get_state()
            if pstate not in playing:
                time_left = False
            print("Sleeping for audio output")
            time.sleep(1)

    def say(self,mrl):
        # this is tricky!!!
        if self.list_player.is_playing():
            self.list_player.pause()
        # TODO wait for the length of what is said, then restart main player!
norbusan commented 5 years ago

This is of course work in progress ..

sansyrox commented 5 years ago

@norbusan , this looks nice!! What is the say function though?

norbusan commented 5 years ago

"say" is for short utterances that may interrupt audio playback (music) but the music would be restarted after that. So if I listen to audio I can ask a question, get an answer, and the music would continue.

norbusan commented 5 years ago

@stealthanthrax I have tried moving all sound playback (including the ding sound etc) to a flask server and make the susi_linux only hit the respective URLs. That does work indeed, but adds a bit of latency. I am not sure which approach I want to suggest. My current dev branch does all the stuff within the python code, no server is used.

I think it boils down how important music playback via mobile apps is - and I guess it is ...

sansyrox commented 5 years ago

@norbusan , I wouldn't recommend removing the flask server as it would restrict a lot of future functionalities.

norbusan commented 5 years ago

@stealthanthrax I agree, see my comments in the gitter channel. But what the server is providing at the moment is just a json answer of the url of the audio track of a video using pafy, not doing any media playback whatsoever.

I probably will go with my last version:

hongquan commented 5 years ago

@norbusan About a dedicated server to play audio and react to pause/play/volume changes, I already thought about MPD. Do you have the same thinking?

norbusan commented 5 years ago

Hmm, I was thinking about some home-grown Flask that used python-vlc. It is so simple, I had already the code finished and working, just about 50 lines or so. MPD is nice, but big, and does much much more that we really need.

hongquan commented 5 years ago

Then, it is Ok.

norbusan commented 5 years ago

@hongquan Ok, here is the code:

My plan is to use the soundserver on the raspi, and play music from susi_linux via the sound server.

norbusan commented 5 years ago

From the git commit:

    add a sound server and vlcplayer

    The vlcplayer is a module that can be loaded and provides
    playback capabilities via python-vlc.
    The soundserver is a Flask server running at port 7070 and
    reacting to requests using an instance of the vlcplayer.

    Supported URL schemata are:
    - /play?ytb=???         play a youtube video, argument is vid
    - /play?mrl=???         play an arbitrary MRL
    - /volume?val=up        volume up by 10% point
    - /volume?val=down      volume down by 10% point
    - /volume?val=NN        adjust volume to NN (0<=NN<=100)
    - /say?mrl=???          reduce volume if currently playing, play mrl,
                            restore volume
    - /beep?mrl=???         like say, but does not restore volume afterwards
    - /pause                pause if playing
    - /resume               resume if paused
    - /stop                 stop playback
    - /save_volume          save current volume
    - /restore_volume       restore current volume
norbusan commented 5 years ago

merged