Fix installation process

hongquan commented 6 years ago

This issue is to collect some fix need to make in the installation script.

The target is to make the installation work on Raspbian Stretch (9) and Ubuntu 18.04, and make the process simpler.

Some core idea:

Try to install the dependency software from the OS repository (via apt) and from Python PyPI (via pip3) first. Only install from source (make) if the previous method fails.

ISSUES To be solved:

[x] SWIG should be installed from Debian repository.
[x] libatlas-dev doesn't exist. Install libatlas-base-dev instead.
[x] Tizonia default installation script only installs Python 2 packages (from https://dl.bintray.com/tizonia/raspbian), not Py3. Doesn't have a fix yet!
[x] The final step in install.sh doesn't prompt the user to use python3 config_generator.py however, the new format is 'python3 config_generator.py stt tts hotword wake'
[x] The 99-com.rules file is of Raspbian. We must not write our rule to it. Write to new file, so that when we uninstall the app, we just delete our file.
[x] Some Python packages, like snowboy, has to be installed from Git source, just because the version published to PyPI is too old.
[x] Try to install the dependency software from the OS repository (via apt) and from Python PyPI (via pip3) first. Only install from source (make) if the previous method fails.

hongquan commented 6 years ago

SWIG should be installed from Debian repository.
libatlas-dev doesn't exist. Install libatlas-base-dev instead.

hongquan commented 6 years ago

Tizonia default installation script only installs Python 2 packages (from https://dl.bintray.com/tizonia/raspbian), not Py3. Doesn't have a fix yet!

sansyrox commented 6 years ago

The final step in install.sh doesn't prompt the user to use python3 config_generator.py however, the new format is 'python3 config_generator.py stt tts hotword wake'

hongquan commented 6 years ago

From @stealthanthrax feedback, we use Tizonia just for playing Youtube, but Tizonia is buggy, support too many services, and doesn't support Python 3, so we should find a replacement.

hongquan commented 6 years ago

Flite can be installed via apt.
The udev rule in _media_daemon/mediadaemon.sh looks weird:

echo "ACTION==\"add\", RUN=\"/home/pi/SUSI.AI/susi_linux/media_daemon/autostart.sh\"" >> 99-com.rules       
echo "ACTION==\"remove\", RUN=\"/home/pi/SUSI.AI/susi_linux/media_daemon/autostop.sh\"" >> 99-com.rules

It is supposed to tell udev to run the _mediadaemon/autostart.sh script when USB drive is plugged in, and run the _mediadaemon/autostop.sh when USB drive is removed. But the full rule

ACTION=="add", RUN="/home/pi/SUSI.AI/susi_linux/media_daemon/autostart.sh"

doesn't tell udev to only look into USB devices!

By the way, after writing these lines of rules, my Pi's wifi stop working (don't know if it is broken by this udev rules or not).

The 99-com.rules file is of Raspbian. We must not write our rule to it. Write to new file, so that when we uninstall the app, we just delete our file.

hongquan commented 6 years ago

When I plug my USB drive in, then run python3 auto_skill.py manually, the script doesn't detect my drive.

$ python3 auto_skills.py                                                                        
[]
<class 'list'>
<class 'list'>
Traceback (most recent call last):
  File "auto_skills.py", line 51, in <module>
    make_skill()
  File "auto_skills.py", line 9, in make_skill
    print(name_of_usb[0])
IndexError: list index out of range

while lsblk still shows my drive:

$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    1  7.5G  0 disk
└─sda1        8:1    1  7.5G  0 part
mmcblk0     179:0    0 29.7G  0 disk
├─mmcblk0p1 179:1    0 43.1M  0 part /boot
└─mmcblk0p2 179:2    0 29.7G  0 part /

hongquan commented 6 years ago

Some Python packages, like snowboy, has to be installed from Git source, just because the version published to PyPI is too old.

To ease the process, I think one of us can make the wheel package of the newer version and upload to some PyPI-like host, so other can install quickly via pip3. And the advantage of wheel package is that, it is prebuilt binary, the next user doesn't need to build again and also doesn't need to install library header packages (libxxx-dev).

This is example, when I built snowboy from Git source to wheel package, hosting it somewhere, and let pip download from that host when installing snowboy in other machine.

$ sudo pip3 install snowboy
Collecting snowboy
  Downloading https://repo.fury.io/agriconnect/-/647580/snowboy-1.3.0-py3-none-any.whl (12.3MB)
    100% |████████████████████████████████| 12.3MB 21kB/s
Requirement already satisfied: PyAudio in /usr/local/lib/python3.5/dist-packages (from snowboy)
Installing collected packages: snowboy
Successfully installed snowboy-1.3.0

In the install.sh script, we are trying to copy snowboy to _susilinux folder:

if [ -f _snowboydetect.so ]; then
    echo "Moving files"
    cp _snowboydetect.so ${root_dir}/main/hotword_engine/snowboy
    cp snowboydetect.py ${root_dir}/main/hotword_engine/snowboy

I think this practice is not good. We should let the 3rd-party software live in its standard folder.

sansyrox commented 6 years ago

@hongquan , the autoskills.py script is working fine on my side , please try using a different USB or the same USB with the different name

hongquan commented 6 years ago

@stealthanthrax Found the cause: Your script tries to call mount command, which requires root/sudo permission. But our app, as every general application, should not be run under root, or sudo.

hongquan commented 6 years ago

More discussion about "Media Discovery not working" are in #223.

More discussion about "Media Discovery breaks wifi" are in #226.

prateekiiest commented 6 years ago

@hongquan are you working on https://github.com/fossasia/susi_linux/issues/219#issuecomment-402227526 to keep the 3rd party software in its required folder ?

May be I can send a PR for you?

hongquan commented 6 years ago

@prateekiiest No, I'm not working on it.

prateekiiest commented 6 years ago

Ok, then I will open an issue for it and send a PR accordingly

hongquan commented 6 years ago

Regarding snowboy issue, we have 3 things to do:

Delete local snowboy folder.
Install snowboy by standard method (pip3 install or python3 setup.py install)
Fix the hotword_engine to let it use standard snowboy. Currently, it is relying on the local snowboy.

It will be good if you can combine these to 1 PR, so that it is not half-cooked and break the system.

prateekiiest commented 6 years ago

https://github.com/fossasia/susi_linux/issues/215#issuecomment-402923779

sansyrox commented 6 years ago

@hongquan , swig versions are being installed from source as well as the debian repo's. Should I delete the one from the source just like that??

hongquan commented 6 years ago

We need to convert susi_api_wrapper to setuptools-based package (https://github.com/fossasia/susi_api_wrapper/issues/57). After that, just use pip to install susi_api_wrapper, don't copy it to susi_linux.

hongquan commented 6 years ago

@stealthanthrax Yes, please delete the step of installing swig from source.

woshikie commented 6 years ago

According to https://github.com/fossasia/susi_linux/blob/master/install.sh I believe that swig is first attempted to be installed from apt at https://github.com/fossasia/susi_linux/blob/7157726e4ce3afd75a5098122a6dffc1c7086b03/install.sh#L161 which is defined at https://github.com/fossasia/susi_linux/blob/7157726e4ce3afd75a5098122a6dffc1c7086b03/install.sh#L6-L12 before the script attempts to install swig by source at https://github.com/fossasia/susi_linux/blob/7157726e4ce3afd75a5098122a6dffc1c7086b03/install.sh#L101 which is defined at https://github.com/fossasia/susi_linux/blob/7157726e4ce3afd75a5098122a6dffc1c7086b03/install.sh#L74-L97

woshikie commented 6 years ago

@hongquan As explained above, I believe you may check the first point

hongquan commented 6 years ago

SWIG always exist in OS repo, and already at version 3.0.12, so please remove the step of installing SWIG from source. That step brings the burden of maintaining the additional code, which has very little helpful.

woshikie commented 6 years ago

@hongquan After digging around a little, I found out that you can install the latest snowboy from npm. Perhaps we can use npm to install instead of manually building it

This may concern #230 and #231

Reference:

hongquan commented 6 years ago

I found out that you can install the latest snowboy from npm.

@woshikie It is to install NodeJS binding of Snowboy. Our project is Python, not NodeJS.

woshikie commented 6 years ago

Ahh I see. My bad

fossasia / susi_linux

Fix installation process #219