search

SynoCommunity / spksrc

Cross compilation framework to create native packages for the Synology's NAS
https://synocommunity.com
Other
3.03k stars 1.23k forks source link

[Package Request] python-firetv #2320

Open sea3pea0 opened 8 years ago

sea3pea0 commented 8 years ago

I would really like to get this python package working on Synology. Could anyone offer some help or a hint to point me in the right direction to get this project started. I have a virtual machine set up to build it. I just haven't worked with a python package in spkrsrc before. Here is the link to the source. Thanks in advanced for any help

firetv-python provides a webserver from which you can control your amazon fire tv with commands from a browser. This is handy if you want to integrate fire tv with other smart home devices.

Update: I started by trying to build the dependency swig which in turn had a dependency pcre. So I tried to compile pcre and didn't have success with that. Here is the output from that attempted build.

Dr-Bean commented 8 years ago

That log doesn't seem to indicate that anything went wrong with compiling pcre though, does it? The libtool warnings can be expected, seeing as you're cross-compiling. Otherwise, the process completes as far as I can tell.

sea3pea0 commented 8 years ago

Okay, I wasn't sure because when I try and compile swig it complains about pcre. Here is the output from my attempt to compile swig. Here is my make file for swig.

Other dependencies for python-firetv are python-dev and libssl-dev. I was looking for source files for those packages and couldn't seem to find those. How should I handle those requirements when I get to that point?

Dr-Bean commented 8 years ago

You might have to tell swig where to find the pcre config file. The location is probably different than swig expects. For example, there may be a config arg you can pass on to swig.

As for the python-dev and libssl-dev requirements, that naming convention (the -dev part) is specific to OS'es where more conventional packaging options are available. Now, I don't know anything about python-firetv in particular, but those requirements seem to tell that firetv should be cross-compiled. You can do that with spksrc. I would suggest to take a look at other packages, such as Mercurial, to get an idea on how to cross-compile and package firetv. Also, take a look at https://github.com/SynoCommunity/spksrc/wiki/Using-wheels-to-distribute-Python-solutions. It's probably outdated as far as implementation goes, but it might serve as a bit of background. In short, you need to cross-compile Python and any other requirements before compiling firetv.

sea3pea0 commented 8 years ago

I was able to compile swig with config arguments = --without-pcre. Whether that will work for fire-tv or not remains to be seen.

So I tried to proceed with the build. I took the homeassistant spk as a template since I know it's a python package too. When I proceeded the first obstacle I found was that cross/openssl fails to compile because openssl-1.0.2g no longer is available. I changed the version in that make file to point to the current version 1.0.2h. That got me past that point. But then I found that python doesn't compile. Here is the ouput I got when trying to compile python.

Dr-Bean commented 8 years ago

That comes up more often...compile native packages separately, before anything else. Here: cd native/python && make clean && make

sea3pea0 commented 8 years ago

Thanks for pointing me to Mercurial. That worked as a good template for firetv. This is what I did with the make file. Things seemed to go okay when compiling. If I wanted to do the following, which was taken from the firetv github page, how would I accomplish this in the make file? Particularly the "[firetv-server]" part?

"If you want the HTTP server component installed as a script, use:

pip install firetv[firetv-server]"

Dr-Bean commented 8 years ago

So-called extras are supported with wheel, according to the pip documentation. I would think it's as simple as creating the wheel, and installing the extra via a requirements file as firetv[firetv-server]. The makefile isn't affected specifically, although the WHEELS line should probably point to a requirements file instead of specifying a particular wheel to install (which you should probably consider anyway, because there are multiple Python module requirements for firetv)

Btw, if you're not doing so already, I would suggest to get firetv from PyPI: https://pypi.python.org/pypi/firetv/1.0.2. Official releases via PyPI are preferred over source installs.

sea3pea0 commented 8 years ago

So I should do something like "WHEELS = src/requirements.txt" and in that lay out the python modules required by firetv including firetv[firetv-server]==1.0.2?

Dr-Bean commented 8 years ago

Sounds about right, yes. Seeing as we've never needed to use extras in spksrc, it's potentially possible that pip doesn't recognize the wheel as valid, but there's only one way to find out ;)

sea3pea0 commented 8 years ago

Okay things seem to be working. Except for the libssl-dev dependencies, I think. It seems that's why it's failing here

Dr-Bean commented 8 years ago

m2crypto is already part of the Python SPK: https://github.com/SynoCommunity/spksrc/blob/master/spk/python/Makefile#L10. If that works, you can skip the dependency. During install, you'll need to add --system-site-packages to the creation of the Python virtualenv (just as it's done in the Mercurial SPK: https://github.com/SynoCommunity/spksrc/blob/master/spk/mercurial/src/installer.sh#L24)

sea3pea0 commented 8 years ago

That didn't work. Here is the install.sh file. After I install the spk onto my syno and I try to run firetv-server from the command line I get the following message:

root@dsm:~# /usr/local/firetv/env/bin/firetv-server Traceback (most recent call last): File "/usr/local/firetv/env/bin/firetv-server", line 7, in from firetv.main import main File "/usr/local/firetv/env/lib/python2.7/site-packages/firetv/init.py", line 11, in from adb import adb_commands File "/usr/local/firetv/env/lib/python2.7/site-packages/adb/adb_commands.py", line 30, in from M2Crypto import RSA ImportError: No module named M2Crypto

Also I am wondering what these lines are for from the mercurial install.sh file:

# Add symlink
mkdir -p /usr/local/bin
ln -s ${INSTALL_DIR}/env/bin/hg /usr/local/bin/hg

Update: Here is a screenshot of what the site-packages folder's contents are after installing on my syno. It seems just the ones that are specified in the requirements.txt file get installed, nothing else.

site-packages

Dr-Bean commented 8 years ago

Maybe the virtualenv isn't setup correctly, because otherwise, there's no reason that M2Crypto isn't available for import. Check if M2Crypto is listed as installed inside the virtualenv by running a pip freeze for example. Alternatively, start the Python console inside the virtualenv, and do a from M2Crypto import RSA there. It should not produce any errors.

The lines do exactly what the comment above says: it creates a symlink for the Mercurial binary in /usr/local/bin.

sea3pea0 commented 8 years ago

This is what is listed when I exec pip freeze:

root@dsm:~# /usr/local/firetv/env/bin/pip freeze adb==1.1.1 bcrypt==2.0.0 cffi==1.3.1 Cheetah==2.4.4 cryptography==1.1.2 enum34==1.1.1 firetv==1.0.2 Flask==0.10.1 idna==2.0 ipaddress==1.0.15 itsdangerous==0.24 Jinja2==2.8 libusb1==1.4.1 lxml==3.5.0 MarkupSafe==0.23 msgpack-python==0.4.6 Pillow==3.0.0 PocketSphinx==0.8 psutil==3.3.0 pyalsa==1.0.29 pyasn1==0.1.9 PyAudio==0.2.9 pycparser==2.14 pycrypto==2.6.1 pycurl==7.19.5.3 pymongo==3.2 pyOpenSSL==0.15.1 python-gflags==2.0 PyYAML==3.11 pyzmq==15.1.0 six==1.10.0 SphinxBase==0.8 SQLAlchemy==1.0.10 Twisted==15.5.0 uWSGI==2.0.11.2 virtualenv==13.1.2 Werkzeug==0.10.4 wheel==0.24.0 yenc==0.4.0 zope.interface==4.1.3 You are using pip version 7.1.2, however version 8.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

Here is my install.sh file. Any ideas why the virtualenv isn't being set up properly?

Update: Here is the make file for the firetv spk. Here is my requirements.txt located in src/requirements.txt

Dr-Bean commented 8 years ago

I'd have started troubleshooting by recreating the virtualenv. Just follow the relevant bits in the installer, but run the commands manually.

Check if Python provided by the Python SPK has M2Crypto installed: that's where the one in the virtualenv comes from. Same as before, just a different location: /usr/local/python/bin/pip freeze. If that does show M2Crypto, then your virtualenv is borked. Remove the virtualenv and recreate it. If M2Crypto does not show up in the list, your Python package is broken somehow. I have no idea how that would be possible, but a reinstall of it would be the first place to start.

sea3pea0 commented 8 years ago

I think my python installation was borked, perhaps because I was trying to install the fire-tv module with the synocummunity python package. Anyway uninstalling and reinstalling took away the problem of the missing mcrypto. Now I am seeing this happen when trying to run firetv-server:

root@dsm:/volume1/@appstore/firetv# firetv-server shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Traceback (most recent call last): File "/usr/local/bin/firetv-server", line 7, in from firetv.main import main File "/usr/local/firetv/env/lib/python2.7/site-packages/firetv/init.py", line 11, in from adb import adb_commands File "/usr/local/firetv/env/lib/python2.7/site-packages/adb/adb_commands.py", line 33, in import common File "/usr/local/firetv/env/lib/python2.7/site-packages/adb/common.py", line 23, in import libusb1 File "/usr/local/firetv/env/lib/python2.7/site-packages/libusb1.py", line 195, in libusb = _loadLibrary() File "/usr/local/firetv/env/lib/python2.7/site-packages/libusb1.py", line 166, in _loadLibrary return dll_loader('libusb-1.0' + suffix, loader_kw) File "/usr/local/python/lib/python2.7/ctypes/init.py", line 365, in init** self._handle = _dlopen(self._name, mode) OSError: libusb-1.0.so: cannot open shared object file: No such file or directory

I've tried compiling the spk both with "DEPENDS = cross/libusb" and without. Any idea how to get past this obstacle? Thanks again for all your help throughout this process!

Dr-Bean commented 8 years ago

Well, clearly firetv is looking for a file called libusb-1.0.so. So, is that file part of the package?

sea3pea0 commented 8 years ago

It is part of the build process but it isn't getting included in the package. I have a PLIST, but unfortunately it still isn't getting included in the package. This is what my PLIST file looks like:

lnk:lib/libusb-1.0.so lnk:lib/libusb-1.0.so.0 lib:lib/libusb-1.0.so.0.0.0 rsc:lib/pkgconfig/libusb-1.0.pc

The plist file that gets created during compile looks like: include/libusb-1.0/libusb.h lib/libusb-1.0.a lib/libusb-1.0.la lib/libusb-1.0.so lib/libusb-1.0.so.0 lib/libusb-1.0.so.0.0.0 lib/pkgconfig/libusb-1.0.pc

Any ideas why none of these files are being included in the package? Here is the portion of the build process that deals with libusb.

Dr-Bean commented 8 years ago

This is not enough info. Put your branch on Github, so I can see how you're putting things together.

sea3pea0 commented 8 years ago

I put my branch up.

Dr-Bean commented 8 years ago

Two things:

sea3pea0 commented 8 years ago

Great! We're getting close to success. So now the necessary files are being included with the package. I did get a warning at the end of the build process "make: warning: Clock skew detected. Your build may be incomplete."

Also, there is still a problem with firetv not being able to find libusb-1.0.so but I was able to get it to run by your suggestion of using "env LD_LIBRARY_PATH=/usr/local/firetv/lib firetv-server -d [IP-of-FireTV-Device:Port]"

I think the next step of making this an actual package that I could contribute to spksrc would be to design a very simple UI to run the package from the Package Center in dsm with the ability to specify the IP and port of the fire tv device. After that people who install the package will be able to control their firetv devices with http commands.

Do you have any suggestions to point me towards accomplishing this? The command to run the program would be what I mentioned above. The ip & port of the fire tv device should come from user input.

Dr-Bean commented 8 years ago

Clock skew can be caused if you're building in a VM: time(-sync) is responsible for this. Unless you see errors or the build halts, you can ignore that.

As for the UI: if you're talking about an actual UI to manage the package (as Debian-Chroot has, for example), then that's a lot of work to implement. I don't think Synology provides any documentation, and our current UI's are still based on the old visuals (and it looks like all packages using one are broken on DSM6) so we can't help there.

If you're talking about a wizard which is run during package installation, that's doable. Assuming setting the IP and port is a one-time thing during package installation, then I suggest you go with that. We have several packages using wizards, take a look around spksrc.

sea3pea0 commented 8 years ago

I guess a wizard is what I had in mind. However, after giving it some thought I think it would be better to just have the package run without any parameters given upon install. All that really needs to be done to get it running is this command: env LD_LIBRARY_PATH=/usr/local/firetv/lib firetv-server Everything else can be done through the web server.

I guess what I need to do is set it up the installer to create a user named firetv-server and have it execute that command to get it running.

Is that something that can be done now with dsm 6.0? That's what I'm running and if I can't get the user to be created by the installer then I won't be able to test my work.

Dr-Bean commented 8 years ago

See #2345. We can't create DSM6 users via the installer, Synology provides some other means. However, I'd suggest to implement DSM5 compatibility as well, it's too early to go for a pure DSM6 package.

sea3pea0 commented 8 years ago

Yea I definitely would like to make it compatible with both dsm 5 and 6. I was just curious if it would be too difficult for me to proceed with my plans of setting up the firetv package to install a user and run as said user if I am running dsm 6.0 seeing as if I can't get the package to work on 6.0, I wouldn't be able to test it. Are you suggesting that I might use sabnzbd-testing as a template for setting up firetv server so it will potentially work on both 5 & 6?

sea3pea0 commented 8 years ago

So after testing this for a while, I am not so sure about setting up the spk as a run-able package with a wizard.

I found that having firetv-server running in the background waiting for commands was unreliable for me. Sometimes it would become unresponsive perhaps due to network disruption or power disruption from the firetv device.

I ended up writing a script to start the server, add the firetv device that I want to control, send the commands to effect the control I want and the kill the firetv server.

So perhaps it's just better to leave the package the way it is and leave it something for scripting and command line usage. I will try to polish around the edges and submit a pull request soon. Thanks again for your help

sea3pea0 commented 8 years ago

@Dr-Bean So I found where the location for "libusb-1.0.so" is defined. It is or can be defined in line 169 of "/usr/local/firetv/env/lib/python2.7/site-packages/libusb1.py". The line looks like this: 

libusb_path = None

I can run "firetv-server" without having to use "env LD_LIBRARY_PATH=" if I change it to :

        libusb_path = "/usr/local/firetv/lib/libusb-1.0.so"

Do you know of a way to patch that file libusb1.py on install or someway to fix this so users can just execute firetv-server without having to define LD_LIBRARY_PATH? Or if LD_LIBRARY_PATH is the only way, is there someplace that it can be defined on install so users don't have to define it every time they want to run firetv-server? I'm just trying to clean this up a little before submitting a PR. Thanks

Dr-Bean commented 8 years ago

While it's possible to patch the file via spksrc, it's probably better and easier to use e.g. a shell script that runs firetv. The libusb module's method of searching isn't really setup for anything else.

Quick 'n' dirty, let's say you create a firetv script with the following contents:

#!/bin/sh

env LD_LIBRARY_PATH=/usr/local/firetv/lib path-to-firetv-server/firetv-server -d $1

...which you call with path-to-firetv-script/firetv-script [IP-of-FireTV-Device:Port]. You could even use a wizard to set the IP:Port combo (assuming those values are perfectly static, otherwise better to have users enter it manually)

sea3pea0 commented 8 years ago

Ok thats about what I've been doing so far with my script:

#!/bin/sh

env LD_LIBRARY_PATH=/usr/local/firetv/lib firetv-server &

Quick n' Dirty. I ended up using another device to add the firetv's through json. So instead of setting a default device I just run it without setting a default.