TVheadend (New Appliance)

[ghost]

Working on a Turnkey DVR server that one can essentially throw a tv tuner at or an iptv account and have a set it and forget it iptv/dvr solution. TVheadend has most of the functionality built in except for a few nice to haves like U.S. EPG and some automatic video editing. I am working from core, but there may be some value in integrating this with the file server appliance, torrent appliance, or media server appliance. I am planning it standalone as of now, but if there is a preference I can build in all or portions of the existing appliances.

Considering some of the initial configuration is location specific (i.e. no reason to configure U.S. EPG if you don't live in the U.S., or just using IPTV). With the concept of bypassing configuration steps not applicable to the end user is there an existing appliance with an example of a yes/no dialog to choose whether to configure an option or bypass the configuration? This might also get complicated for seeded installs.

[JedMeister]

Hey @MPTMG Awesome work mate!

My inclination would be to do all the default/common config within the inithooks, then any additional config options you'd like to provide, include them as confconsole plugins.

Also, I haven't publicised it yet, because it's still work in progress, but if you'd like to start playing with the new Stretch based builds, you can. To build a Stretch based appliance, do the following on your TKLDev (assuming that turnkey is the git remote for the TurnKey GitHub repos, if you are still using origin - or any other remote name, please adjust the first line as necessary):

TKL_REMOTE=turnkey
cd common
git fetch $TKL_REMOTE
git checkout -b stretch-dev -t $TKL_REMOTE/stretch-dev
cd tvheadend # assuming that's the name of your repo
export RELEASE=debian/stretch
export DEV_TEST=y

Please note that by setting the DEV_TEST environment variable, you will be using my private TurnKey repo (dev-test.jeremydavis.org) so please only use anything you build for personal testing. DO NOT redistribute any ISOs that you build as they will be locked to my repo and won't get any updates (and will likely break when I clean up my repo). We're fairly close to releasing v15.0 Core as a beta/RC. Before we do that, we'll upload the packages to the proper TurnKey repo so you won't need to set the DEV_TEST var. Once we get to that point, then you can redistribute any ISOs you build.

[ghost]

I will setup the test environment, so I can ensure everything is ready for v15.0 when it releases. Thanks for the confconsole plugins suggestion as that alleviates my concerns with inithooks and bypassing those not applicable to the end user. I don't redistribute any ISO as my personal opinion is that TKL apps should be distributed by TKL. If someone were to use what I have done they would have to clone my repo and build it themselves. As is such... when time permits and if the work warrants it consider it for inclusion in TKL.

[JedMeister]

We'll certainly be aiming to include it, but we won't be able to add it until we get the v15.0 release a bit further along (we're not building any new v14.2 apps now).

[ghost]

confconsole plugins... quick question. For development purposes I assume I create a plugins.d directory inside /etc/confconsole and add applicable sub directories and applicable python files/text files for carrying out necessary configuration/setup. The current confconsole (advanced menu) configurations (lets encrypt.....) are not stored in a similar manner I assume because no plugins.d folder exists in the file system from a simple "find" search. Just trying to make sure I understand https://www.turnkeylinux.org/docs/confconsole/plugins dev instructions as it never explicitly states where plugins.d should reside other than:

"Plugins are python source files which are placed within the ConfConsole directory. ConfConsole will load any applicable files and provide them to the user as menu options."

Is /etc/confconsole = ConfConsole directory?

[JedMeister]

Sorry if that is unclear. I didn't actually work on that much myself, but IIRC the directory for plugins should be /usr/lib/confconsole/plugins.d. We should probably update the docs to make that really explicit.

New confconsole plugins would be added to the appliance overlay, using the same sort of model that we currently use for including appliance specific inithooks.

Also just as a clarifcation (in case you were wondering) technically end users putting files in /usr (other than /usr/local) is considered bad form as it's normally a package managed area of the filesystem. But we're not end users, and we're only doing it for packages which we have created! :smile: Ideally at some point in the future, we should actually package all the appliance specific additional files to separate packages. So in this case, a confconsole-tvheadend package. But that won't happen anytime soon, so no need to consider that for now. I just wanted to share the dream. :smile:

[ghost]

A bit of work left to automate EPG grabbing in the U.S. via confconsole menu.

[JedMeister]

Great work!

[ghost]

Republished my repo after code clean due to rights/limitations on use of script to retrieve Electronic Program Guide(EPG) in the United States. Code to automatically fetch the EPG came with the following disclaimer:

downloads are for personal use only, not for redistribution, conversion, integration or other use

If they update the Python versions (which are no longer working due to changes in how Zap2it displays data), I will consider integrating those. It appears though that they are essentially just ports of the perl script which I think would fall under the conversion clause of the original programmers limitations. zap2xml is provided to anyone free of charge, but it seems the original coder would prefer it not be integrated or redistributed even in it's original form. Anyhow work complete waiting on release of TKL-dev for stretch to be updated.

[ghost]

Builds and works fine on TKL-dev 15.0 rc1

[ghost]

Made the necessary changes to using third party repo, formatted README, and added logo and screenshots. This should be feature complete for you to test.

[JedMeister]

In essence this looks like a great app, but after having a fairly extensive look, the only thing I'm not really happy about is that I can't lock down access to the WebUI and the streaming protocol (HTSP). Considering that users may wish to run this online, that's not really ideal...

I tried using their "Access Controls" and whilst it does appear to stop login, it doesn't actually stop access. Besides, it doesn't really have any way of encrypting the traffic.

So the web UI is still accessible over plain http via the port you set (9980). TBH, that in and of itself, isn't that bad I guess (not ideal, but not that bad). But then there is also the streaming protocol that is enabled (HTSP).

After a bit of digging, I have discovered that I can lock the whole thing down (bind it to localhost). That resolves the web UI access, as now it's only via the Nginx reverse proxy.

However, now we have the issue of not being able to access the streaming protocol (HTSP via port 9082). From what I can gather, that will likely be a problem... I had a quick google but couldn't find anything OOTB that might be of help. I did find this doc page that talks about it a bit, but I'm still not really clear whether we can leverage Nginx to also reverse proxy that or not?! TBH, I don't think so (it is TCP, but not http AFAIK). Perhaps we use Squid to proxy that or some other proxy mechanism?

TBH, I'm also not sure what other implications there may be?!

Should I push the code I have so you can have a play locally and see what implications there might be?

[JedMeister]

FWIW you can see what I've done so far in the code-review branch I've pushed to my fork of the repo. FWIW here's a link to the diff.

[ghost]

Yeah I had concerns about htsp not being encrypted as well. As far as blocking the unencrypted webui access it is as easy as a firewall rule preventing access from anything but localhost, such that the nginx proxy does it's job. So essentially just lock down 9980 from all but localhost.

From reading forum posts it seems that password transmission for htsp is sha1 hash protected but the stream itself is not. https://tvheadend.org/boards/5/topics/22465

AFAIK there is no way to proxy htsp and have it function as designed. Push the code and I will make changes to see if there is some way to proxy htsp (trial and error). Have not seen anyone report it working, but somebody has to be first.

Similar to issues with anything designed for LAN access (samba, afp, etc...) there is always the vpn option and locking it down via firewall to only connections from the vpn network.

Happy to continue tweaking until your happy with it. I can throw a usb tv tuner at a vm and see about htsp streaming behind a proxy.

[JedMeister]

I agree that leveraging the firewall is ideal, but until we enable that by default (which we probably should consider at some point - but we're too close to v15.0 release to do that now) unfortunately we can't rely on that.

Seeing as the stream isn't encrypted perhaps having it totally locked down by default is the best option?!

Re proxy, it looks like what we'd need is a SOCKS proxy. Apparently SOCKS5 supports authentication too. My reading suggests that OpenSSH (what provides the SSH server on TurnKey; clients available for Linux and Mac) and PuTTY (client for Windows) can provide SOCKS.

Alternatively, it seems that arbitrary TCP packets can still be wrapped in TLS (doesn't have to be http) and TLS can also support authentication. Stunnel is a cross platform TLS wrapper server/client.

The down side of all that though, is that there would also need to be client side decryption. It could be documented for Windows, Mac and Linux. But if it's some sort of off the shelf media player, then that may not be possible.

We'd still want to document how to unlock it (for users running it on a LAN - or those who want to live dangerously). Or how to use [insert proxy of choice here] to provide the HTSP connection (when running it on the internet). Then end users could decide the best compromise for themselves?!

It does make the setup a bit more complex and documentation would be really important. Also, like you say, the VPN route is also an option, although IMO, that makes the setup a fair bit more complex (although arguably, no more complex than using SOCKS/SSH or TLS).

[JedMeister]

Push the code and I will make changes to see if there is some way to proxy htsp (trial and error).

Instead of me pushing the code back to TurnKey (which I'm not really keen to do just yet), you could add my repo as a remote. I.e.:

cd /path/to/tvheadend-repo
git add remote jeremy https://github.com/JedMeister/tvheadend.git
git fetch jeremy
git checkout code-review

The above should work fine on TKLDev v15.0 (git should realise that you don't have a local code-review branch and checkout mine instead). Although if you have issues with the last line, try this instead:

git checkout -b code-review jeremy/code-review

[ghost]

Integrated all the necessary firewall bits to make port 9982 only accessible from private networks and pushed a code-review branch to my repo. Using plan/main added ufw and via conf.d/main added necessary rules and enabled firewall. Protects port 9980 as default is deny. Protects 9982 as only private networks allowed. Enables access to 22, 80, 443, 9981, 12320, 12321 from anywhere.

For VPN access I integrated Zero Tier One into confconsole. (Just remembered I need to pin repo for zerotier-one so not complete. Done now.)

https://github.com/MPTMG/TKL-TVheadend/tree/code-review (will probably merge this with my master branch soon)

I understand from your comments this appliance is low priority for you, I am just addressing the concerns in the best way I know how to make it as close to production ready as possible.

Enabling a firewall using ufw is dead simple even when it needs to be done manually for each appliance. The number of ports for any given appliance is very limited. I know this is probably nothing new to you but....

Additions to plan/main ufw

Additions to conf.d/main

Step 1: Deny everything ufw default deny

Step 2: Enable Open Internet Ports ufw allow port#

Step 3: Enable Local Access Only Ports ufw allow from 10.0.0.0/8 to any port port# ufw allow from 172.16.0.0/16 to any port port# ufw allow from 192.168.0.0/16 to any port port#

Step 4: Enable Firewall ufw --force enable

[JedMeister]

Ok thanks. I'll have a look ASAP, but not sure when that'll be as I really need to focus on finishing the updates for the current library. Feel free to ping me if you don't hear anything soonish. FWIW Zoneminder is in the v15.0 release though and I'll definitely try to squeeze in Syncthing.

Re ufw, could you do me a massive favour and have a look how it interacts with the firewall in Webmin? Webmin leverages IPTables directly. AFAIK ufw does something similar, but I'm not really clear how they will interact?! There was another request to add ufw by default and the only reason I didn't was because I just haven't had a chance to look at that.

[ghost]

Will take a look and report back with IPTables interaction with ufw.

[ghost]

Simply a frontend for iptables.

The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls. ufw provides a framework for managing netfilter, as well as a command-line interface for manipulating the firewall. ufw aims to provide an easy to use interface for people unfamiliar with firewall concepts, while at the same time simplifies complicated iptables commands to help an adminstrator who knows what he or she is doing. ufw is an upstream for other distributions and graphical frontends. https://wiki.ubuntu.com/UncomplicatedFirewall

From the Man Page: https://www.systutorials.com/docs/linux/man/8-ufw-framework/ Further indication UFW is just a frontend for iptables and built to co-exist with any other iptables rules generated elsewhere.

As mentioned, ufw loads its rules files into the kernel by using the iptables-restore and ip6tables-restore commands. Users wanting to add rules to the ufw rules files manually must be familiar with these as well as the iptables and ip6tables commands.

ufw uses many user-defined chains in addition to the built-in iptables chains. If MANAGE_BUILTINS in /etc/default/ufw is set to 'yes', on stop and reload the built-in chains are flushed. If it is set to 'no', on stop and reload the ufw secondary chains are removed and the ufw primary chains are flushed. In addition to flushing the ufw specific chains, it keeps the primary chains in the same order with respect to any other user-defined chains that may have been added. This allows for ufw to interoperate with other software that may manage their own firewall rules.

[JedMeister]

Oh ok. I recall reading that because they store their rules in different places, after a reboot, only the rules for the last one used would be applied (and not the other...). But perhaps I was incorrect?!

[ghost]

Only mention of potential negative effect on rules added outside the ufw framework is

If the default policy is set to REJECT, ufw may interfere with rules added outside of the ufw framework. See README for details.

Seems like an edge case that can be avoided. from: https://helpmanual.io/man8/ufw/

After manually testing using webmin there is an issue. If you set the firewall configured in webmin to start at boot it disables ufw. Rules can be added dynamically via iptables after ufw is started, but if the iptables configuration is loaded at startup ufw gets disabled automatically.

[ghost]

FWIW I can accomplish this without UFW by adding the firewall rules in conf.d/main and enabling the firewall in the same manner webmin does through /etc/network/interfaces. Hence out of box compatible with existing webmin configured firewall you can lock down ports to just private networks.

Relevant Code in conf.d/main

Using Just iptables

sed -i '/state --state/a -A INPUT -p tcp --dport 9982 -s 10.0.0.0\/8 -j ACCEPT' /etc/iptables.up.rules sed -i '/state --state/a -A INPUT -p tcp --dport 9982 -s 172.16.0.0\/16 -j ACCEPT' /etc/iptables.up.rules sed -i '/state --state/a -A INPUT -p tcp --dport 9982 -s 192.168.0.0\/16 -j ACCEPT' /etc/iptables.up.rules

sed -i '/iface eth0/a \tpost-up iptables-restore \< \/etc\/iptables.up.rules' /etc/network/interfaces Screenshots of configuration OOB.

iptables -L:

Webmin firewall config:

FYI post screenshots I removed 9982 from Makefile so it only allows from private networks now.

Current Code: https://github.com/MPTMG/TKL-TVheadend