Including CasparCG server in Debian?

[petterreinholdtsen]

Hi

Are you interested in having CasparCG included as an official package in Debian and derivatives like Ubuntu? I've created a request for a CasparCG package in Debian (aka a WNPP request), and thought it best to check with you if this is something you welcome and would like to help make happen.

I am a Debian developer and can assist. It is a lot easier to do when the Debian maintainer and the upstream developers cooperate, so I find it best to ask up front if you are interested.

[petterreinholdtsen]

Anyone had time to look at this yet?

[petterreinholdtsen]

I tried to build the current code on Debian Buster/testing, and while most dependencies seem to be available, at least one piece is missing: Chromium Embedded Framework. I've requested this too into Debian just now.

[petterreinholdtsen]

The draft build rules for Debian is now available from https://github.com/Frikanalen/casparcg-server/tree/debian/debian . The code disable the HTML module because CEF is missing, and adjust cmake rules to accept boost versions after 1.66 too.

[walterav1984]

Thanks for requesting a official CEF package for Debian.

Another Debian repo package which also rely(additional features) on CEF but doesn't include it is called Nageru a live videomixer. The author and if I'm not mistaking maintainer of the Debian package @sesse has included notes to build the nageru with CEF in its readme.

My guess for not directly including/shipping is the License of CEF (new BSD) vs Nageru (GPLv3) / CasparCG(GPLv3). However I see that Debian allows (modified BSD license) in main repo but I'm not sure if "modified" is the same as "new".

Would it be a option/possibility to submit a official CEF build (independent) Debian package that both CasperCG and Nageru can link with so they don't have to ship with conflicting licenses themselves?

Or is a CEF framework build so application specific that its impossible to ship a common build/module that more applications can use?

Is this somewhat similar in the License/module issue/solution like BMD Decklink / AJA -drivers perhaps Newtek NDI option?

[Julusian]

Thanks for doing this, I know there is some interest within the wider community for this, but I am not sure if there is from SVT currently. One thing to note is that linux support is still not a supported way of running, but I believe that is intended to change in 2.3 or 3.0

One concern I have with this is that development resources are very limited and so progress is slow and supporting multiple distros/versions will potentially add more work.

As for ffmpeg and CEF dependencies, the only potential problem I can think of with using system versions is that it limits the version that can be used. For something like CEF, I expect we will want to consider updating the version used again in the next version, and so using a system version will block that from happening unless the system will offer multiple versions. FFmpeg is less of an issue, but might face something similar if there is an important change in support of a common broadcast codec.

The current method of dependencies while not perfect works well as we can control exactly what versions we use, and ensure they are consistent across all supported platforms.

[sesse]

Hi,

I'm not sure why you pull in licensing here. CEF is under a free license that is acceptable for inclusion in Debian and also compatible with GPLv3.

The primary reason why CEF is not in the Debian archive is security support. Chromium is a big and complicated package to support, and it's out of the question to have an extra bundled copy of it within a CEF package. CEF upstream only supports the latter (and really only supports building by pulling live git snapshots out of Bitbucket—the source tarballs available for download are explicitly not supported), but I've managed to package everything up so that one can build CEF against Debian's version of Chromium. I have some packages at https://storage.sesse.net/cef/ that demonstrate this, but they're without security support.

However, this requires the Chromium source code to be available in Debian package form, and it requires that CEF be rebuilt and/or upgraded whenever Chromium gets a security update. So far, nobody has been able to give me a definite yes or no on whether this would be acceptable—it comes down to that the Chromium maintainer is already overworked, and that this kind of rippling security updates is not currently well supported in the archive.

See Debian bug #893448 for more discussion.

[petterreinholdtsen]

I take it from the comments here (as well as the handling of the Debian related pull requests #1116, #1117, #1120 and #1121 as well as issues #1118 #1119) that there is interest to have casparcg in Debian.

I'll see if there is interest in the Debian multimedia team to maintain it within that framework.

[petterreinholdtsen]

Btw, as the Debian release freeze is quickly approaching, it would be useful to know when you plan to move 2.2 out of beta and make a new release. Do you have a timeline for it? Debian freezes in a few weeks, most likely early February 2019. I would like to use a source version with clear copyright information, to reduce the chance of having the package rejected by the ftpmasters, and version 2.0.7 (the last non-beta release tagged in git) is filled with embedded copies of libraries that are going to raise questions from the Debian ftpmasters.

[Julusian]

Thanks for that explanation @sesse. I assume that means we are also unable to include libcef.so as part of our package?

I take it from the comments here (as well as the handling of the Debian related pull requests #1116, #1117, #1120 and #1121 as well as issues #1118 #1119) that there is interest to have casparcg in Debian. I believe the general policy here at the moment is that bug fixes will be happily merged in, so I would say the handling of the other issues is very conclusive.

@dotarmin Is there an eta for the release of 2.2? You can rule out trying to use 2.0.7, as it does not have any support for linux.

[sesse]

Thanks for that explanation @sesse. I assume that means we are also unable to include libcef.so as part of our package?

In what form? Compiling all of CEF yourself as part of the CasparCG build? Using the binary builds provided by CEF? How would you provide security support for such a package when Chromium needs upgrades? (The latter would certainly be unworkable for policy, license and portability reasons.) Or are you thinking of something else entirely?

[Julusian]

In what form? Compiling all of CEF yourself as part of the CasparCG build? Using the binary builds provided by CEF? How would you provide security support for such a package when Chromium needs upgrades? (The latter would certainly be unworkable for policy, license and portability reasons.) Or are you thinking of something else entirely?

Either way really. Compiling CEF during the build would be a pretty horrible task, but if that is what has to be done, then it would be worth considering. I would prefer to use the binaries, but I can understand that doesn't conform to debian policy. I don't expect we would provide provide security updates, which I do also expect to be a problem with debian. I certainly wouldn't want the major version of chromium to increase unexpectedly. I have had websites break in small ways from chrome updates, so the potential for that happening to caspar graphics would encourage me to never update debian

[sesse]

So there's two sides to the story here: Making a working Debian package, and having it in Debian proper. I assume pere wants the latter.

Compiling CEF yourself and putting it into the package (or in a separate package in the same repository) would be possible for the former. You would get a package that nobody cares about security-wise, but perhaps for most Caspar use case, it doesn't matter as you're likely to run trusted source code and never access the network. Shipping a Caspar package with a private libcef.so is unlikely to be accepted for Debian, but again, maybe it's possible to work with the Chromium maintainer and the security team to find a more official solution that also Nageru and OBS could depend on.

Using the prebuilt binaries would not confirm to GPL (you need to include the source and build scripts for everything you link against), so that's not an option even if you don't want to stay in Debian proper.

[petterreinholdtsen]

Steinar is right, I am interested in uploading the package to Debian for inclusion in the official package repository there.

And for the record, my plan is to upload as soon as a new release is tagged including the updated copyright statements currently in HEAD.

[Julusian]

Thanks for the explanations @sesse. This is something to discuss upstream with Debian then, but its not a blocker so that isn't urgent.

@petterreinholdtsen Could you add some instruction to BUILDING.md on how to build this deb locally? That would be useful to ensure it doesn't get broken in future (and means I can try it out)

[sesse]

@petterreinholdtsen: So you intend to upload a version without support for CEF (nor presumably Flash)? I'm not sure how useful that would be.

[petterreinholdtsen]

[Steinar H. Gunderson]

@petterreinholdtsen: So you intend to upload a version without support for CEF (nor presumably Flash)? I'm not sure how useful that would be.

Yes. It will solve the immediate need for the Frikanalen TV channel, which is the reason I look at CasparCG in the first place. Hopefully CEF will make it into Debian before the channel need to cross that bridge. :)

-- Happy hacking Petter Reinholdtsen

[petterreinholdtsen]

[Julian Waller]

@petterreinholdtsen Could you add some instruction to BUILDING.md on how to build this deb locally? That would be useful to ensure it doesn't get broken in future (and means I can try it out)

I'll see what I can do. In short, you need a copy of a debian/ directory with build rules and packaging files, and to run 'debuild' (or 'dpkg-buildpackage') when the directory is in place.

-- Happy hacking Petter Reinholdtsen

[petterreinholdtsen]

Are there any plans to make a new beta release before Christmas?

[dotarmin]

Are there any plans to make a new beta release before Christmas?

Main goal is to release a stable before Christmas. Otherwise a RC will be released before Christmas.

[k0d]

@petterreinholdtsen Awesome plan, I've worked about with debian packaging before so I know the technical side of it. I started to prepare a bit to get CasparCG to be more package-able. If/when I can get the MacOS version sorted, then I wish to have it inside of 'homebrew'!

Personally I'm only really interested in getting it packaged etc, bringing it into debian proper is an obvious goal, but dealing with red-tape/licences etc...I'm very glad if someone else can ensure we're doing the right thing there!

I'm going to read through this topic a bit more and will check out your commits to get a better idea where we are at the moment with it.

Do you have some tips on how we can make development easier? I'm using Ubuntu Bionic on my laptop, and have mostly compiled against that, of course we can build in Docker, but I prefer to build direct on my laptop when doing development work.

Does the Debian project have some build system we can use, I have access to plenty of server power, even at home, but it's nice if there is some official servers we can get the builds made on.

Are you on the CasparCG Slack channel? It could be nice to chat whilst fixing some thing.

[k0d]

Do you feel CMake is good still for the build scripts, is there one that integrates nicer with debian packaging, I didn't really look into that side of things yet? I'd like to enhance the build scripts anyway, Of course a lot of people want HTML templates, but it could be nice to be able to easily build without some features, depending on licences, it'd be nice even if we can have the modules in separate packages (as long as it doesn't affect performance).

[k0d]

Another thought: I'd prefer to have the build scripts outside of this repository, what do you think? I think it's cleaner to have just the source for CasparCG inside one repo etc...

[petterreinholdtsen]

[Mark Olsson]

I'm going to read through this topic a bit more and will check out your commits to get a better idea where we are at the moment with it.

At the moment I am ready to upload to Debian for get the package reviewed by the ftp masters as soon as a new beta or stable release is done.

I doubt it will make it into the next stable version, as the freeze day is approaching and I suspect there is not enough time to get the review in place and any problems fixed before the Debian stable freeze is implemented, but I hope I am wrong.

Do you have some tips on how we can make development easier? I'm using Ubuntu Bionic on my laptop, and have mostly compiled against that, of course we can build in Docker, but I prefer to build direct on my laptop when doing development work.

Not quite sure what you mean. I build directly on a machine with Debian testing. Development seem easy enough.

Does the Debian project have some build system we can use, I have access to plenty of server power, even at home, but it's nice if there is some official servers we can get the builds made on.

Once accepted into Debian, the automatic build system in Debian will build on all architectures currently handled by Debian.

If you want to learn more about Debian packaging, I recommend you start with <URL: https://www.debian.org/doc/manuals/developers-reference/index.en.html >

Are you on the CasparCG Slack channel? It could be nice to chat whilst fixing some thing.

I do not like the terms of use for Slack, so I do not use it. I am using IRC, XMPP and Signal, if you want to have a chat. I guess it is fairly easy to find me on IRC, I am 'pere' on for example #frikanalen (irc.freenode.net).

Do you feel CMake is good still for the build scripts, is there one that integrates nicer with debian packaging, I didn't really look into that side of things yet? I'd like to enhance the build scripts anyway, Of course a lot of people want HTML templates, but it could be nice to be able to easily build without some features, depending on licences, it'd be nice even if we can have the modules in separate packages (as long as it doesn't affect performance).

Another thought: I'd prefer to have the build scripts outside of this repository, what do you think? I think it's cleaner to have just the source for CasparCG inside one repo etc...

I have no problem with CMake, and would strongly recommend to keep the build scripts alongside the source, not in a separate repository.

-- Happy hacking Petter Reinholdtsen

[k0d]

Ok, what I meant with the development, was that I use Ubuntu Bionic on the laptop, mostly due to driver issues. So it's hard to compile for debian directly on this machine for the obvious deps. reasons. Not so important though, maybe I can switch to debian stable.

I'll try to find you on IRC later....

I might have been a bit unclear, it's not the build scripts (cmake) that I think should be in another repo, but the packaging scripts. Homebrew/apt/rpm/windoze....

[petterreinholdtsen]

[Mark Olsson]

Ok, what I meant with the development, was that I use Ubuntu Bionic on the laptop, mostly due to driver issues. So it's hard to compile for debian directly on this machine for the obvious deps. reasons. Not so important though, maybe I can switch to debian stable.

Ah, right. For this, you can use a debian chroot. I use a chroot myself to do test builds on Debian testing. To create it, run for example these commands:

sudo apt install debootstrap sudo debootstrap buster /some/path

There is also pbuilder and other frameworks to build and maintain such build chroot.

I'll try to find you on IRC later....

Very good. :)

I might have been a bit unclear, it's not the build scripts (cmake) that I think should be in another repo, but the packaging scripts. Homebrew/apt/rpm/windoze....

Then I definitely misunderstood. At least for the official Debian packaging, I would prefer to keep the debian/ directory with the packaging rules out of the official upstream repository. If both upstream and debian have a debian/ directory, there will be merge conflicts when updating to a new version in Debian.

-- Happy hacking Petter Reinholdtsen

[k0d]

I know a little about chroot...have used it to build some firmware's for routers etc. But I didn't know so much about using it against different releases. Thanks for that info!

@dotarmin @5opr4ni What's your opinion on having a new repository to just have the packaging scripts in? Probably the build server scripts (Jenkins) could live there also.

[Julusian]

When I have needed to build across distros I use a temporary docker container. Nice and quick to get running and easy to optionally throw away straight after, but chroot also sounds like a good way.

In general I'm not keen on the packaging being outside of the repo, but it depends on who is intended to look after it. It sounds like it makes sense for debian to be done that way, but for producing a standard standalone windows/linux/mac build I think it best to keep it in the repo

[k0d]

My thoughts/reasons behind this is: If you want to develop CasparCG you will clone this repo, and follow the instructions for setting up a development environment. So you won't care about packaging If you want to package CasparCG, apart from distro maintainers, it's most likely just SVT that would want to modify those scripts for the 'official builds', or maybe some other broadcasters if they have some server management tools for example. If you just want to use CasparCG then you just download the prebuilt images/packages etc etc.

[Julusian]

I often find myself packaging a build locally so that I can send it to others for testing. I think it would be quite common that any developer who makes a custom build to want to do the same. On windows especially the build output produced by visual studio without packaging is a complete mess with some intermediate, junk and duplicate files so isnt great to have to distribute. Linux produces the correct and clean structure, so I am fine with nothing there. So as long as for the current platform/distro you are running its only a couple of simple commands or there are scripts then I am happy.

[k0d]

Basically...if you wanted to do a local package...it'd be just a case of cloning that repo, and running the script to build for your system.

We already did a lot of cleanup etc, but it's certainly possible to clean up and streamline the repo, packages even more.

My thought anyway, is that the source repo would output a very clean folder ready for distro packaging, or even to just zip up. The packaging scripts would be more about adding installation scripts, dependency stuff and making sure things are put in the right destination folders according to the distro/OS.

[dotarmin]

@k0d: Another thought: I'd prefer to have the build scripts outside of this repository, what do you think? I think it's cleaner to have just the source for CasparCG inside one repo etc...

I totally disagree on this one actually (even if I also thought about same thing as you). The build-scripts should be bundled with the repository it-self. The repository should be self contained and the server should be buildable without any extra forking etc.

The CasparCG project has a lot of various users and all are not experts. Even if this is natural having various levels of knowledge we should still aim to make it as simple as possible for the community members and users of CasparCG projects to build the server, even if we provide auto builds.

:)

[k0d]

@dotarmin I didn't mean the build scripts....but the packaging and jenkins (build-server) scripts.

I've thought about this a lot, in some ways I still feel having them in one repo is great, but I often find that it's confusing when cloning a new project and seeing so many folders for packaging and different people's editor configs etc. Look at this repo for example https://github.com/nodejs/node, where would one even start to develop on that project :) And that's a project that I feel is actually quite clean compared to others!

If we have them in one repo, then I suggest a folder structure similar to this... ./src (the actual c++/c code) ./tools (scripts to do with building, packaging etc) ./Makefile (this is just a suggestion, but if we can have it so people can just "git clone ... && cd ...&& make" it's so nice and simple) ./resources (files used by the build/packaging scripts that are 'data' not 'scripts', such as the debian control/rules files, app logos...)

That way, a developer can easily clone, build and use the server.

[Julusian]

The problem with trying to tidy it further is the number of files that expect to be at the root of the repo. But what is the problem with having packaging scripts hidden away under tools?

I think we should aim to have build-scripts that produce a folder ready for zipping as part of this repo. Perhaps the scripts could even default to zipping it up, but with a param to skip doing that. From what I can tell, the default packaging step will be solely to zip it up, with mostly only build servers wanting to do something else first.

And I think it is reasonable to assume you are running it on the os/distro/version you want to build for, so the building in docker could be replaced with some simpler script.

[k0d]

I don't want to sound grumpy, but I have mentioned a few times now, build-scripts should be in this repo, packaging scripts (could) be in a separate one.

[Julusian]

Sorry, my last comment was meant to be more along the lines of I agree build-scripts should be here. But if done correctly, then default packaging is a single extra command, so why not optionally do that too, so I don't know what the other repo would contain

[k0d]

It's purely to keep the main source repo from being polluted with code that's specific to certain distros.

My way of doing things is to hide platform specific code if possible. I hate to see code with so many #ifdef's etc in them. It's just so confusing. Of course, I'm not saying we change the source code, just explaining my way of thinking.

I ❤️Clean Code

[petterreinholdtsen]

I decided to reserve a slot in the Debian NEW queue for casparcg-server, <URL: https://ftp-master.debian.org/new/casparcg-server_2.2.0~beta7%2Bdfsg-1.html >.

[k0d]

How well does it work? can it find the config file for example? what do we need to do to get it ”finished”.

Shall we create a new milestone and create issues under that for this ’debianise casparcg’ project?

[petterreinholdtsen]

[Mark Olsson]

How well does it work? can it find the config file for example? what do we need to do to get it ”finished”.

I do not know how well it work, and know it will look for casparcg.config in the current working directory, not where it is located in /usr/share/. A new release with updated copyright headers in the source would help to get it finished. Gettin CEF into Debian would make the package a lot more useful

Shall we create a new milestone and create issues under that for this ’debianise casparcg’ project?

Not sure if it make sense. The remaining tasks besides making a new upstream release is internal to Debian. It would be great if you could ensure all the source have updated copyright headers, as it make it easier to automatically check the copyright status of the code. I assume the ftpmasters find some errors in the packaging and once we know what they find we need to fix those problems before doing a new upload.

As for Debian build rules included in the upstream casparcg git repo, one useful approach that do not make problems for the Debian packageing is to place the upstream build rules and script somewhere not debian/, and symlink debian -> somewhere during. It could for example be placed in contrib/debian/ and a script to symlink and run debuild can be placed alongside it.

-- Happy hacking Petter Reinholdtsen

[petterreinholdtsen]

Main goal is to release a stable before Christmas. Otherwise a RC will be released before Christmas.

Very good to know. It would be useful for my own planning to know at what point in time the 'Christmas' you refer to occur. Can you provide some details on date and year? :)

[dotarmin]

We found a bug yesterday that Julian has fixed in branch 2.2.x (thanks Julian) but it also needs to be fixed in master branch. Estimated date is somewhere between 27-29 December.

Merry Christmas 🎅

[Julusian]

That fix not being in master doesn't stop 2.2 being released, as master is 2.3/3.0, but the fix is merged to master now too

[dotarmin]

That fix not being in master doesn't stop 2.2 being released

Well, I use master in production as is now and that’s a fix needed related to mxf files in my case 😊 But it’s fixed now in master thanks to Julian 😍

[petterreinholdtsen]

I uploaded version 2.2.0-stable to Debian a few minutes ago. The package has not yet passed NEW processing, so it is not available in Debian yet. No idea if the ftpmasters will find time to look at it before Debian freeze the next stable release, nor if it will propagate into the next stable release in time for the freeze. It seem quite unlikely at the moment, as there are few days left.

[petterreinholdtsen]

I am happy to report that CasparCG server just was accepted into Debian, <URL: https://tracker.debian.org/pkg/casparcg-server >.

[5opr4ni]

Super BIG smiley. Nice work.

[ghost]

Freaking awesome!

@5opr4ni: It is time for SVT to roll out Debian on your servers :)

[dimitry-ishenko]

Doesn't have to be Debian. Can do Ubuntu too :stuck_out_tongue_closed_eyes:

[sesse]

@5opr4ni: It is time for SVT to roll out Debian on your servers :)

That would seem difficult without any support for Flash nor CEF.

[dimitry-ishenko]

Oh, is cef not supported on 2.2?...