OpenSSL

[peregilk]

Could you build this with -enable-openssl

[sdvillal]

We briefly discussed it in https://github.com/conda-forge/ffmpeg-feedstock/pull/45 (which points to https://github.com/conda-forge/ffmpeg-feedstock/pull/43#issuecomment-362471495).

Why do you need it?

[peregilk]

Thanks for the useful links to the discussion.

I am using python and opencv with the ffmpeg backend for doing machine learning on live videos. Lots of video-sources are today only available over https.

Currently I am getting too many errors with the gnutls-options. Typically I am getting "Error in the pull function" a couple of times each hour, and then an error saying "A TLS package with unexpected length is received". The first ones resulting in a few skipped frames, and the last one in a crash that is hard to recover from without loosing several seconds of video. Ive been googling the errors and this seems to be ancient stuff. No idea how to fix this by using gnutls.

Using the openssl backend to ffmpeg seem to be the stable way of doing this, and Ive been running this as a standalone without any issues.

On Mon, 3 Dec 2018 at 21:46, Santi Villalba notifications@github.com wrote:

We briefly discussed it in #45 https://github.com/conda-forge/ffmpeg-feedstock/pull/45 (which points to #43 (comment) https://github.com/conda-forge/ffmpeg-feedstock/pull/43#issuecomment-362471495 ).

Why do you need it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/ffmpeg-feedstock/issues/68#issuecomment-443863862, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqMABDPL1TGMsqn4GRTn36ujF71w6Oqks5u1Y2VgaJpZM4Y_VDw .

[sdvillal]

I personally think that, for the default version of ffmpeg, we should not compile against openssl until openssl version 3 gets released with the new apache license. ffmpeg is already a patent minefield as it is, adding yet more mines and changing the license on the way is, in my opinion, not a good idea. I think this is shared, at least in my linux distro, the official ffmpeg package does not compile against openssl.

Having said that, compiling against openssl used to be trivial: just add openssl to the dependencies, add the compilation flag and change the package license accordingly. So for the time being I would suggest for you to build your openssl-enabled version and upload it to your own channel. I can try to give a hand if you decide to go that way and find any problem.

[peregilk]

Thanks. I agree with you regarding the licensing issues here.

I am using opencv with the ffmpeg backend. Do you know if it would be sufficient just to rebuild and install the ffmpeg package here, or would I also have to take additional steps to build the new ffmpeg into opencv?

On Tue, 4 Dec 2018 at 06:12, Santi Villalba notifications@github.com wrote:

I personally think that, for the default version of ffmpeg, we should not compile against openssl until openssl version 3 gets released with the new apache license https://www.openssl.org/blog/blog/2018/11/28/version/. ffmpeg is already a patent minefield as it is, adding yet more mines and changing the license on the way is, in my opinion, not a good idea. I think this is shared, at least in my linux distro, the official ffmpeg package does not compile against openssl.

Having said that, compiling against openssl used to be trivial: just add openssl to the dependencies, add the compilation flag and change the package license accordingly. So for the time being I would suggest for you to build your openssl-enabled version and upload it to your own channel. I can try to give a hand if you decide to go that way and find any problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/ffmpeg-feedstock/issues/68#issuecomment-443974181, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqMACwPeTE_hHYWc9U3f3qn1t53fHjpks5u1gRMgaJpZM4Y_VDw .

[sdvillal]

It should be enough to rebuild the ffmpeg package. We just need to make sure we compile the right version - one that is compatible with the version used to build conda-forge opencv. Latest opencv builds pin ffmpeg like ffmpeg >=4.0.2,<4.1. That unfortunately is not the latest version we ship of ffmpeg, which is 4.1 - conda-forge runs a little wild with a kind of rolling distro spirit were individual packages get upgraded as their maintainers see fit without necessarily rebuilding downstream packages. But this is just a small inconvenience with two easy workarounds: open a PR to opencv simply bumping its build number, so it gets the latest pinning, or simply checkout the conda-forge recipe for the old ffmpeg version (see below).

You might even decide to run your build against public CIs, but I feel that might be overkill. A quick linux recipe for building ffmpeg locally :

# prepare your tools:

#  1) install docker (e.g. in arch linux: https://wiki.archlinux.org/index.php/docker#Installation)

#  2) in your conda base environment, install conda-smithy and conda-forge-pinning
#      you might want to actually have a separated miniconda install just for that
conda install conda-smithy conda-forge-pinning

# 3) clone conda-forge ffmpeg recipe (perhaps you could use your own fork instead)
git clone git@github.com:conda-forge/ffmpeg-feedstock.git
cd ffmpeg-feedstock

# 4) latest conda-forge opencv needs ffmpeg >=4.0.2,<4.1
#     a quick look at ffmpeg recipe history tells us to checkout this commit
git checkout 06ff602d4942 -b with-openssl-4.0

# 5) make the needed changes to the recipe. In this case:
#        - in build.sh:
#            * change "disable-openssl" to "enable-openssl"
#            * remove "--enable-gnutls"
#            * add "--enable-nonfree"
#        - in meta.yaml:
#            * add "- openssl  # [not win]" to the host and run sections
#            * remove all references to gnutls
#            * change LICENSE, as we are not anymore GPL (optional but advisable)
#              for example, "custom: nonfree and unredistributable "
#              See: https://www.ffmpeg.org/legal.html
#           * bump build number; 
#             since this is for "internal use", I would put it to something like 2000

# 6) rerender the recipe
conda-smithy rerender

# 7) build (run and go have a coffee, it that is your thing)
#     note that this could change with future versions of smithy 
#    (it evolves as our usage of the CIs change)
CONFIG="linux_" .circleci/run_docker_build.sh

# 8) if all went well, your package is in "build_artifacts/linux-64/ffmpeg*"
# now you have a few options and will likely need to play with 
# channels and channel priorities:
#   - install manually from that file
#   - put it in a local repository, install 
#   - upload to your channel in anaconda.org.
#     I think the "unredistibutable" might not apply to you depending on where do you live.
#     But with #legalities I cannot really help much.
conda install anaconda-client
anaconda upload -u peregilk build_artifacts/linux-64/ffmpeg*

Note that right now conda-forge is still in between a rather large change of the compiler infrastructure. I do not know how this will play, but if there is any problem we can ask the conda-forge team (or simply wait until migration is done, a horizon that seems relatively close now).

HTH

[peregilk]

Thanks a lot for the thorough instructions. I will try this.

Per

On Tue, 4 Dec 2018 at 11:22, Santi Villalba notifications@github.com wrote:

It should be enough to rebuild the ffmpeg package. We just need to make sure we compile the right version - one that is compatible with the version used to build conda-forge opencv. Latest opencv builds pin ffmpeg like ffmpeg

=4.0.2,<4.1. That unfortunately is not the latest version we ship of ffmpeg, which is 4.1 - conda-forge runs a little wild with a kind of rolling distro spirit were individual packages get upgraded as their maintainers see fit without necessarily rebuilding downstream packages. But this is just a small inconvenience with two easy workarounds: open a PR to opencv simply bumping its build number, so it gets the latest pinning, or simply checkout the conda-forge recipe for the old ffmpeg version (see below).

You might even decide to run your build against public CIs, but I feel that might be overkill. A quick linux recipe for building ffmpeg locally :

prepare your tools:

1) install docker (e.g. in arch linux: https://wiki.archlinux.org/index.php/docker#Installation)

2) in your conda base environment, install conda-smithy and conda-forge-pinning# you might want to actually have a separated miniconda install just for that

conda install conda-smithy conda-forge-pinning

3) clone conda-forge ffmpeg recipe (perhaps you could use your own fork instead)

git clone git@github.com:conda-forge/ffmpeg-feedstock.gitcd ffmpeg-feedstock

4) latest conda-forge opencv needs ffmpeg >=4.0.2,<4.1# a quick look at ffmpeg recipe history tells us to checkout this commit

git checkout 06ff602d4942 -b with-openssl-4.0

5) make the needed changes to the recipe. In this case:# - in build.sh:# change "disable-openssl" to "enable-openssl"# remove "--enable-gnutls"# remove "--enable-gpl"# add --enable-nonfree# # - in meta.yaml:# add "- openssl # [not win]" to the host and run sections# remove all references to gnutls# change LICENSE, as we are not anymore GPL (optional but advisable)# for example, "custom: nonfree and unredistributable "# See: https://www.ffmpeg.org/legal.html# bump build number; since this is for "internal use", I would put it to something like 2000

6) rerender the recipe

conda-smithy rerender

7) build (run and go have a coffee, it that is your thing)# note that this could change with future versions of smithy / evolves as our usage of the CIs change

CONFIG="linux_" .circleci/run_docker_build.sh

8) if all went well, your package is in "build_artifacts/linux-64/ffmpeg*"# now you have a few options and will likely need to play with channels and channel priorities:# - install manually from that file# - put it in a local repository, install # - upload to your channel in anaconda.org.# I think the "unredistibutable" might not apply to you depending on where do you live.# But with #legalities I cannot really help much.

conda install anaconda client anaconda upload -u peregilk build_artifacts/linux-64/ffmpeg*

Note that right now conda-forge is still in between a rather large change of the compiler infrastructure. I do not know how this will play, but if there is any problem we can ask the conda-forge team (or simply wait until migration is done, a horizon that seems relatively close now https://conda-forge.org/status/).

HTH

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/ffmpeg-feedstock/issues/68#issuecomment-444049074, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqMADVoofxKBLEMRlPzmrAd0JrwUkNYks5u1kzegaJpZM4Y_VDw .

[peregilk]

Hi, Thanks again for the instructions. Everything runs fine until I am trying to build the docker.

It exits with the following error

conda.exceptions.CondaHTTPError: HTTP 404 NOT FOUND for url < https://ffmpeg.org/releases/ffmpeg-4.0.2000.tar.gz>

Why is it trying to download this file?

Per

On Tue, 4 Dec 2018 at 11:22, Santi Villalba notifications@github.com wrote:

It should be enough to rebuild the ffmpeg package. We just need to make sure we compile the right version - one that is compatible with the version used to build conda-forge opencv. Latest opencv builds pin ffmpeg like ffmpeg

=4.0.2,<4.1. That unfortunately is not the latest version we ship of ffmpeg, which is 4.1 - conda-forge runs a little wild with a kind of rolling distro spirit were individual packages get upgraded as their maintainers see fit without necessarily rebuilding downstream packages. But this is just a small inconvenience with two easy workarounds: open a PR to opencv simply bumping its build number, so it gets the latest pinning, or simply checkout the conda-forge recipe for the old ffmpeg version (see below).

You might even decide to run your build against public CIs, but I feel that might be overkill. A quick linux recipe for building ffmpeg locally :

prepare your tools:

1) install docker (e.g. in arch linux: https://wiki.archlinux.org/index.php/docker#Installation)

2) in your conda base environment, install conda-smithy and conda-forge-pinning# you might want to actually have a separated miniconda install just for that

conda install conda-smithy conda-forge-pinning

3) clone conda-forge ffmpeg recipe (perhaps you could use your own fork instead)

git clone git@github.com:conda-forge/ffmpeg-feedstock.gitcd ffmpeg-feedstock

4) latest conda-forge opencv needs ffmpeg >=4.0.2,<4.1# a quick look at ffmpeg recipe history tells us to checkout this commit

git checkout 06ff602d4942 -b with-openssl-4.0

5) make the needed changes to the recipe. In this case:# - in build.sh:# change "disable-openssl" to "enable-openssl"# remove "--enable-gnutls"# remove "--enable-gpl"# add --enable-nonfree# # - in meta.yaml:# add "- openssl # [not win]" to the host and run sections# remove all references to gnutls# change LICENSE, as we are not anymore GPL (optional but advisable)# for example, "custom: nonfree and unredistributable "# See: https://www.ffmpeg.org/legal.html# bump build number; since this is for "internal use", I would put it to something like 2000

6) rerender the recipe

conda-smithy rerender

7) build (run and go have a coffee, it that is your thing)# note that this could change with future versions of smithy / evolves as our usage of the CIs change

CONFIG="linux_" .circleci/run_docker_build.sh

8) if all went well, your package is in "build_artifacts/linux-64/ffmpeg*"# now you have a few options and will likely need to play with channels and channel priorities:# - install manually from that file# - put it in a local repository, install # - upload to your channel in anaconda.org.# I think the "unredistibutable" might not apply to you depending on where do you live.# But with #legalities I cannot really help much.

conda install anaconda client anaconda upload -u peregilk build_artifacts/linux-64/ffmpeg*

Note that right now conda-forge is still in between a rather large change of the compiler infrastructure. I do not know how this will play, but if there is any problem we can ask the conda-forge team (or simply wait until migration is done, a horizon that seems relatively close now https://conda-forge.org/status/).

HTH

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/ffmpeg-feedstock/issues/68#issuecomment-444049074, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqMADVoofxKBLEMRlPzmrAd0JrwUkNYks5u1kzegaJpZM4Y_VDw .

[sdvillal]

I think the problem might be on how you have bumped the build number. You do not need to touch the version in meta.yaml. It should look something like this:

{% set version = "4.0.2" %}

...

build:
  number: 2000

[peregilk]

Thanks a lot for the help! I just misread your instructions and changed the version number instead of the build number. After that it worked.

My server is now running ffmpeg with openssl, and has now been running stable for six hours. Hopefully this fixes my problem.

Your instructions are perfect to follow if someone else would like to do the same thing. The only change I needed to do was do both have "--enable-gpl" and "--enable-nonfree" in build.sh. There was another library that required the gpl-license.

On Fri, 7 Dec 2018 at 09:30, Santi Villalba notifications@github.com wrote:

I think the problem might be on how you have bumped the build number. You do not need to touch the version in meta.yaml. It should look something like this:

{% set version = "4.0.2" %}

... build: number: 2000

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/ffmpeg-feedstock/issues/68#issuecomment-445159472, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqMALRlJ-l10d3F23jcGkqxRfjCxvPUks5u2ichgaJpZM4Y_VDw .

[sdvillal]

Cool, I have updated the instructions with your fixes and I will close the issue.

Just a last thought, it could be great if conda would allow "source packages" (ala gentoo, arch AUR and the like) that get built upon installation. It could make for a nice solution for this kind of situations.

[jakirkham]

Given OpenSSL 3 has since shipped and conda-forge has migrated to it, maybe we can revisit this?

Reopening so we can discuss