search

jonathanong / heroku-buildpack-ffmpeg-latest

A Heroku buildpack for ffmpeg that always downloads the latest static build
MIT License
360 stars 719 forks source link

Buildpack not working #71

Open yusjacobs opened 1 year ago

yusjacobs commented 1 year ago

Getting this when deploying in heroku:

Using buildpacks:

  1. https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest.git
  2. heroku/python -----> ffmpeg app detected -----> Installing ffmpeg Variable FFMPEG_DOWNLOAD_URL isn't set, using default value Downloading https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz Unable to download ffmpeg: 000
stevenmaguire commented 1 year ago

I am seeing the same thing and the ENV variable suggestion is also not working - and I think that is related to the two open PRs here.

yusjacobs commented 1 year ago

This started happening today. At the same time the scheduled tests started to fail

knagode commented 1 year ago

This is happening in our environment too.

I can download https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz without any issues. I wonder how to resolve this? Can we help? How?

agieche commented 1 year ago

Same here, started to fail today. Moving the buildpack to the top or bottom of the buildpack list did not help either. Setting FFMPEG_DOWNLOAD_URL is not helping, since it is somehow not picked up by the deployment.

agieche commented 1 year ago

I filed a ticket at heroku support. Will keep you in loop, if there is a suggestion from them.

yusjacobs commented 1 year ago

In the meantime i downloaded the binary from https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz and added it to my repository. I then deployed the app to Heroku and it is working. Just make sure you call the correct PATH to the binary in your code.

knagode commented 1 year ago

@yusjacobs

Just make sure you call the correct PATH to the binary in your code.

Did you simply put this file in public directory and then use:

FFMPEG_DOWNLOAD_URL

to download it? (In open PR-s I see that this doesn't work for some people)

yusjacobs commented 1 year ago

I created a bin folder in my repo and placed the binaries in there, both ffmpeg and ffmprobe. You can download both here: https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz

I then updated my commands like this:

command = ["./bin/ffmpeg", "-ss",.....

ekaj2 commented 1 year ago

Same issue on my end

mindtonic commented 1 year ago

👋🏻

sasharevzin commented 1 year ago

I filed a ticket at heroku support. Will keep you in loop, if there is a suggestion from them.

We need someone with an enterprise account so they can escalate it quickly and not wait for two days for their reply 😭

agieche commented 1 year ago

Looked a little deeper into the issue. There seems to be a SSL TLS issue, when the heroku machine curls the tar.xz file, resultung in a status code 000, which is interpreted as error by the buildpack. However, when moving the archive to a different location, the buildpack works flawlessly again.

I forked the original buildpack and adjusted the url to a copy of the archive on my servers. Problem solved, deployment is working again.

If anyone wants to have a look on my workaround fork in order to build their own: https://github.com/agieche/heroku-buildpack-ffmpeg-latest

You can use it for now as drop in replacement of the current non working build pack until things have been sorted out:

heroku buildpacks:add --index 1 https://github.com/agieche/heroku-buildpack-ffmpeg-latest.git

Keep in mind, that I will remove the file in a resaonable amount of time after the original url is working again due to a future fix of heroku.

stevenmaguire commented 1 year ago

Thanks @agieche. To clarify, 1) your diagnosis is that johnvansickle.com SSL cert is causing the problem and 2) your fork is only pointing to a different remote URL where an SSL issue is not present, correct?

The thing that I am still not seeing function correctly is the FFMPEG_DOWNLOAD_URL env variable. It seems that if I am correct in both of those points, we should not need a new fork of the package, merely a successful path to use FFMPEG_DOWNLOAD_URL to point to a new remote URL.

agieche commented 1 year ago

Thanks @agieche. To clarify, 1) your diagnosis is that johnvansickle.com SSL cert is causing the problem and 2) your fork is only pointing to a different remote URL where an SSL issue is not present, correct?

The thing that I am still not seeing function correctly is the FFMPEG_DOWNLOAD_URL env variable. It seems that if I am correct in both of those points, we should not need a new fork of the package, merely a successful path to use FFMPEG_DOWNLOAD_URL to point to a new remote URL.

I tried using the env variable at first, but I could not find an obvious way to access the Heroku env vars from within the shell script. Otherwise, this fork would be unnecessary.

I don't think, it is an issue at the end of https://www.johnvansickle.com/ffmpeg/ since curling it from my local machine is working with a correct 200 status code. Seems more of an issue in the local networking of heroku, but I'm not a 100% on this.

agieche commented 1 year ago

Checked my forked buildpack on two different production apps and deployments went fine.

ekaj2 commented 1 year ago

Just fixed it for my production app as well. Appreciate the quick hotfix @agieche

agieche commented 1 year ago

I added support for reading the ENV variables within my fork thanks to this PR: https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest/pull/61/files

I set the ENV variable FFMPEG_DOWNLOAD_URL and restarted dynos and the variable was picked up correctly.

stevenmaguire commented 1 year ago

I set the ENV variable FFMPEG_DOWNLOAD_URL and restarted dynos and the variable was picked up correctly.

Did you have success setting this variable in the Heroku App (heroku config ... etc) or the app.json "env" key, or both?

agieche commented 1 year ago

I set the ENV variable FFMPEG_DOWNLOAD_URL and restarted dynos and the variable was picked up correctly.

Did you have success setting this variable in the Heroku App (heroku config ... etc) or the app.json "env" key, or both?

I added the code to my fork. Then set the env variable within web interface (https://dashboard.heroku.com/apps/my-app-16815/settings) and deployed from CLI. Then, the Env variable was picked up correctly.

stevenmaguire commented 1 year ago

I added the code to my fork. Then set the env variable within web interface (https://dashboard.heroku.com/apps/my-app-16815/settings) and deployed from CLI. Then, the Env variable was picked up correctly.

Awesome. Thanks!

jharrilim commented 1 year ago

I've taken a different approach to resolving this. Instead of using the precompiled binaries from johnvansickle.com, I've moved to using the official Ubuntu apt package.

This can be done by:

  1. Adding the Heroku apt buildpack to your app
  2. Creating an Aptfile in your repo with this as its contents:
ffmpeg
  1. Removing this buildpack
mmaloon commented 1 year ago

With the Apt buildpack, when I re-deploy then try to run ffmpeg -version in the heroku shell, I receive an error with the messageerror while loading shared libraries.... The deploy also results in a much larger slug size (~250MB with no other buildpacks installed)

Another alternative buildpack is the heroku-buildpack-activestorage-preview which also installs the ffmpeg binary.

heroku buildpacks:add -i 1 https://github.com/heroku/heroku-buildpack-activestorage-preview

This buildpack also seems to fix the segfault issue on heroku-22 stack (https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest/issues/69)

jonathanong commented 1 year ago

Thanks @agieche, I added you as a collaborator. I don't use this build pack anymore. If Heroku wants to own this package, let me know!

agieche commented 1 year ago

Merged #61, so you can set a correct download url via env variable now. No definitive answer from heroku, but they are looking into it.

diegotorpoco commented 1 year ago

@agieche What value should the env variable FFMPEG_DOWNLOAD_URL in heroku client should take? many thanks!

kayvaninvemo commented 1 year ago

@agieche What value should the env variable FFMPEG_DOWNLOAD_URL in heroku client should take? many thanks!

Just add a new env variable in heroku->setting FFMPEG_DOWNLOAD_URL : https://cookycookstaging.s3.amazonaws.com/ffmpeg-git-amd64-static.tar.xz

image

and redeploy. Before redeploying I also restarted the dynos, but I guess that it is not necessary.

This solution is not safe. Do not use it for enterprises and banks.

sasharevzin commented 1 year ago

@kayvaninvemo is it safe to use this bucket? it looks very suspicious to me

agieche commented 1 year ago

@kayvaninvemo is it safe to use this bucket? it looks very suspicious to me

Its a personal bucket by me. Feel free to upload the archive on your own space and point the env variable to your personal url. This makes it more robust. For the time being, you may use my copy, since Im planning to keep it up until things have been sorted out by heroku support.

diegotorpoco commented 1 year ago

@agieche What value should the env variable FFMPEG_DOWNLOAD_URL in heroku client should take? many thanks!

Just add a new env variable in heroku->setting FFMPEG_DOWNLOAD_URL : cookycookstaging.s3.amazonaws.com/ffmpeg-git-amd64-static.tar.xz

image

and redeploy. Before redeploying I also restarted the dynos, but I guess that it is not necessary.

Hey, thanks! In the mean time I tried to use @dzuelke recommendation of using heroku active storage buildpack (https://github.com/jonathanong/heroku-buildpack-ffmpeg-latest/pull/72)

I have tested in production and it works.

https://github.com/heroku/heroku-buildpack-activestorage-preview.git on heroku buildpacks section

kayvaninvemo commented 1 year ago

@kayvaninvemo is it safe to use this bucket? it looks very suspicious to me

It is not safe, do not use it if it is for an enterprise or a bank. :D

palinair commented 1 year ago

Hi! We have had the same problem in our project. Thanks to those who are sharing their valuable solutions!

BTW, I did some tests to download the file via curl (using options similar to this buildpack) and on a few times it failed for 2 different reasons:

$ curl "https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz" -L --fail --retry 5 --retry-max-time 15 -o ./ffmpeg.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) Unknown SSL protocol error in connection to johnvansickle.com:443

and

$ curl "https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz" -L --fail --retry 5 --retry-max-time 15 -o ./ffmpeg.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 69 39.4M   69 27.2M    0     0   520k      0  0:01:17  0:00:53  0:00:24  593k
curl: (18) transfer closed with 12809142 bytes remaining to read
cjreeve commented 1 year ago

Any update on this? Is this for Heroku to fix?

cjreeve commented 1 year ago

As a quick work around for hosting the ffmpeg-git-amd64-static.tar.xz file, I copied the file into my development public folder, started ngrok and set the Heroku config var FFMPEG_DOWNLOAD_URL to the secure ngrok link to the file.

agieche commented 1 year ago

Got a response by heroku. Seems to be something, that the remote site could fix.

It seems that the issue you're encountering is related to a connection error when trying to retrieve the file from the specified URL. The error message "OpenSSL SSL_connect: Connection reset by peer" indicates a connection reset from the remote server (johnvansickle.com) during the SSL handshake.

This type of error is typically caused by issues on the remote server's end, rather than something you can fix from your side. It could be due to temporary server problems, network issues, or misconfigurations on their part.

To troubleshoot, you might want to:

Double-check the URL to ensure it's correct. Confirm that the remote server is up and running. Attempt the download from a different network or location to see if the issue persists. If the problem persists, reaching out to the website's administrator or support might be the best course of action to get more information about the issue and any potential fixes.

edmorley commented 1 year ago

This issue is not related to Heroku.

The server's TLS handling seems broken (or there is otherwise some other issue with the server) - local requests fail too depending on the client used, for example this is what I get using curl locally on macOS:

$ curl -v https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 107.180.57.212:443...
* Connected to johnvansickle.com (107.180.57.212) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [322 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) Recv failure: Connection reset by peer

Ultimately I think this buildpack is not a great fit for production workloads, since:

I'd recommend that people either:

  1. Switch to https://github.com/heroku/heroku-buildpack-activestorage-preview - which is aimed at Ruby Rails apps, but includes ffmpeg so should work fine. And whilst it's not an "officially supported language buildpack" it's still fairly regularly maintained and uses the same security best practices of the official Heroku buildpacks.
  2. Use the APT buildpack
  3. Download https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz yourselves, host somewhere secure of your chosing (your own S3 bucket), then reference using the newly fixed FFMPEG_DOWNLOAD_URL
keShraa commented 1 year ago

Thanks @edmorley for the very detailed explanation 🙏🏻

ByTE1974 commented 1 year ago

Looks like it's working again!

jonathanong commented 1 year ago

If you are using this in production, I would definitely store the binary in my own file storage. you don't want to rely on third party services like this. When I was using this repo, I was using it at an early stage start up.

thanks for the help @agieche, let me know if you can help @edmorley. please email me if any issues come up because I don't code nowadays