search

gaffneyc / heroku-buildpack-jemalloc

Heroku buildpack that installs the Jemalloc memory allocator
BSD 3-Clause "New" or "Revised" License
252 stars 77 forks source link

Buildpack suddenly failing our builds #13

Closed rubendinho closed 4 years ago

rubendinho commented 4 years ago

We've been using this buildback with no problems for some time now.

Today, our builds started failing with the below message. 

-----> jemalloc app detected
-----> Vendoring jemalloc 5.2.1
       Fetching https://github.com/gaffneyc/heroku-buildpack-jemalloc/releases/download/heroku-18/jemalloc-5.2.1.tar.bz2
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now
 !     Push rejected, failed to compile jemalloc app.
 !     Push failed

There have been no buildpack/configuration changes and we did not have this problem as recently as yesterday.

After removing the buildpack and retrying, the build was successful.

Any ideas on what could be the issue?

Thanks

ghost commented 4 years ago

same here

markysharky70 commented 4 years ago

same here

dorthwein commented 4 years ago

same ;(

markets commented 4 years ago

Exactly same problem here ☝️

remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> jemalloc app detected
remote: -----> Vendoring jemalloc 5.2.1
remote:        Fetching https://github.com/gaffneyc/heroku-buildpack-jemalloc/releases/download/heroku-18/jemalloc-5.2.1.tar.bz2
remote: bzip2: (stdin) is not a bzip2 file.
remote: tar: Child returned status 2
remote: tar: Error is not recoverable: exiting now
remote:  !     Push rejected, failed to compile jemalloc app.
markysharky70 commented 4 years ago

Interesting. I changed my heroku env var JEMALLOC_VERSION to 5.2.1, that didn't fix the problem so I rolled it back using the heroku interface under "activity" which reset to 5.1.0 and now it deployed successfully. I was on 5.1.0 originally and not sure why it was failing trying to vendor 5.2.1 and ignoring my JEMALLOC_VERSION setting in the heroku ENV. Hope this helps someone. To clarify - it now vendored 5.1.0 as my env var states.

gaffneyc commented 4 years ago

First thought is that there was a hiccup on GitHub’s end with their releases endpoint. We don’t do a good job of validating that the request was successful so a 404 ends up erroring in bzip.

I’ll look into this more once I’m able to get to a computer.

alecgorge commented 4 years ago

Specifying JEMALLOC_VERSION=5.1.0 worked for us as well

gaffneyc commented 4 years ago

GitHub is returning 403 Forbidden for 5.2.1 right now. Could be a permissions issue on their S3 bucket.

wget https://github.com/gaffneyc/heroku-buildpack-jemalloc/releases/download/heroku-18/jemalloc-5.2.1.tar.bz2
--2020-05-07 19:18:20--  https://github.com/gaffneyc/heroku-buildpack-jemalloc/releases/download/heroku-18/jemalloc-5.2.1.tar.bz2
Resolving github.com... 140.82.112.4
Connecting to github.com|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/113065003/685c2e00-b7a1-11e9-87f8-a8905d4da36a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200507%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200507T231801Z&X-Amz-Expires=300&X-Amz-Signature=53bf3dc2f0af4dbe0dbf7c9dfc99c93e1428c3b4d437f56c652557b354942775&X-Amz-SignedHeaders=host&actor_id=0&repo_id=113065003&response-content-disposition=attachment%3B+filename%3Djemalloc-5.2.1.tar.bz2&response-content-type=application%2Foctet-stream [following]
--2020-05-07 19:18:20--  https://github-production-release-asset-2e65be.s3.amazonaws.com/113065003/685c2e00-b7a1-11e9-87f8-a8905d4da36a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200507%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200507T231801Z&X-Amz-Expires=300&X-Amz-Signature=53bf3dc2f0af4dbe0dbf7c9dfc99c93e1428c3b4d437f56c652557b354942775&X-Amz-SignedHeaders=host&actor_id=0&repo_id=113065003&response-content-disposition=attachment%3B+filename%3Djemalloc-5.2.1.tar.bz2&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.92.211
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.92.211|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-05-07 19:18:20 ERROR 403: Forbidden.
gaffneyc commented 4 years ago

There is an open issue for errors downloading releases

https://www.githubstatus.com/incidents/1p369ttzrnqy

alecgorge commented 4 years ago

I'm getting 302 -> 200s in any authenticated request and signature errors for anonymous requests:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <AWSAccessKeyId>AKIAIWNJYAX4CSVEH53A</AWSAccessKeyId>
    <StringToSign>AWS4-HMAC-SHA256
20200507T232450Z
20200507/us-east-1/s3/aws4_request
2ed18a106fbed2bb740aaac554aab63be163e9e9f68a1aa6c0271b84aac98891</StringToSign>
    <SignatureProvided>33271da11e2ae3f394724a061294e257401af1c4a7e1dd309d82cfef6e77d23e</SignatureProvided>
    <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 30 30 35 30 37 54 32 33 32 34 35 30 5a 0a 32 30 32 30 30 35 30 37 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 32 65 64 31 38 61 31 30 36 66 62 65 64 32 62 62 37 34 30 61 61 61 63 35 35 34 61 61 62 36 33 62 65 31 36 33 65 39 65 39 66 36 38 61 31 61 61 36 63 30 32 37 31 62 38 34 61 61 63 39 38 38 39 31</StringToSignBytes>
    <CanonicalRequest>GET
/113065003/685c2e00-b7a1-11e9-87f8-a8905d4da36a
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200507%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20200507T232450Z&amp;X-Amz-Expires=300&amp;X-Amz-SignedHeaders=host&amp;actor_id=0&amp;repo_id=113065003&amp;response-content-disposition=attachment%3B%20filename%3Djemalloc-5.2.1.tar.bz2&amp;response-content-type=application%2Foctet-stream
host:github-production-release-asset-2e65be.s3.amazonaws.com

host
UNSIGNED-PAYLOAD</CanonicalRequest>
    <CanonicalRequestBytes>47 45 54 0a 2f 31 31 33 30 36 35 30 30 33 2f 36 38 35 63 32 65 30 30 2d 62 37 61 31 2d 31 31 65 39 2d 38 37 66 38 2d 61 38 39 30 35 64 34 64 61 33 36 61 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 49 57 4e 4a 59 41 58 34 43 53 56 45 48 35 33 41 25 32 46 32 30 32 30 30 35 30 37 25 32 46 75 73 2d 65 61 73 74 2d 31 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 30 30 35 30 37 54 32 33 32 34 35 30 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 26 61 63 74 6f 72 5f 69 64 3d 30 26 72 65 70 6f 5f 69 64 3d 31 31 33 30 36 35 30 30 33 26 72 65 73 70 6f 6e 73 65 2d 63 6f 6e 74 65 6e 74 2d 64 69 73 70 6f 73 69 74 69 6f 6e 3d 61 74 74 61 63 68 6d 65 6e 74 25 33 42 25 32 30 66 69 6c 65 6e 61 6d 65 25 33 44 6a 65 6d 61 6c 6c 6f 63 2d 35 2e 32 2e 31 2e 74 61 72 2e 62 7a 32 26 72 65 73 70 6f 6e 73 65 2d 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3d 61 70 70 6c 69 63 61 74 69 6f 6e 25 32 46 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 68 6f 73 74 3a 67 69 74 68 75 62 2d 70 72 6f 64 75 63 74 69 6f 6e 2d 72 65 6c 65 61 73 65 2d 61 73 73 65 74 2d 32 65 36 35 62 65 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
    <RequestId>766A2F7938E03094</RequestId>
    <HostId>yfkv3G5Ojad9Q2h/UsGfU2AltkuORvpdbjyklZZsqdSKhfhSv6+rsgueVzynoyO001hmKpcnUbE=</HostId>
</Error>

Interesting that it is only for some releases. Luck 5.1.0 works!

markysharky70 commented 4 years ago

Thank you @gaffneyc for your quick responses, appreciate it!

gaffneyc commented 4 years ago

GitHub has marked the incident as resolved, so everything should be working again.

This does highlight a couple areas that could be improved with the buildpack.

  1. Better errors when downloads fail
  2. Storing downloads in $CACHE_DIR
  3. Option to specific an alternative download location

I'm looking at 1 right now. For 2, I've done something similar for the spiped buildpack where it compiles a cached version into $CACHE_DIR. This might be a good chance to look at solving #12 as well. We've had a number of folks fork the repo so they can guarantee the binaries they're using are safe and being able to provide an alternative location would help work around this kind of thing in the future.

rubendinho commented 4 years ago

I can confirm that everything is working again. Thanks @gaffneyc for your prompt attention and for all your work with this project! Closing this since it sounds like there's a long-term fix in the works as well.