[ERROR] No RateLimit present in response

[thoys]

I get the following error when trying to build my docker file which worked fine a couple hours ago.

Step 51/64 : RUN phive --no-progress install --global --trust-gpg-keys 31C7E470E2138192 squizlabs/php_codesniffer
 ---> Running in d4b64f83fa46
Phive 0.13.2 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors

[ERROR]   No RateLimit present in response

In a previous run it failed one package earlier:

Step 50/64 : RUN phive --no-progress install --global --trust-gpg-keys E82B2FB314E9906E FriendsOfPHP/PHP-CS-Fixer
 ---> Running in daaa9daa3027
Phive 0.13.2 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]   No RateLimit present in response

The 6 packages I downloaded before this worked fine. What could be going on? Am I downloading too fast?

[theseer]

No, probably not "too fast" but github's http response headers should contain a rate limit information that we parse to avoid just that. But for whatever reason, that header seems to be missing for your requests.

I cannot reproduce this though:

theseer@nyda /tmp/xx $ phive install FriendsOfPHP/PHP-CS-Fixer
Phive 0.13.2-12-gbf2bb2e-dirty - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/repos/friendsofphp/php-cs-fixer/releases
Downloading https://github.com/FriendsOfPHP/PHP-CS-Fixer/releases/download/v2.16.1/php-cs-fixer.phar
Downloading https://github.com/FriendsOfPHP/PHP-CS-Fixer/releases/download/v2.16.1/php-cs-fixer.phar.asc
Linking /home/theseer/.phive/phars/friendsofphp/php-cs-fixer-2.16.1.phar to /tmp/xx/tools/php-cs-fixer

Are you having any type of (transparent?) proxy in between which might filter our these headers?

I guess we could change our download code to not barf on missing headers and just assume everything is fine, but it feels weird...

[theseer]

May also have been a temporary issue on github's end ...

[thoys]

I probably had exceeded the request limit or something before and got myself blocked. It worked fine again today. Do you have a check for that?

[thoys]

Next time it happens I could try an make a wireshark capture if thats helpful.

[thoys]

I read For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests. on the page https://developer.github.com/v3/#rate-limiting . If I ran the docker build (which contains about 8 phive calls) multiple times in one hour it might have exceeded the limit?

[theseer]

You certainly could exceed your limits. I ran into that while developing phive a lot before we implemented authentication token support.

But: Github usually supplies a set of X-RateLimit-XXXX headers, regardless whether the request was successful or (already) blocked. That's why it is confusing to me that - at least according to our error message - those headers seem to be missing in the response.

If you run into this problem again, you could try to do a curl/wget request to the URL shown before it happend in verbose mode so you get the HTTP response headers along. (If no URL is shown in the output, please make sure you get STDOUT as well as STDERR output ;)

Phive supports using access tokens - simply set the GITHUB_AUTH_TOKEN environment variable. We do have pending tickets to document and enhance its usage, though (see #178 and https://github.com/phar-io/phar.io/issues/25).

[thoys]

Great thanks, for now I don't think this ticket needs action, if someone finds it through a search engine when having the same issue it could be re-opened. I will set a GITHUB_AUTH_TOKEN and don't expect it to happen again in the near future.

[Surf-N-Code]

I just ran into this issue as well.

export GITHUB_AUTH_TOKEN=gittoken fixed it. Thanks for the tip!

[theseer]

Looks like the code is having issues with either not seeing or not receiving the RateLimit headers from github. I'll reopen this and have a look.

[mjnbock]

I am having the problem now. wget -S 'https://github.com/phar-io/phive/releases/download/0.13.2/phive-0.13.2.phar shows

--2020-03-30 14:46:14--  https://github.com/phar-io/phive/releases/download/0.13.2/phive-0.13.2.phar
Resolving github.com (github.com)... 140.82.118.3
Connecting to github.com (github.com)|140.82.118.3|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  date: Mon, 30 Mar 2020 14:45:05 GMT
  content-type: text/html; charset=utf-8
  server: GitHub.com
  status: 302 Found
  vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
  location: https://github-production-release-asset-2e65be.s3.amazonaws.com/31674534/04998680-faac-11e9-87d1-3a90ceae0228?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T144505Z&X-Amz-Expires=300&X-Amz-Signature=d695b84022e0458a46bf048568c3b445e29c80d85be530c60be3fc05240729cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dphive-0.13.2.phar&response-content-type=application%2Foctet-stream
  cache-control: no-cache
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  x-frame-options: deny
  x-content-type-options: nosniff
  x-xss-protection: 1; mode=block
  expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
  Set-Cookie: _gh_sess=4bQzi%2FKOHKW88fgcQlTSn5UYiwD7htihXvbp1gJN1xjkGATQLMAYFQCfaGhwjkkQyda%2BxifEY%2FOMioar4u1WXGwuRhg877dQ8qVJjfzEe3RaLYH2et8QAQhI4d9Kfc%2Bkm2wCrvhgwyO18Uuzzuaa53mPCRuVkxEE4OVn1PeqY9FP2rRhgwD2qKs0fzpwVIbxZ%2BafS5VFLJ76cU9t1N4vo%2BqcJCeT989RUsEgODvqkQnuvkYdTR6c8PVG%2Fvzr5b%2FTyMtT%2FAHHIGHRXjW14pyVZQ%3D%3D--O0lwOhxdwgWIxDDS--dgTNJg7j7F49X8TBYtQDFQ%3D%3D; Path=/; HttpOnly; Secure
  Set-Cookie: _octo=GH1.1.576589217.1585579575; Path=/; Domain=github.com; Expires=Tue, 30 Mar 2021 14:46:15 GMT; Secure
  Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Tue, 30 Mar 2021 14:46:15 GMT; HttpOnly; Secure
  Content-Length: 607
  X-GitHub-Request-Id: BA81:EEC4:3FF0BE8:5D4A8F7:5E820636
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/31674534/04998680-faac-11e9-87d1-3a90ceae0228?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T144505Z&X-Amz-Expires=300&X-Amz-Signature=d695b84022e0458a46bf048568c3b445e29c80d85be530c60be3fc05240729cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dphive-0.13.2.phar&response-content-type=application%2Foctet-stream [following]
--2020-03-30 14:46:15--  https://github-production-release-asset-2e65be.s3.amazonaws.com/31674534/04998680-faac-11e9-87d1-3a90ceae0228?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T144505Z&X-Amz-Expires=300&X-Amz-Signature=d695b84022e0458a46bf048568c3b445e29c80d85be530c60be3fc05240729cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dphive-0.13.2.phar&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.143.140
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.143.140|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  x-amz-id-2: K4ZSELOa7GSDEhp6/vpPwBw5IyQi6/GqYH3lkyh+gz1a/G57lxwidIYvd1TNjTrT7VX6CQz1Aks=
  x-amz-request-id: 23A7CB96BB664D4F
  Date: Mon, 30 Mar 2020 14:46:16 GMT
  Last-Modified: Tue, 29 Oct 2019 23:27:15 GMT
  ETag: "1e44235059d526839dd1fbece1e1d5b8"
  Content-Disposition: attachment; filename=phive-0.13.2.phar
  Accept-Ranges: bytes
  Content-Type: application/octet-stream
  Content-Length: 163055
  Server: AmazonS3
Length: 163055 (159K) [application/octet-stream]
Saving to: 'phive-0.13.2.phar.1'

     0K .......... .......... .......... .......... .......... 31%  263K 0s
    50K .......... .......... .......... .......... .......... 62%  526K 0s
   100K .......... .......... .......... .......... .......... 94% 16.0M 0s
   150K .........                                             100% 17.8M=0.3s

2020-03-30 14:46:15 (551 KB/s) - 'phive-0.13.2.phar.1' saved [163055/163055]

[dnaber-de]

I face the same issue when running phive on Codeship.com CI. On a local docker container it works though.

[theseer]

Looks like Github changed the download of assets to be a redirect to s3 (of all places, why not azure? ;-p ) rather than having it count as an API hit.

While that's a good thing, phive seems to be confused about it. Not sure why it would work in some cases then and not fail all the time.

This needs a bit more investigation but i'll probably relax the code to simply be happy if no RateLimit header is there.

[theseer]

The code already is relatively flexible in that regard.

But: As it seems, Github no longer allows HEAD requests against their rate limit check api endpoint.

Using HEAD for other endpoints still counts against the API limit, so we'll use GET now and ignore the response data.

[theseer]

It would be awesome if someone could try this before I make a new release to see if that actually fixes this issue... :)

[mjnbock]

On Debian the problem persists:

$ git clone https://github.com/phar-io/phive
$ cd phive && composer install
$ php phive selfupdate
Phive 0.13.2-14-g37a22dc - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/rate_limit
[ERROR]    No RateLimit present in response 

[theseer]

sigh

I wish I could reproduce this ;)

theseer@nyda /tmp/xxx $ git clone https://github.com/phar-io/phive
Cloning into 'phive'...
remote: Enumerating objects: 85, done.
remote: Counting objects: 100% (85/85), done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 9045 (delta 32), reused 61 (delta 28), pack-reused 8960
Receiving objects: 100% (9045/9045), 2.26 MiB | 1.77 MiB/s, done.
Resolving deltas: 100% (6329/6329), done.
theseer@nyda /tmp/xxx $ cd phive && composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 6 installs, 0 updates, 0 removals
  - Installing phar-io/filesystem (dev-master 4fc0abe): Cloning 4fc0abed6a from cache
  - Installing phar-io/executor (dev-master 73ee06f): Cloning 73ee06f202 from cache
  - Installing phar-io/gnupg (dev-master 788b591): Cloning 788b591d04 from cache
  - Installing phar-io/version (2.0.1): Loading from cache
  - Installing phar-io/manifest (1.0.3): Loading from cache
  - Installing mikey179/vfsstream (v1.6.7): Loading from cache
Generating autoload files
theseer@nyda /tmp/xxx/phive master $ php phive selfupdate
Phive 0.13.2-14-g37a22dc - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/rate_limit
Downloading https://api.github.com/repos/phar-io/phive/releases
[ERROR]    An error occurred while processing your request:

          Version string '0.13.2-14-g37a22dc' does not follow SemVer semantics

The rate limit check works just fine? Anyone got any idea how to force the error this issue is about?

[theseer]

Looks like I managed:

[theseer@chase phive]$ php phive selfupdate
Phive 0.13.2-14-g37a22dc - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Downloading https://api.github.com/rate_limit
[ERROR]   No RateLimit present in response

I have no idea what's different on that box though ;)

[mjnbock]

GithubAliasResolver.php:

private function initRateLimit(): void {
    if ($this->rateLimit !== null) {
        return;
    }

    $response        = $this->httpClient->get(new Url('https://api.github.com/rate_limit'));
    $this->rateLimit = $response->getRateLimit(); // XXX throws
}

HttpResponse.php:

public function __construct(int $httpCode, string $responseBody, ETag $etag = null, RateLimit $rateLimit = null) {
    $this->responseBody = $responseBody;
    $this->httpCode     = $httpCode;
    $this->etag         = $etag;
    $this->rateLimit    = $rateLimit; // defaults to null
}
// ...
public function hasRateLimit(): bool {
    return $this->rateLimit !== null;
}
// ...
public function getRateLimit(): RateLimit {
    if (!$this->hasRateLimit()) {
        throw new HttpResponseException('No RateLimit present in response');
    }

    return $this->rateLimit;
}

CurlHttpClient.php

private function execRequest(): HttpResponse {
    $this->rateLimitHeaders = [];

    $result = $this->curl->exec();

    $httpCode = $this->curl->getHttpCode();

    if ($httpCode >= 400 || \in_array($httpCode, [200, 304], true)) {
        return new HttpResponse($httpCode, $result ?: '', $this->etag, $this->parseRateLimitHeaders());
    }
// ...
private function parseRateLimitHeaders(): ?RateLimit {
    $required  = ['Limit', 'Remaining', 'Reset'];
    $exisiting = \array_keys($this->rateLimitHeaders); // empty []

    if (\count(\array_intersect($required, $exisiting)) < 3) {
        return null; // XXX so HttpResponse hasRateLimit() returns false
    }

[theseer]

Found it!

Technically, it's a bug on Github's side as their response headers are inconsistent. Some servers' response header use X-RateLimit-xxxxx (as is "correct") while others respond with X-Ratelimit-xxxx (which, according to their own documentation, is incorrect).

Phive only looks for the variant with capital L. I'll change that to be case independent...

[thoys]

Great news :). Mystery solved

[theseer]

@mjnbock I know where the exception came from but I wanted a scenario to reproduce the issue. But thanks :)

[theseer]

Okay, fix implemented and tested. Will be in the next release.

[ravage84]

Okay, fix implemented and tested. Will be in the next release.

@theseer I just ran into this on a Windows 7 box. Is there an ETA for the next release?

[theseer]

@ravage84 Working on it, with a bit of luck (read: my daughter sleeping a bit earlier today ;) ) I'll release 0.14.0 today.

[dereuromark]

I also just ran into it today for "selfupdate". So the release is very welcome indeed :)

[ostrolucky]

To encourage fixing this, I am posting obligatory "me too"

[michaelKaefer]

Me too on phive selfupdate.

[amenk]

There is no backport for PHP7.1, right? We still build docker images for PHP7.1 and currently phive does not seem to work here anymore :(

[theseer]

I created a 0.13.3 Release for you.

This will mark the definite end for PHP 7.1 compatible releases though.

[amenk]

wow, awesome, thanks :)

[ramsey]

I ran into this problem just now, and adding a GITHUB_AUTH_TOKEN environment variable fixed it for me.

[theseer]

@ramsey What version where you having this issue with? While you of course might have run into a Rate Limit issue in general, this particular bug should no longer exist.

[ramsey]

@theseer I was contributing to CaptainHook when I ran into this issue, so it's possible they're using an older version of Phive. Their CONTRIBUTING.md instructions show this command, which is where I saw the error:

tools/phive --home ./.phive install --trust-gpg-keys 4AA394086372C20A,31C7E470E2138192,8E730BA25823D8B5 --force-accept-unsigned

[theseer]

Thanks!

The command call looks sane, but indeed they are on 0.13.2. The problem is fixed in 0.13.3 and up.

/ping @sebastianfeldmann

[sebastianfeldmann]

Thanks for the ping, I updated phive to the latest version

[MacDada]

I'm getting the error on version 0.13.3:

$ bin/phive.phar install
Phive 0.13.3 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]    No RateLimit present in response 

$ bin/phive.phar version
Phive 0.13.3 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors

$ cat phive.xml 
<?xml version="1.0" encoding="UTF-8"?>
<phive xmlns="https://phar.io/phive">
  <phar name="wapmorgan/phpcodefixer" version="^2.0" installed="2.0.15" location="./tools/phpcf" copy="false"/>
</phive>

[theseer]

I don't see how this would be possible. And, worse, I cannot reproduce it. Is that the only output you get?

[theseer]

Okay, managed. Interesting.

[theseer]

Looks like I didn't backport all of the Fix in 0.13.3. Try 0.13.4.

[MacDada]

$ bin/phive.phar selfupdate
Phive 0.13.3 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]    No RateLimit present in response 

xD

[MacDada]

I downloaded latest 0.14.4 manually. When I look into phive.phar/.phar/stub.php I can see that it is 0.14.4 –– OK.

It no longer complains about RateLimit.

It does complain that I use PHP 7.1, but that's a different issue…

[theseer]

I said 0.13.4 for PHP <7.2 ;-)

Am Montag, 21. September 2020 schrieb Dawid Nowak:

I downloaded latest 0.14.4 manually. When I look into phive.phar/.phar/stub.php I can see that it is 0.14.4 –– OK.

It no longer complains about RateLimit.

It does complain that I use PHP 7.1, but that's a different issue…

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/phar-io/phive/issues/228#issuecomment-69612475

-- Gesendet von meinem Sailfish-Gerät.

[MacDada]

I said 0.13.4 for PHP <7.2 ;-)

Oh, my bad. It works! Thx :)

[MacDada]

Well, it worked for a few times…

$ bin/phive.phar install
Phive 0.13.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Copying phpcs-3.5.2.phar to ./tools/phpcs
Linking /home/vagrant/.phive/phars/php-cs-fixer-2.14.0.phar to ./tools/php-cs-fixer

$ bin/phive.phar install
Phive 0.13.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors
Copying phpcs-3.5.2.phar to ./tools/phpcs
[ERROR]    No RateLimit present in response 

$ bin/phive.phar --version
Phive 0.13.4 - Copyright (C) 2015-2020 by Arne Blankerts, Sebastian Heuer and Contributors

[theseer]

I admit I'm confused.

The exception is thrown when phive tries to retrieve rate limiting information from the response. The only point in time where that is not wrapped into a check (by calling $response->hasRateLimit()) is when initializing the RateLimit in the github alias resolver.

So that must be the culprit. But for that to fail, Github must not be responding with X-RateLimit-*-Headers on the API call for Rate Limits. Which I consider unlikely - and fail to trigger.

So I'm lost as to why this might be happening. Would you be interesting i trying a debug-build I'd provide to you?

[theseer]

Ah! Github started to block HEAD requests to their rate limit API endpoint.

[MacDada]

Ah! Github started to block HEAD requests to their rate limit API endpoint.

Okeeeeey… So i guess you know how to fix that?

Would you be interesting i trying a debug-build I'd provide to you?

Sure, bring it on :)

[theseer]

Try 0.13.5 :)

[JSabelli-FDX]

It's happening again on 0.15.2

$ phive install phpmetrics/PhpMetrics
Phive 0.15.2 - Copyright (C) 2015-2024 by Arne Blankerts, Sebastian Heuer and Contributors
[ERROR]   No RateLimit present in response