Composer lookups with multiple repositories broke #20661

Closed herndlm closed 1 year ago

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.153.2

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

It worked in 34.152.4 and broke in 34.152.5. This issue is a follow-up of the comment https://github.com/renovatebot/renovate/issues/20649#issuecomment-1446498357 which was not caused by the issue there.

We have a composer.json with multiple repositories ala

"repositories": [
        {
            "type": "composer",
            "url": "<PRIVATE REPO 1>"
        },
        {
            "type": "composer",
            "url": "<PRIVATE REPO 2>"
        }
]

and of course some packages are installed from those. Renovate up to 34.152.4 could successfully look them up, with 34.152.5 this broke. According to the logs it looks like it doesn't even try to look it up on those repos anymore, but directly goes to packagist instead, where it will result in 404 which is expected.

Relevant debug logs

Logs

``` DEBUG: GET https://packagist.org/p2/satispress/advanced-custom-fields-pro.json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=707) (repository=) DEBUG: GET https://packagist.org/p2/satispress/advanced-custom-fields-pro~dev.json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=713) (repository=) DEBUG: Aggregate error is thrown (repository=) "err": { "name": "AggregateError", "message": "\n HTTPError: Response code 404 (Not Found)\n at Request. (/opt/buildpack/tools/renovate/34.153.1/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n HTTPError: Response code 404 (Not Found)\n at Request. (/opt/buildpack/tools/renovate/34.153.1/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)", "stack": "AggregateError: \n HTTPError: Response code 404 (Not Found)\n at Request. (/opt/buildpack/tools/renovate/34.153.1/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n HTTPError: Response code 404 (Not Found)\n at Request. (/opt/buildpack/tools/renovate/34.153.1/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at next (/opt/buildpack/tools/renovate/34.153.1/node_modules/p-map/index.js:43:14)\n at /opt/buildpack/tools/renovate/34.153.1/node_modules/p-map/index.js:67:7\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)" } DEBUG: Datasource 404 (repository=) "datasource": "packagist", "packageName": "satispress/advanced-custom-fields-pro", "url": "https://packagist.org/p2/satispress/advanced-custom-fields-pro.json DEBUG: Failed to look up packagist dependency satispress/advanced-custom-fields-pro (repository=, packageFile=composer.json, dependency=satispress/advanced-custom-fields-pro) ```

Have you created a minimal reproduction repository?

No reproduction repository

Is there any way to check whether it was solved in 34.153.2 or not?

sure, I can use any tag on Docker, it was unfortunately not solved :/

Okay, in order to narrow the search, can I ask you about of package's corresponding packages.json structure? Which of these fields are present?

packages
includes
provider-includes

The package.json from <PRIVATE REPO 2> (where the lookups currently fail, wait that also means they don't fail on PRIVATE REPO 1 I guess 😅, must still be some weird edge case and not a super global broken thing then!) looks like the following

package.json

```json { "packages": [], "includes": { "include/all$5cabeeb48f3db3d5f1a0fef6445dc0858e7d2d89.json": { "sha1": "5cabeeb48f3db3d5f1a0fef6445dc0858e7d2d89" } }, "metadata-url": "/p2/%package%.json", "available-packages": [ ] } ```

packages being empty is also weird, isn't it.. the packages.json from PRIVATE REPO 1 has a completely different format btw, that explains some things I guess.

@viceice Seems like you were right

By the way, https://github.com/renovatebot/renovate/pull/20626 will fix it

thx and sorry for coming up with this special case. that one registry is most likely super outdated...

No, no, it supports v2 so it isn't outdated, we just need to update our logic to support v2

ah ok, then it's newer than I thought and the other one is outdated 😅

thank you for the quick action here, I'll test this later when it gets released and hits dockerhub :)

:tada: This issue has been resolved in version 34.154.0 :tada:

The release is available on:

GitHub release
34.154.0

Your semantic-release bot :package::rocket:

Works, thx again!

Renovate 34.154.2 (self-hosted)

Should https://packages.drupal.org/8/packages.json work?

I see requests with https://packagist.org/p2/drupal/allowed_formats.json and https://packages.drupal.org/8/files/packages/8/p2/drupal/allowed_formats.json. But both are 403/404.

Repository url is https://packages.drupal.org/8 Metadata-url is /files/packages/8/p2/%package%.json

Works not: https://packages.drupal.org/8/files/packages/8/p2/drupal/allowed_formats.json Works: https://packages.drupal.org/files/packages/8/p2/drupal/allowed_formats.json

So the question is. Means Metadata-url append the part to the repository url or the hostname?

@tobiasbaehr can you create a reproduction repo? I'm not sure every registry treats this consistently, so @zharinov will need to test multiple types he's aware of

@tobiasbaehr Good catch, thanks

I created a repository https://github.com/tobiasbaehr/openculturas-distribution . now I need to wait until the renovate app does something?

Please create a new discussion with minimal reproduction. The one you link to will be very hard to debug due to the large amount of config and dependencies

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

I created a new repository https://github.com/tobiasbaehr/renovate-drupal-org-packages.

Renovate 34.154.2 (self-hosted)

Should https://packages.drupal.org/8/packages.json work?

Thanks for reporting, I'm experiencing the same issue! Could someone with the required permissions reopen this issue (or move it to a new one)? Thanks.

Hopefully, it's already fixed with fresh release

Fixed in release: https://github.com/renovatebot/renovate/releases/tag/34.154.4

I have setup a Drupal-related repo (and been supporting it) to provide zero-config Renovate setup for Drupal projects (GitHub app and self-hosted in GitHub and CircleCI) https://github.com/drevops/renovate-drupal-example

renovatebot / renovate