🐛 Bug Report: catalog-imports towards bitbucketCloud does not url-encode wildcards in catalogPath

[O5ten]

📜 Description

I have configured my catalog provider like this:

catalog:
  providers: 
     bitbucketCloud: 
        templates:
          catalogPath: /**/template.yaml
          schedule: 
            frequency: { minutes: 30 }
            timeout: { minutes: 3 }
          workspace: <my-workspace>
          filters: 
            repository: '.*scaffolding'

And upon startup of backstage i can see the request crashing and burning like this:

[1] 2023-09-27T07:54:37.546Z catalog error bitbucketCloud-provider:templates refresh failed Unexpected response for GET https://api.bitbucket.org/2.0/workspaces//search/code?page=1&pagelen=100&fields=-values.content_matches%2C%2Bvalues.file.commit.repository.mainbranch.name%2C%2Bvalues.file.commit.repository.project.key%2C%2Bvalues.file.commit.repository.slug%2C-values..links%2C-values...links%2C-values....links%2C%2Bvalues.file.commit.repository.links.html.href&search_query=%22template.yaml%22+path%3A%2F**%2Ftemplate.yaml. Expected 200 but got 400 - Bad Request type=plugin target=bitbucketCloud-provider:templates class=BitbucketCloudEntityProvider taskId=bitbucketCloud-provider:templates:refresh taskInstanceId=54eef599-8b77-4867-9d33-9dc9dd85c18f

The API spews out this error when i access the URL manually:

{
  "type": "error",
  "error": {
    "message": "Invalid path modifier segment: **",
    "data": {
      "key": "PATH_MODIFIER_INVALID"
    }
  },
  "data": {
    "key": "PATH_MODIFIER_INVALID"
  }
}

which led me to believe that the wildcards aren't properly encoded. So i changed my configuration to this:

catalog:
  providers: 
     bitbucketCloud: 
        templates:
          catalogPath: /%2A%2A/template.yaml

Which made the request go through properly and find my templates.

Workaround

Ditch the wildcards, just giving the name of the file it is looking for works well aswell as provided urlencoded settings directly in the app-config.yaml

👍 Expected behavior

My templates should be found if i follow the documentation and the wildcards should be properly urlencoded when targeting bitbucket. I'm not sure if it makes sense in the other providers.

👎 Actual Behavior with Screenshots

👟 Reproduction steps

1. Configure wildcards in your catalog importer towards bitbucket cloud
2. Start Backstage
3. Discover the error

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS:   Linux 5.15.0-83-generic - linux/x64
node: v18.15.0
yarn: 1.22.19
cli:  0.22.13 (installed)
backstage:  1.18.1

Dependencies:
  @backstage/app-defaults                                          1.4.3
  @backstage/backend-app-api                                       0.5.3
  @backstage/backend-common                                        0.13.5, 0.19.5
  @backstage/backend-dev-utils                                     0.1.1
  @backstage/backend-openapi-utils                                 0.0.4
  @backstage/backend-plugin-api                                    0.5.4, 0.6.3
  @backstage/backend-tasks                                         0.5.8
  @backstage/catalog-client                                        1.4.4
  @backstage/catalog-model                                         1.4.2
  @backstage/cli-common                                            0.1.12
  @backstage/cli-node                                              0.1.4
  @backstage/cli                                                   0.22.13
  @backstage/config-loader                                         1.5.0
  @backstage/config                                                1.1.0
  @backstage/core-app-api                                          1.10.0
  @backstage/core-components                                       0.13.5
  @backstage/core-plugin-api                                       1.6.0
  @backstage/errors                                                1.2.2
  @backstage/eslint-plugin                                         0.1.3
  @backstage/integration-aws-node                                  0.1.6
  @backstage/integration-react                                     1.1.19
  @backstage/integration                                           1.7.0
  @backstage/plugin-api-docs                                       0.9.11
  @backstage/plugin-app-backend                                    0.3.51
  @backstage/plugin-app-node                                       0.1.3
  @backstage/plugin-auth-backend-module-gcp-iap-provider           0.1.0
  @backstage/plugin-auth-backend-module-github-provider            0.1.0
  @backstage/plugin-auth-backend-module-gitlab-provider            0.1.0
  @backstage/plugin-auth-backend-module-google-provider            0.1.0
  @backstage/plugin-auth-backend-module-oauth2-provider            0.1.0
  @backstage/plugin-auth-backend                                   0.19.0
  @backstage/plugin-auth-node                                      0.2.19, 0.3.0
  @backstage/plugin-bitbucket-cloud-common                         0.2.12
  @backstage/plugin-catalog-backend-module-bitbucket-cloud         0.1.18
  @backstage/plugin-catalog-backend-module-msgraph                 0.5.10
  @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.0
  @backstage/plugin-catalog-backend                                1.13.0, 1.13.1
  @backstage/plugin-catalog-common                                 1.0.16
  @backstage/plugin-catalog-graph                                  0.2.36
  @backstage/plugin-catalog-import                                 0.10.0
  @backstage/plugin-catalog-node                                   1.4.4
  @backstage/plugin-catalog-react                                  1.8.4
  @backstage/plugin-catalog                                        1.13.0
  @backstage/plugin-events-node                                    0.2.12
  @backstage/plugin-github-actions                                 0.6.5
  @backstage/plugin-org                                            0.6.14
  @backstage/plugin-permission-common                              0.7.8
  @backstage/plugin-permission-node                                0.7.14
  @backstage/plugin-permission-react                               0.4.15
  @backstage/plugin-proxy-backend                                  0.4.0
  @backstage/plugin-scaffolder-backend                             1.17.1, 1.17.0
  @backstage/plugin-scaffolder-common                              1.4.1
  @backstage/plugin-scaffolder-node                                0.1.5, 0.2.3
  @backstage/plugin-scaffolder-react                               1.5.5
  @backstage/plugin-scaffolder                                     1.15.0
  @backstage/plugin-search-backend-module-catalog                  0.1.7
  @backstage/plugin-search-backend-module-pg                       0.5.12
  @backstage/plugin-search-backend-module-techdocs                 0.1.7
  @backstage/plugin-search-backend-node                            1.2.7
  @backstage/plugin-search-backend                                 1.4.3
  @backstage/plugin-search-common                                  1.2.6
  @backstage/plugin-search-react                                   1.7.0
  @backstage/plugin-search                                         1.4.0
  @backstage/plugin-tech-radar                                     0.6.8
  @backstage/plugin-techdocs-backend                               1.7.0
  @backstage/plugin-techdocs-module-addons-contrib                 1.1.0
  @backstage/plugin-techdocs-node                                  1.8.0
  @backstage/plugin-techdocs-react                                 1.1.11
  @backstage/plugin-techdocs                                       1.7.0
  @backstage/plugin-user-settings                                  0.7.10
  @backstage/release-manifests                                     0.0.10
  @backstage/test-utils                                            1.4.3
  @backstage/theme                                                 0.4.2
  @backstage/types                                                 1.1.1
  @backstage/version-bridge                                        1.0.5
Done in 0.50s.

👀 Have you spent some time to check if this bug has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

Are you willing to submit PR?

No, I don't have time to work on this right now

[Rugvip]

Thank you for reporting! 👍

Needs a fix somewhere in this areas: https://github.com/backstage/backstage/blob/388c62b03a494050aa778bf2fd170b078f54faba/plugins/catalog-backend-module-bitbucket-cloud/src/providers/BitbucketCloudEntityProvider.ts#L354

I'm not entirely sure if Bitbucket cloud supports wildcards in the way that we expect and that the fix here is to make sure things are encoded, or whether we should skip the search altogether if there's a wildcard and fall back to the URL reader search with readTree. Any thoughts on this @pjungermann?

We'll be relying on contributions for this to be fixed, appreciate any help 🙏

[pjungermann]

the problem is not about the encoding of the wildcards I believe.

You can replace /**/template.yaml with template.yaml.

By default, we look for all files regardless of their location inside the repository, supporting monorepos etc out of the box.

You can use /template.yaml (or /catalog-info.yaml) if you want only files at the root (or generalized: a specific location).

Regarding the error message: We use the code search API under the hood. There, we use the path modifier (filter) path:catalog-info.yaml. It does not support those placeholders. You can try it at Bitbucket code search yourself.

Valid:

path:catalog-info.yaml
path:/catalog-info.yaml

Invalid:

path:/**/catalog-info.yaml

See also the documentation about the catalogPath configuration option at the Bitbucket Cloud discovery documentation at backstage.io. It contains a reference to the documentation about the path modifier, too.

[pjungermann]

also valid

path:/*/catalog-info.yaml

according to Bitbucket's documentation.

[Rugvip]

@pjungermann thank you! 👍

Gonna close this, I feel the documentation sufficiently explains the catalogPath parameter. Please do open a PR if you have any suggested documentation changes though @O5ten! 🙏

It's somewhat unfortunate that it works different that other providers but it's really the reality of different capabilities of the integrated services.