403 When Trying to Register Service

[ImIOImI]

Describe the bug Received the error message:

Failed to register service with DataDog. Status Code: 403 Body: {"status":"error","code":403,"errors":["Forbidden"],"statuspage":"http://status.us5.datadoghq.com","twitter":"http://twitter.com/datadogops","email":"support@datadoghq.com"}

When providing action with valid credentials, confirmed via curl command as described in the API docs

To Reproduce Steps to reproduce the behavior:

1. Created the GH Action as below
2. Set up secrets
3. manually dispatch the action

Expected behavior I expected the action to trigger, and the service catalog to have been updated

Screenshots If applicable, add screenshots to help explain your problem.

Workflow definition (PLEASE REDACT ANY SENSITIVE INFORMATION):

---
name: Datadog Service Catalog Metadata Provider 

on:
  push:
    branches:
      - main
    paths:
      - 'services/myService/**'
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      # This uses the custom action to push the service definition to Datadog.
      - uses: arcxp/datadog-service-catalog-metadata-provider@v2
        with:
          datadog-hostname: api.us5.datadoghq.com
          datadog-key: ${{ secrets.DD_API_KEY }}
          datadog-app-key: ${{ secrets.DD_APP_KEY }}
          github-token: ${{ secrets.GH_TOKEN }}
          service-name: redacted.api
          team: My Team
          email: engineering@mydomain.com
#          slack-support-channel: 'https://team-name-here.slack.com/archives/ABC123'
          tags: |
            - marval:true
          repos: |
            - name: primary service repo
              url: https://github.com/Path/To/Repo
              provider: github
            - name: artifacts repo
              url: https://github.com/Path/To/Artifacts/Repo
              provider: github

Additional context Add any other context about the problem here.

[manchicken]

@ImIOImI, this authentication error typically has one of two causes:

1. The datadog-hostname is pointing to the wrong region
2. The datadog-app-key or datadog-key keys are either incorrect or not provisioned.

Can you check to make sure that you don't have any whitespace in your keys, and verify that the datadog-hostname is the correct region you use for Datadog?

[manchicken]

If those don't help, please run the following command with the appropriate values, and paste the results as text (REDACT ALL SECRETS): https://docs.datadoghq.com/api/latest/service-definition/?code-lang=curl#get-all-service-definitions

curl -X GET "https://api.datadoghq.com/api/v2/services/definitions" \
  -H "Accept: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"

Please note: I do not currently work for Datadog, nor have I have worked for them. I do not have any visibility into Datadog systems that you won't have.

[ImIOImI]

Thanks for your surprisingly prompt reply! I reset the api and app keys to ensure there were no spaces.

You literally linked to the same documentation I did to confirm my credentials worked... in my case I'm in the same region as the above yaml I posted and I ran:

curl -X GET "https://api.us5.datadoghq.com/api/v2/services/definitions" \
-H "Accept: application/json" \
-H "DD-API-KEY: <redacted>" \
-H "DD-APPLICATION-KEY: <redacted>"

and I got a successful response. I pointed it to an alternate region as well to reproduce the forbidden response, and confirmed the same as well.

[manchicken]

Also, could you run the GitHub Action with debugging enabled and post the output here? As-is, I don't have enough information to troubleshoot, so I'm trying to get more.

1. Did your curl test use the same secrets as your Action?
2. Are you sure there're no space characters in your secrets in GitHub Actions?
3. How long have you been using Service Catalog for this service?
4. Has this Custom GitHub Action ever worked for you?

As always, please redact any sensitive information, and feel free to truncate service definitions in any logs or outputs. Since this is an authentication question, it is unlikely that the service definitions themselves will make any difference here.

[ImIOImI]

1) yes 2) as sure as I can be, I copied them directly from the DD api and app key pages, pasted them into the org secrets, looked for any spaces and created the secret. 3) This is the first time. This would be to create meta data for a service that never had it before. 4) No. I was browsing through DD and wondering why filtering/tagging/documentation and all that sucked so bad on all my services and discovered they need metadata and this action looked like a pretty awesome way to configure all that... then I got really sad when it didn't work.

Here are the output logs... its pretty sparse as it is, I kinda just wanted to figure out if it worked or not before going much deeper:

Run arcxp/datadog-service-catalog-metadata-provider@v[2](https://github.com/<redacted>/marval/actions/runs/7632726345/job/20793504013#step:2:2)
  with:
    datadog-hostname: api.us5.datadoghq.com
    datadog-key: ***
    datadog-app-key: ***
    github-token: ***
    service-name: marval.api
    team: B2B Backend Engineers
    email: engineering@<redacted>.com
    tags: - marval:true

    repos: - name: primary service repo
    url: https://github.com/<redacted>/marval/tree/main/services/marval
    provider: github
  - name: artifacts repo
    url: https://github.com/<redacted>/artifacts-marval/tree/main/marval
    provider: github

    schema-version: v2
Error: Failed to register service with DataDog. Status Code: 40[3](https://github.com/<redacted>/marval/actions/runs/7632726345/job/20793504013#step:2:3) Body: {"status":"error","code":[4](https://github.com/<redacted>/marval/actions/runs/7632726345/job/20793504013#step:2:4)03,"errors":["Forbidden"],"statuspage":"http://status.us[5](https://github.com/<redacted>/marval/actions/runs/7632726345/job/20793504013#step:2:5).datadoghq.com","twitter":"http://twitter.com/datadogops","email":"support@datadoghq.com"}

[manchicken]

I do need all of the debug logs. There should be a lot more than this. This looks very different from what I see when I run the tests with debug info.

This Action doesn't do anything fancy with the credentials, it just puts them in the authorization header going out to the server.

I'm wondering if there's an issue with the secrets.

Could you try adding the curl command that worked on your local box as a separate step, using exactly the same syntax for accessing the secrets as what you're using in the workflow?

This would help us confirm that A) the secrets are accessible and do work, and B) that we definitely have the right Datadog host.

[ImIOImI]

@manchicken I figured it out. I didn't catch that someone overrode my org secret with an invalid repository secret. Thanks for the suggestion of running the curl script as an action, that helped me figure out what was up. Sorry for wasting your time.

[manchicken]

Not at all, friend! I love troubleshooting, it's just difficult to do from a distance when sensitive information is involved.

Thank you for bringing the issue up, and thanks for letting me know what the problem is.

Please let me know if you have any more issues, or even feature suggestions.