New 24.10.1 Self-hosted Issues

[chaseconey]

Describe the bug

Ever since the newest release (24.10.1), PRs are no longer getting coverage reported in the system.

To Reproduce Steps to reproduce the behavior:

1. Be running latest published docker version of all components (24.10.1)
2. Submit PR with a change that is sure to have test coverage deltas
3. No PR comment is available after report uploaded to system

Expected behavior A new comment on the PR that shows the test delta, as well as an entry in the Codecov UI for that PR that shows the difference in coverage.

Screenshots

Additional context

Ever since the update, all the components surrounding the process seem to be working. However, there are no longer deltas shown in PRs or in the main area for a repo.

There is an error showing in the PR area inside of codecov that may be a hint?

Missing Head Commit
Unable to compare commits because the head commit was not found.
[Learn more here](https://docs.codecov.com/docs/error-reference#section-missing-head-commit)

The logs show that it may be related to a GraphQL rate-limiting issue, but I don't know exactly why this is happening suddenly.

Relevant log entries:

{
    "message": "GraphQL Request",
    "asctime": "2024-10-14 19:42:51,280",
    "name": "graphql_api.views",
    "levelname": "INFO",
    "lineno": 228,
    "pathname": "/app/graphql_api/views.py",
    "funcName": "post",
    "threadName": "ThreadPoolExecutor-478_0",
    "taskName": "Task-8921",
    "server_hostname": "ip-<REDACTED>.ec2.internal",
    "request_method": "POST",
    "request_path": "/graphql/github",
    "request_body": {
        "query": "query GetRepoOverview($owner: String!, $repo: String!) { owner(username: $owner) { isCurrentUserActivated repository(name: $repo) { __typename ... on Repository { private defaultBranch oldestCommitAt coverageEnabled bundleAnalysisEnabled testAnalyticsEnabled languages } ... on NotFoundError { message } } } }",
        "variables": {
            "owner": "<REDACTED>",
            "repo": "<REDACTED>"
        }
    },
    "user": "User object (1)",
    "logger.name": "graphql_api.views",
    "logger.thread_name": "ThreadPoolExecutor-478_0",
    "level": "INFO"
}

and the very next entry in the logs:

{
    "message": "[GQL Rate Limit] - Incrementing rate limit for key",
    "asctime": "2024-10-14 19:42:51,282",
    "name": "graphql_api.views",
    "levelname": "WARNING",
    "lineno": 340,
    "pathname": "/app/graphql_api/views.py",
    "funcName": "_check_ratelimit",
    "threadName": "ThreadPoolExecutor-478_0",
    "taskName": "Task-8921",
    "key": "rl-user:1",
    "limit": 300,
    "count": "b'25'",
    "user_id": 1,
    "logger.name": "graphql_api.views",
    "logger.thread_name": "ThreadPoolExecutor-478_0",
    "level": "WARNING"
}

Really unsure of where to go from here. Are there any additional changes on the GitHub App side or configuration side that need to be updated for this latest version? Something else?

Any assistance would be helpful!

Thanks!

[drazisil-codecov]

We enabled rate limits for GraphQL. I have raised the feedback that we appear to have not disabled them for self-hosted installs.

[drazisil-codecov]

Hi @chaseconey ,

I don't think the rate limiting on the GraphQL API calls is the cause here. Do you have coverage for the HEAD SHA of your PR? Is it possible that coverage was uploaded under a different SHA and so the error you see is correct?

[chaseconey]

@drazisil-codecov Hey there 👋 .

We have coverage on ~15 or so repos and all of them stopped reporting after the upgrade. They all have head/main coverage in the way that you describe.

[drazisil-codecov]

Hmm. So far, I think it's only you, so the next step in troubleshooting Unable to compare commits because the head commit was not found. would be in the Database. You would need to locate the PRs in the pulls table, and see if Codecov has the head commit saved as the one you think it should be.

You may also be able to get this information from the logs for the Notify task, though I'm not sure if the SHAs are logged at that time. Let me check the code.

[drazisil-codecov]

Oh, Unable to compare commits because the head commit was not found. is a UI message, I thought it was part of the PR comment. What do those look like, or are they not posting? You will want to look for logs that say "Notifications done"

This log message will list all statuses and the PR comment, and if they were sent or not for a specific reason.

[chaseconey]

I thought it was part of the PR comment. What do those look like, or are they not posting?

There are no PR comments coming through any longer.

You will want to look for logs that say "Notifications done"

I am not seeing anything with this in the logs, unfortunately.

Looking at the order of events more closely I am seeing a new possibility.

Order of events:

• PR is opened
• Tests run, coverage generated
• Uploader: Coverage report uploads successfully to codecov
• Api: upload successful message
• Worker: Received upload task
• Worker: grabs repo info a few times
• Worker: Creating new report for commit
• ERROR (message below)

{
    "message": "Task app.tasks.upload.Upload[04d3b26b-95ad-4df1-9e90-373f8a639f64] raised unexpected: ResponseError(\"wrong number of arguments for 'lpop' command\")",
    "asctime": "2024-10-16 13:38:14,175",
    "name": "celery.app.trace",
    "levelname": "ERROR",
    "lineno": 270,
    "pathname": "/usr/local/lib/python3.12/site-packages/celery/app/trace.py",
    "funcName": "_log_error",
    "threadName": "MainThread",
    "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/celery/app/trace.py\", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/celery/app/trace.py\", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/worker/tasks/base.py\", line 296, in run\n    return self.run_impl(db_session, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/worker/tasks/upload.py\", line 344, in run_impl\n    return self.run_impl_within_lock(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/sentry_sdk/tracing_utils.py\", line 673, in func_with_tracing\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/worker/tasks/upload.py\", line 505, in run_impl_within_lock\n    for arguments in upload_context.arguments_list():\n  File \"/worker/tasks/upload.py\", line 178, in arguments_list\n    while arguments := self.redis_connection.lpop(uploads_list_key, count=50):\n                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/commands/core.py\", line 2685, in lpop\n    return self.execute_command(\"LPOP\", name, count)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1258, in execute_command\n    return conn.retry.call_with_retry(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/retry.py\", line 46, in call_with_retry\n    return do()\n           ^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1259, in <lambda>\n    lambda: self._send_command_parse_response(\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1235, in _send_command_parse_response\n    return self.parse_response(conn, command_name, **options)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1275, in parse_response\n    response = connection.read_response()\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/connection.py\", line 882, in read_response\n    raise response\nredis.exceptions.ResponseError: wrong number of arguments for 'lpop' command",
    "taskName": null,
    "data": {
        "hostname": "celery@ip-<REDACTED>.ec2.internal",
        "id": "04d3b26b-95ad-4df1-9e90-373f8a639f64",
        "name": "app.tasks.upload.Upload",
        "exc": "ResponseError(\"wrong number of arguments for 'lpop' command\")",
        "traceback": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/celery/app/trace.py\", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/celery/app/trace.py\", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/worker/tasks/base.py\", line 296, in run\n    return self.run_impl(db_session, *args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/worker/tasks/upload.py\", line 344, in run_impl\n    return self.run_impl_within_lock(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/sentry_sdk/tracing_utils.py\", line 673, in func_with_tracing\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/worker/tasks/upload.py\", line 505, in run_impl_within_lock\n    for arguments in upload_context.arguments_list():\n  File \"/worker/tasks/upload.py\", line 178, in arguments_list\n    while arguments := self.redis_connection.lpop(uploads_list_key, count=50):\n                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/commands/core.py\", line 2685, in lpop\n    return self.execute_command(\"LPOP\", name, count)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1258, in execute_command\n    return conn.retry.call_with_retry(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/retry.py\", line 46, in call_with_retry\n    return do()\n           ^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1259, in <lambda>\n    lambda: self._send_command_parse_response(\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1235, in _send_command_parse_response\n    return self.parse_response(conn, command_name, **options)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/client.py\", line 1275, in parse_response\n    response = connection.read_response()\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/redis/connection.py\", line 882, in read_response\n    raise response\nredis.exceptions.ResponseError: wrong number of arguments for 'lpop' command\n",
        "args": "()",
        "kwargs": "{'repoid': <REDACTED>, 'commitid': '<REDACTED>', 'report_type': 'coverage', 'report_code': None, 'debug': False, 'rebuild': False}",
        "description": "raised unexpected",
        "internal": false
    },
    "task_name": "???",
    "task_id": "???",
    "logger.name": "celery.app.trace",
    "logger.thread_name": "MainThread",
    "level": "ERROR"
}

[drazisil-codecov]

That's... an error. Give me a few, please. What version of Redis are you running, by the way?

[drazisil-codecov]

The docs say it needs to be later then 6.2.0

https://redis.io/docs/latest/commands/lpop/

Starting with Redis version 6.2.0: Added the count argument.

https://github.com/codecov/worker/commit/276c39f62368558575ef6215b06472ca9a6dcde2#diff-7ea195247ce946074a2441630db2be4bda78feed427fc0a1f1bd170cfc84a7a7

[chaseconey]

@drazisil-codecov I think that was it! We were pinned to 6.0-alpine, it seems.

Thank you so much for digging into this with me! There is almost certainly no way I would have figured this out by myself. 🙏

[drazisil-codecov]

Happy to help. :)