Failed to collect and upload profile: dictionary changed size during iteration

GoogleCloudPlatform / cloud-profiler-python

Stackdriver Profiler Python agent is a tool that continuously gathers CPU usage information from Python applications

Apache License 2.0

28 stars 23 forks source link

Failed to collect and upload profile: dictionary changed size during iteration #68

Closed wasdee closed 3 years ago

wasdee commented 3 years ago

this is the bug raised in production capture by sentry.

"Failed to collect and upload profile: dictionary changed size during iteration"

CPython, 3.8.2(3.8.2 (default, Apr 23 2020, 14:22:33) [GCC 8.3.0])

wasdee commented 3 years ago

It don't know how to reproduce it exactly.

aalexand commented 3 years ago

@CircleOnCircles I assume there wasn't a stack?

wasdee commented 3 years ago

Could you explain more? What is the stack?

wasdee commented 3 years ago

this is a fastapi application.

tool.poetry.dependencies]
python = "^3.7"
fastapi = "^0.61.0"
httpx = { extras = ["http2"], version = "^0.16.1" }
uvicorn = "^0.11.8"
loguru = "^0.5.1"
pydantic = {extras = ["email", "dotenv"], version = "^1.7.3"}
pybase62 = "^0.4.3"
arrow = "^0.15.8"
autoname = "^0.1.1"
sentry-sdk = "^0.16.5"
google-api-python-client = "^1.8.4"
google-auth-httplib2 = "^0.0.4"
google-auth-oauthlib = "^0.4.1"
grpcio = "^1.31.0"
motor = "^2.3.0"
dnspython = "^2.0.0"
line-bot-sdk = "^1.18.0"
tenacity = "^6.3.1"
notifiers = "^1.2.1"
omise = "^0.9.0"
python-multipart = "^0.0.5"
furl = "^2.1.0"
imagekitio = "^2.2.5"
jingdi = "^0.1.0"
pyzbar-x = "^0.2.1"
Pillow = "^8.1.0"
fastapi-utils = "^0.2.1"
google-python-cloud-debugger = "^2.15"
tomlkit = "^0.7.0"
locate = "^0.0.1"
parse-with-dot-access = "^1.18.0"

wyk9787 commented 3 years ago

Could you explain more? What is the stack?

I think what @aalexand meant was "Was there a stack trace being printed when your program threw out 'dictionary changed size during iteration'"? If so, could you post it here?

wasdee commented 3 years ago

What I see in the GCP log explorer. there is no stack trace.

wyk9787 commented 3 years ago

Thanks for the screenshot. To better understand the issue, how often do you see this error message in your log?

wasdee commented 3 years ago

like a few times a week

wyk9787 commented 3 years ago

Thanks! I have applied a patch to the Python agent to print out the stack trace on that code path. Could you update your agent to version 2.0.5: https://pypi.org/project/google-cloud-profiler/ and post this error log again once it happens with the new version?

wasdee commented 3 years ago

sure, let close for now, i will reopen it when i faced the bug again