[ ] Remove DD_TRACE_PYMONGO_ENABLED=false from configuration
Important note: Follow roll-out of proving out on Stage, then moving to Edge, then moving to Prod.
When we tried instrumenting edxapp with Datadog APM (via ddtrace-run) it caused some pymongo calls to break. This was with ddtrace-run 2.7.3 and pymongo 3.13.0. Using DD_TRACE_PYMONGO_ENABLED=false prevents the errors.
Versions 2.7.9 and 2.8.2 contain a fix; I've verified the fix in 2.7.9 in devstack.
Stack trace:
Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/trace_utils.py", line 334, in wrapper
return func(mod, pin, wrapped, instance, args, kwargs)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/django/patch.py", line 301, in wrapped
return func(*args, **kwargs)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 133, in __call__
response = self.process_request(request)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/trace_utils.py", line 334, in wrapper
return func(mod, pin, wrapped, instance, args, kwargs)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/django/patch.py", line 301, in wrapped
return func(*args, **kwargs)
File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/contentserver/middleware.py", line 84, in process_request
content = self.load_asset_from_location(loc)
File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/contentserver/middleware.py", line 292, in load_asset_from_location
content = AssetManager.find(location, as_stream=True)
File "/edx/app/edxapp/edx-platform/xmodule/assetstore/assetmgr.py", line 33, in find
return contentstore().find(asset_key, throw_on_not_found, as_stream)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/mongodb_proxy.py", line 53, in wrapper
return func(*args, **kwargs)
File "/edx/app/edxapp/edx-platform/xmodule/contentstore/mongo.py", line 134, in find
fp = self.fs.get(content_id)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/gridfs/__init__.py", line 153, in get
gout._ensure_file()
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/gridfs/grid_file.py", line 482, in _ensure_file
self._file = self.__files.find_one({"_id": self.__file_id}, session=self._session)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/collection.py", line 1491, in find_one
for result in cursor.limit(-1):
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
if len(self.__data) or self._refresh():
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/cursor.py", line 1195, in _refresh
self.__send_message(q)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/cursor.py", line 1078, in __send_message
response = client._run_operation(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1515, in _run_operation
return self._retryable_read(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1617, in _retryable_read
return func(session, server, sock_info, secondary_ok)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1511, in _cmd
return server.run_operation(
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/pymongo/client.py", line 172, in run_operation
result = self.__wrapped__.run_operation(sock_info, operation, *args, **kwargs)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/server.py", line 94, in run_operation
use_cmd = operation.use_command(sock_info)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/message.py", line 341, in use_command
sock_info.validate_session(self.client, self.session)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/pool.py", line 892, in validate_session
raise InvalidOperation(
pymongo.errors.InvalidOperation: Can only use session with the MongoClient that started it
A/C:
Important note: Follow roll-out of proving out on Stage, then moving to Edge, then moving to Prod.
When we tried instrumenting edxapp with Datadog APM (via
ddtrace-run
) it caused some pymongo calls to break. This was with ddtrace-run 2.7.3 and pymongo 3.13.0. UsingDD_TRACE_PYMONGO_ENABLED=false
prevents the errors.Versions 2.7.9 and 2.8.2 contain a fix; I've verified the fix in 2.7.9 in devstack.
Stack trace:
Private support ticket: https://help.datadoghq.com/hc/en-us/requests/1600528