search

man-group / notebooker

Productionise & schedule your Jupyter Notebooks as easily as you wrote them.
GNU Affero General Public License v3.0
852 stars 79 forks source link

pymongo.errors.DocumentTooLarge: command document too large error #45

Closed vinopm closed 2 years ago

vinopm commented 3 years ago

Hi,

I am seeing this error occur when I run a notebook in Notebooker. The notebook runs fine when run locally.

Traceback (most recent call last):
  File "/default-medusa-venv/lib/python3.6/site-packages/notebooker-1!202105050846+n47d39d7-py3.6.egg/notebooker/execute_notebook.py", line 184, in run_report
    result_serializer.save_check_result(result)
  File "/default-medusa-venv/lib/python3.6/site-packages/notebooker-1!202105050846+n47d39d7-py3.6.egg/notebooker/serialization/mongo.py", line 124, in save_check_result
    self._save_to_db(notebook_result)
  File "/default-medusa-venv/lib/python3.6/site-packages/notebooker-1!202105050846+n47d39d7-py3.6.egg/notebooker/serialization/mongo.py", line 73, in _save_to_db
    self._save_raw_to_db(out_data)
  File "/default-medusa-venv/lib/python3.6/site-packages/notebooker-1!202105050846+n47d39d7-py3.6.egg/notebooker/serialization/mongo.py", line 62, in _save_raw_to_db
    self.library.replace_one({"_id": existing["_id"]}, out_data)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/collection.py", line 907, in replace_one
    collation=collation, session=session),
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/collection.py", line 835, in _update_retryable
    _update, session)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/mongo_client.py", line 1099, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/mongo_client.py", line 1076, in _retry_with_session
    return func(session, sock_info, retryable)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/collection.py", line 831, in _update
    retryable_write=retryable_write)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/collection.py", line 796, in _update
    retryable_write=retryable_write).copy()
  File "/default-medusa-venv/lib/python3.6/site-packages/man.core-1!202105071906+ndc84b65-py3.6-linux-x86_64.egg/ahl/mongo/decorators.py", line 247, in _wrapped
    raise e
  File "/default-medusa-venv/lib/python3.6/site-packages/man.core-1!202105071906+ndc84b65-py3.6-linux-x86_64.egg/ahl/mongo/decorators.py", line 241, in _wrapped
    return orig_method(self, *args, **kwargs)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/pool.py", line 501, in command
    self._raise_connection_failure(error)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/pool.py", line 649, in _raise_connection_failure
    raise error
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/pool.py", line 496, in command
    collation=collation)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/network.py", line 107, in command
    name, size, max_bson_size + message._COMMAND_OVERHEAD)
  File "/default-medusa-venv/lib/python3.6/site-packages/pymongo-3.6.0-py3.6-linux-x86_64.egg/pymongo/message.py", line 709, in _raise_document_too_large
    raise DocumentTooLarge("command document too large")
pymongo.errors.DocumentTooLarge: command document too large

The notebook result itself is maybe too large to be stored in mongo. Please address this issue.

Thank you