search

materialsproject / maggma

MongoDB aggregation machine
https://materialsproject.github.io/maggma/
Other
38 stars 31 forks source link

GridFSStore collection has no database attribute #733

Open pmiam opened 1 year ago

pmiam commented 1 year ago

Any atomate2 TaskDocument larger than the document limit cannot spill into gridfs additional stores successfully because the gridfs collection has no database property

Traceback (truncated):

Traceback (most recent call last):
  File "/home/pmangana/src/atomate2/a2a.py", line 124, in update_store
    s.update(data, key="uuid")
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/jobflow/core/store.py", line 325, in update
    self.docs_store.update(dict_docs, key=key)
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/maggma/stores/mongolike.py", line 487, in update
    raise e
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/maggma/stores/mongolike.py", line 475, in update
    self._collection.bulk_write(requests, ordered=False)
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/pymongo/_csot.py", line 105, in csot_wrapper
    return func(self, *args, **kwargs)
....
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/pymongo/message.py", line 1162, in _do_batched_op_msg
    return _batched_op_msg(operation, command, docs, ack, opts, ctx)
pymongo.errors.DocumentTooLarge: update command document too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pmangana/src/atomate2/a2a.py", line 97, in worker
    update_store(store=store, data=data)
  File "/home/pmangana/src/atomate2/a2a.py", line 123, in update_store
    with store as s:
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/maggma/core/store.py", line 351, in __exit__
    self.close()
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/jobflow/core/store.py", line 116, in close
    additional_store.close()
  File "/depot/amannodi/apps/bigconda/envs/a2/lib/python3.10/site-packages/maggma/stores/gridfs.py", line 450, in close
    self._collection.database.client.close()
AttributeError: 'GridFS' object has no attribute 'database'

https://github.com/materialsproject/maggma/blob/0680fa9508fe29e8e84963d54662a6c9e3e94d55/src/maggma/stores/gridfs.py#L450

versions: db> db.version() 6.0.2

I deployed the official docker container on an enterprise server.

conda list (truncated) jobflow 0.1.7 pyhd8ed1ab_0 conda-forge maggma 0.49.4 pyhd8ed1ab_0 conda-forge atomate2 0.0.1.post912+g1fd2322 pypi_0 pypi (my fork main) pymongo 4.2.0 pypi_0 pypi

In context, I've written a very simple script to try backfilling a database with legacy vasp runs, obviously there's some shortcomings to this, but this cannot be one of them. Very new to all this. Advice appreciated.

munrojm commented 1 year ago

Hi @PanayotisManganaris. I will take a look at this.