search

MolSSI / QCFractal

A distributed compute and database platform for quantum chemistry.
https://molssi.github.io/QCFractal/
BSD 3-Clause "New" or "Revised" License
144 stars 47 forks source link

QCFractal 0.15.6 šŸ™…ā€ā™€ļø QCElemental 0.23.0 ? #690

Closed lilyminium closed 2 years ago

lilyminium commented 2 years ago

Describe the bug

I have not looked deeply into this, but for some reason the numpy arrays in ResultRecord are not getting JSON serialized with qcelemental 0.23.0. Works fine in 0.22.0.

Error message

[I 211009 13:56:56 handlers:218] QueueManager: Received completed tasks from FractalServer-hpc3-14-07-4a537e32-66cf-4d2b-b422-e9fc014052dd.
[I 211009 13:56:56 handlers:219]               Task ids: 1
[I 211009 13:56:56 handlers:289] QueueManager: Found 1 complete tasks (1 successful, 0 failed).
[E 211009 13:56:56 web:1789] Uncaught exception POST /queue_manager (::1)
    HTTPServerRequest(protocol='http', host='localhost:40721', method='POST', uri='/queue_manager', version='HTTP/1.1', remote_ip='::1')
    Traceback (most recent call last):
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1204, in _execute_context
        context = constructor(dialect, self, conn, *args)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 873, in _init_compiled
        param = dict(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 876, in <genexpr>
        processors[key](compiled_params[key])
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 2438, in process
        serialized = json_serializer(value)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/__init__.py", line 231, in dumps
        return _default_encoder.encode(obj)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/encoder.py", line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/encoder.py", line 257, in iterencode
        return _iterencode(o, 0)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/encoder.py", line 179, in default
        raise TypeError(f'Object of type {o.__class__.__name__} '
    TypeError: Object of type ndarray is not JSON serializable

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/tornado/web.py", line 1702, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/qcfractal/queue/handlers.py", line 343, in post
        success, error = self.insert_complete_tasks(self.storage, body, self.logger)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/qcfractal/queue/handlers.py", line 299, in insert_complete_tasks
        com = procedure_parser.handle_completed_output(v)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/qcfractal/procedures/optimization.py", line 267, in handle_completed_output
        ret = self.storage.add_results(list(results.values()))
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/qcfractal/storage_sockets/sqlalchemy_socket.py", line 1248, in add_results
        session.commit()
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
        self.transaction.commit()
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
        self._prepare_impl()
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
        self.session.flush()
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
        self._flush(objects)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
        transaction.rollback(_capture_exception=True)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
        compat.raise_(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
        flush_context.execute()
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
        rec.execute(self)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
        persistence.save_obj(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
        _emit_insert_statements(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1083, in _emit_insert_statements
        c = cached_connections[connection].execute(statement, multiparams)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
        return meth(self, multiparams, params)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
        ret = self._execute_context(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1206, in _execute_context
        self._handle_dbapi_exception(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
        util.raise_(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
        raise exception
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1204, in _execute_context
        context = constructor(dialect, self, conn, *args)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 873, in _init_compiled
        param = dict(
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 876, in <genexpr>
        processors[key](compiled_params[key])
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 2438, in process
        serialized = json_serializer(value)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/__init__.py", line 231, in dumps
        return _default_encoder.encode(obj)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/encoder.py", line 199, in encode
        chunks = self.iterencode(o, _one_shot=True)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/encoder.py", line 257, in iterencode
        return _iterencode(o, 0)
      File "/data/homezvol0/lilyw7/miniconda3/envs/psiresp-3.8/lib/python3.8/json/encoder.py", line 179, in default
        raise TypeError(f'Object of type {o.__class__.__name__} '
    sqlalchemy.exc.StatementError: (builtins.TypeError) Object of type ndarray is not JSON serializable
    [SQL: INSERT INTO result (id, program, driver, method, basis, molecule, keywords, return_result, properties, wavefunction, wavefunction_data_id) VALUES (%(id)s, %(program)s, %(driver)s, %(method)s, %(basis)s, %(molecule)s, %(keywords)s, %(return_result)s, %(properties)s, %(wavefunction)s, %(wavefunction_data_id)s)]
    [parameters: [{'properties': {'calcinfo_nbasis': 13, 'calcinfo_nmo': 13, 'calcinfo_nalpha': 5, 'calcinfo_nbeta': 5, 'calcinfo_natom': 3, 'nuclear_repulsion_energy' ... (5355 characters truncated) ... +00, -2.7167876974210081e-06,
            -1.9662543738613314e-06]]), 'program': 'psi4', 'molecule': '2', 'keywords': None, 'wavefunction_data_id': None}]]
[E 211009 13:56:56 web:2239] 500 POST /queue_manager (::1) 70.81ms
[W 211009 13:56:56 managers:584] Post complete tasks was not successful. Attempting again on next update.

To Reproduce

The example procedure is just copied from the docs. The error also arises for driver='energy', etc.

#!/usr/bin/env python

from qcfractal import FractalSnowflakeHandler, FractalSnowflake, FractalServer
from qcfractal import interface as ptl

if __name__ == "__main__":

    with FractalSnowflake(logging=True) as server:
        client = ptl.FractalClient(server, verify=False)

        mol = ptl.Molecule.from_data("""
        O 0 0 0
        H 0 0 2
        H 0 2 0
        units bohr
        """)

        spec = {
            "keywords": None,
            "qc_spec": {
                "driver": "gradient",
                "method": "b3lyp",
                "basis": "6-31g",
                "program": "psi4"
            },
        }

        # Ask the server to compute a new computation
        response = client.add_procedure("optimization", "geometric", spec, [mol])

        print(server.await_results())

Expected behavior

Either that it works, or the packages are set up that these versions conflict.

Additional context

I did not fiddle around with FractalServer (vs. FractalSnowflake) settings so I'm not sure if I just set it up incorrectly, but this was quite difficult to debug. The only log output I could see was this fairly abstruse message:

Post complete tasks was not successful. Attempting again on next update.

And then something about it being stale.

I'm not sure why, since FractalSnowflake(logging=True) showed very useful output and seems to just wrap FractalServer; if there are settings I should enable for all the output to show up in the log file, do please let me know.

Environment (bug)

openff-qcsubmit           0.2.3              pyhd8ed1ab_0    conda-forge
qcelemental               0.23.0             pyhd8ed1ab_0    conda-forge
qcengine                  0.19.0             pyhd8ed1ab_0    conda-forge
qcfractal                 0.15.6           py38h578d9bd_0    conda-forge
qcfractal-core            0.15.6           py38h578d9bd_0    conda-forge
qcportal                  0.15.6             pyhd8ed1ab_0    conda-forge
bennybp commented 2 years ago

This has been fixed in #686. The next release shouldn't have this problem. Sorry!

lilyminium commented 2 years ago

I see, thank you @bennybp :-) do you know when the next release might be coming out?

bennybp commented 2 years ago

Tentatively tomorrow (Oct 14) :crossed_fingers: