Psi4 tests failing

[mattwthompson]

https://github.com/openforcefield/openff-qcsubmit/actions/runs/8225781166/job/22491290735

https://github.com/openforcefield/openff-qcsubmit/actions/runs/8234860096/job/22517602570

[mattwthompson]

Looks like production is using 1.8.2: https://github.com/openforcefield/qca-dataset-submission/blob/ac79bc1fcc84b16db5e74fb3a0d3b9639c1b389e/devtools/prod-envs/qcarchive-worker-openff-psi4.yaml#L12

[mattwthompson]

https://github.com/psi4/psi4/releases/tag/v1.9.1

[j-wags]

Unable to reproduce natively on mac. The following are docker instructions to spin up a linux container and reproduce the error:

docker run -it mambaorg/micromamba

in another terminal

docker cp ~/oe_license.txt <container_id>:/home/mambauser

in container

micromamba install -c conda-forge git
git clone https://github.com/openforcefield/openff-qcsubmit.git
micromamba env create -f openff-qcsubmit/devtools/conda-envs/psi4.yaml -n qcsubmit-dev
cd openff-qcsubmit/
pip install -e .
OE_LICENSE=/home/mambauser/oe_license.txt pytest "openff/qcsubmit/_tests/test_submissions.py::test_basic_submissions_single_spec[PSI4 hf 3-21g energy]" 

Full output:

=================================== FAILURES ===================================
___________ test_basic_submissions_single_spec[PSI4 hf 3-21g energy] ___________

fulltest_client = PortalClient(server_name='QCFractal Server', address='http://127.0.0.1:39423/', username='None')
specification = ({'basis': '3-21g', 'method': 'hf', 'program': 'psi4'}, 'energy')

    @pytest.mark.parametrize(
        "specification",
        [
            pytest.param(
                ({"method": "hf", "basis": "3-21g", "program": "psi4"}, "energy"),
                id="PSI4 hf 3-21g energy",
            ),
            pytest.param(
                (
                    {
                        "method": "openff-2.1.0",
                        "basis": "smirnoff",
                        "program": "openmm",
                    },
                    "energy",
                ),
                id="SMIRNOFF openff-2.1.0 energy",
            ),
            pytest.param(
                ({"method": "uff", "basis": None, "program": "rdkit"}, "gradient"),
                id="RDKit UFF gradient",
            ),
        ],
    )
    def test_basic_submissions_single_spec(fulltest_client, specification):
        """Test submitting a basic dataset to a snowflake server."""

        client = fulltest_client

        qc_spec, driver = specification

        program = qc_spec["program"]
        if not has_program(program):
            pytest.skip(f"Program '{program}' not found.")

        molecules = Molecule.from_file(get_data("butane_conformers.pdb"), "pdb")

        factory = BasicDatasetFactory(driver=driver)
        factory.add_qc_spec(
            **qc_spec,
            spec_name="default",
            spec_description="testing the single points",
            overwrite=True,
        )

        dataset = factory.create_dataset(
            dataset_name=f"Test single points info {program}, {driver}",
            molecules=molecules,
            description="Test basics dataset",
            tagline="Testing single point datasets",
        )

        # force a metadata validation error
        dataset.metadata.long_description = None

        with pytest.raises(DatasetInputError):
            dataset.submit(client=client)

        # re-add the description so we can submit the data
        dataset.metadata.long_description = "Test basics dataset"

        # now submit again
        dataset.submit(client=client)
>       await_results(client)

openff/qcsubmit/_tests/test_submissions.py:186: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = PortalClient(server_name='QCFractal Server', address='http://127.0.0.1:39423/', username='None')
timeout = 120
check_fn = <function PortalClient.get_singlepoints at 0x401a973060>, ids = [1]

    def await_results(client, timeout=120, check_fn=PortalClient.get_singlepoints, ids=[1]):
        import time

        for i in range(timeout):
            time.sleep(1)
            recs = check_fn(client, ids)
            from pprint import pprint

            finished = 0
            from qcportal.record_models import OutputTypeEnum

            for rec in recs:
                print(rec.status)
                if rec.status == RecordStatusEnum.error:
                    print("stderr", rec._get_output(OutputTypeEnum.stderr))
                    print("stdout", rec._get_output(OutputTypeEnum.stdout))
                    print("error: ")
                    pprint(rec._get_output(OutputTypeEnum.error))
>                   raise RuntimeError(f"calculation failed: {rec}")
E                   RuntimeError: calculation failed: <SinglepointRecord id=1 status=RecordStatusEnum.error>

openff/qcsubmit/_tests/test_submissions.py:61: RuntimeError
----------------------------- Captured stdout call -----------------------------
RecordStatusEnum.waiting
RecordStatusEnum.waiting
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.running
RecordStatusEnum.error
stderr None
stdout None
error: 
{'error_message': 'QCEngine Random Error: Unknown error, error message is not '
                  'found, possible segmentation fault!',
 'error_type': 'random_error',
 'extras': None}
----------------------------- Captured stderr call -----------------------------
Deduplication                 : 100%|█████████████| 7/7 [00:00<00:00, 19.25it/s]
Preparation                   : 100%|█████████████| 1/1 [00:00<00:00,  1.86it/s]
=========================== short test summary info ============================
FAILED openff/qcsubmit/_tests/test_submissions.py::test_basic_submissions_single_spec[PSI4 hf 3-21g energy] - RuntimeError: calculation failed: <SinglepointRecord id=1 status=RecordStat...
============================== 1 failed in 58.44s ==============================

[j-wags]

Tests pass when I downgrade psi4 and libint. So this might be an echo of https://github.com/psi4/psi4/issues/3144

  - libint      2.9.0  h9bbc0ff_0       conda-forge                      Cached
  + libint  2.7.3dev1  h95a0634_0       conda-forge/label/libint_dev       65MB
  - psi4          1.9  py311hed31b30_1  conda-forge                      Cached
  + psi4        1.8.2  py311hed31b30_2  conda-forge                        26MB

[j-wags]

I see - psi4 1.9.1 build 1 works locally, but not 1.9 (and probably not 1.9.0). We'd been requiring psi4=1.9 previously and the envs were resolving to psi4 1.9 exactly (NOT 1.9.0 or 1.9.1). I'm seeing that the meaning of a single equals sign isn't discussed in the package match specification. So I'm gonna try resolving this by up-pinning to psi4 >=1.9.1.

[mattwthompson]

It's described as a "fuzzy constraint" in the CLI examples, but nowhere else. I gather that =1.9 is equivalent to ==1.9.*, whereas ==1.9 is closer to ==1.9.0.0.0 and so on. = is not in PEP 440 nor listed as an additional symbol.

https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#id4

Probably ~=1.9 would be best here, though it's not a huge difference.