QCSubmit CI is segfaulting

[j-wags]

Segfaulting test seems to be openff/qcsubmit/_tests/test_datasets.py::test_dataset_to_pdf_no_torsions[Openeye by default.]

https://github.com/openforcefield/openff-qcsubmit/actions/runs/8697856360/job/23853864399#step:7:193

OE hasn't pushed a new package recently (last main label package was 3 months ago)

[mattwthompson]

Having dealt with similar (-ly cryptic) failures in the past, I'd try to parry down which modules that are bundled with OpenEye and might have had releases recently. (I quickly checked a few of the typical suspects and nothing was obvious.)

[j-wags]

5 days ago there were passing CI runs that used (QCF=0.54.1 (pip installed) and OE=2023.2.3) and (QCF=0.53 and OE=2023.2.3), but then things started failing after that.

[mattwthompson]

If it helps narrow it down, it's likely in the OEDepict import, so maybe something changed with a visualization-adjacent upstream. (It'd be much messier if this is not related to a visualization tool.)

[j-wags]

AFAICT the openeye conda packages vendor everything. They only have python and pytest as deps. Am I overlooking a source of outside code?

[mattwthompson]

Unfortunately, whatever magic happens in their build systems can cause mysterious incompatibilities with non-Python dependencies: https://github.com/conda-forge/qcfractal-feedstock/issues/43

[mattwthompson]

Nothing sticks out with naive inspection of the changes in imported Python modules:

In [1]: import sys

In [2]: mod1 = set(sys.modules.keys())

In [3]: from openeye import oechem

In [4]: mod2 = set(sys.modules.keys())

In [5]: from openeye import oedepict

In [6]: mod3 = set(sys.modules.keys())

In [7]: mod1.difference(mod2)
Out[7]: set()

In [8]: mod2.difference(mod1)
Out[8]:
{'_oechem4003',
 'importlib.abc',
 'importlib.resources',
 'importlib.resources._adapters',
 'importlib.resources._common',
 'importlib.resources._legacy',
 'importlib.resources.abc',
 'libs',
 'libs.openeye_platform',
 'oechem',
 'openeye',
 'openeye.oechem',
 'swig_runtime_data4'}

In [9]: mod3.difference(mod1)
Out[9]:
{'_oechem4003',
 '_oedepict2533',
 'importlib.abc',
 'importlib.resources',
 'importlib.resources._adapters',
 'importlib.resources._common',
 'importlib.resources._legacy',
 'importlib.resources.abc',
 'libs',
 'libs.openeye_platform',
 'oechem',
 'oedepict',
 'openeye',
 'openeye.oechem',
 'openeye.oedepict',
 'swig_runtime_data4'}

In [10]: mod3.difference(mod2)
Out[10]: {'_oedepict2533', 'oedepict', 'openeye.oedepict'}

[j-wags]

I can reproduce the error locally and downgrading to openff-toolkit 0.15.2 fixes it 😿 Time to do some digging.

[mattwthompson]

I bisected it to https://github.com/openforcefield/openff-toolkit/pull/1798

This commit is fine: https://github.com/openforcefield/openff-toolkit/commit/b739698d4187943ac8e31c59a0866b60247c6ed5

I can look into this more closely tomorrow if you don't get to the bottom of it today - sorry for introducing a regression.

[j-wags]

Hah, no worries. Great sleuthing. It's a weird enough error that normal CI didn't catch it. Looking into it, I'll let you know what I find.

[mattwthompson]

My suggestion earlier wasn't the right path to go down, but in case we go down it once again, here's a better starting point for getting what might be a part of compiled OpenEye code: https://docs.eyesopen.com/applications/license_3rdparty.html

[j-wags]

I think the actual problem might be that qcsubmit directly sets offmol._conformers = [], but OFFTK might expect "no conformers" to only be represented by _conformers = None. Testing that out now...

[j-wags]

Resolved by #277