OpenFF Gen 2 Optimization (v1.0) series

jmaat commented 4 years ago

These jobs had errors or were incomplete in the following data sets -

'OpenFF Gen 2 Opt Set 4 eMolecules Discrepancy': IDs for 5 Errors: 18435692 18436178 18436876 18436926 18437240

'OpenFF Gen 2 Opt Set 2 Coverage': IDs for 5 Errors: 18433779 18433828 18434085 18434088 18434121

IDs for 16 INCOMPLETE: 18433929 18433930 18433937 18433938 18433944 18433945 18433984 18433988 18433998 18434004 18434006 18434010 18434013 18434015 18434069 18434075

dotsdl commented 4 years ago

Update!

OpenFF Gen 2 Opt Set 4 eMolecules Discrepancy

Current errored OptimizationRecords: 18436178 18437465 18437470 18438079 18438248

The errors for this dataset all appear to be geometric.errors.GeomOptNotConvergedErrors, with output like the following:

geomeTRIC run_json error:
Traceback (most recent call last):
  File "/opt/conda/envs/qcfractal/lib/python3.6/site-packages/geometric/run_json.py", line 214, in geometric_run_json
    geometric.optimize.Optimize(coords, M, IC, engine, None, params)
  File "/opt/conda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1331, in Optimize
    return optimizer.optimizeGeometry()
  File "/opt/conda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1301, in optimizeGeometry
    raise GeomOptNotConvergedError("Optimizer.optimizeGeometry() failed to converge.")
geometric.errors.GeomOptNotConvergedError: Optimizer.optimizeGeometry() failed to converge.

@jmaat are there changes you would like to make for these optimizations to achieve converged results?

OpenFF Gen 2 Opt Set 2 Coverage

I have resubmitted 10 of the errored OptimizationRecords for this dataset; these 10 showed only an unknown error from QCEngine. The ids for these are: 18433929 18433930 18433937 18433944 18433984 18433988 18434010 18434013 18434015 18434075

There are 4 additional errored records: 18433828 18434085 18434088 18434121

Each of these gives a convergence error as below:

geomeTRIC run_json error:
Traceback (most recent call last):
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/run_json.py", line 214, in geometric_run_json
    geometric.optimize.Optimize(coords, M, IC, engine, None, params)
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1331, in Optimize
    return optimizer.optimizeGeometry()
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1298, in optimizeGeometry
    self.calcEnergyForce()
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/optimize.py", line 1002, in calcEnergyForce
    spcalc = self.engine.calc(self.X, self.dirname)
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/engine.py", line 873, in calc
    return self.calc_new(coords, dirname)
  File "/home/chodera/miniconda/envs/qcfractal/lib/python3.6/site-packages/geometric/engine.py", line 865, in calc_new
    raise QCEngineAPIEngineError("QCEngineAPI computation did not execute correctly. Message: " + ret["error"]["error_message"])
geometric.errors.QCEngineAPIEngineError: QCEngineAPI computation did not execute correctly. Message: QCEngine Unknown Error: Traceback (most recent call last):
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/json_wrapper.py", line 217, in run_json
    json_data = run_json_qcschema(copy.deepcopy(json_data), clean)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/json_wrapper.py", line 312, in run_json_qcschema
    val, wfn = methods_dict_[json_data["driver"]](method, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/driver.py", line 691, in gradient
    wfn = procedures['gradient'][lowername](lowername, molecule=molecule, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 2064, in run_scf_gradient
    ref_wfn = run_scf(name, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 2002, in run_scf
    scf_wfn = scf_helper(name, post_scf=False, **kwargs)
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 1363, in scf_helper
    e_scf = scf_wfn.compute_energy()
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 94, in scf_compute_energy
    raise e
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 87, in scf_compute_energy
    self.iterations()
  File "/home/chodera/miniconda/envs/qcfractal/lib//python3.6/site-packages/psi4/driver/procrouting/scf_proc/scf_iterator.py", line 447, in scf_iterate
    raise SCFConvergenceError("""SCF iterations""", self.iteration_, self, Ediff, Dnorm)
psi4.driver.p4util.exceptions.SCFConvergenceError: Could not converge SCF iterations in 200 iterations.

Are there changes you would like to make for these optimizations to achieve converged results?

dotsdl commented 4 years ago

All optimization tasks with state 'ERROR' have been restarted for the Gen 2 Optimization series of collections. All torsiondrive services with state 'ERROR' have also been restarted.

~~We still see many cases of 'INCOMPLETE' optimization tasks. All of these appear to be coming from Lilac. @jchodera, can you verify that the manager(s) on Lilac have been shut down?~~

dotsdl commented 4 years ago

Correction: 'INCOMPLETE's for this dataset appear to be few, and they are currently coming from PRP. It's the Gen 2 Torsion sets that feature many INCOMPLETEs that appear to be from Lilac.

dotsdl commented 4 years ago

OpenFF Gen 2 Opt Set 1 Roche

No unfinished Optimizations, but 4 NaNs present:

Cc1ccccc1OC2CCCCC2-0a    NaN
Cc1ccccc1OC2CCCCC2-1a    NaN
Cc1ccccc1OC2CCCCC2-2a    NaN
Cc1ccccc1OC2CCCCC2-3a    NaN

OpenFF Gen 2 Opt Set 2 Coverage

14 INCOMPLETE Optimizations; each 1 week old since restart from ERROR last week

last known running on PRP, UCI, MolSSI resources

OpenFF Gen 2 Opt Set 3 Pfizer Discrepancy

All Optimizations COMPLETE!

OpenFF Gen 2 Opt Set 4 eMolecules Discrepancy

14 INCOMPLETE Optimizations; each 1 week old since restart from ERROR last week

last known running on PRP, UCI

OpenFF Gen 2 Opt Set 5 Bayer

26 INCOMPLETE Optimizations; all except one a week old since restart from ERROR last week

last known running on PRP, UCI, Galileo

openforcefield / qca-dataset-submission