search

Quantum-Accelerators / quacc

quacc is a flexible platform for computational materials science and quantum chemistry that is built for the big data era.
https://quantum-accelerators.github.io/quacc/
BSD 3-Clause "New" or "Revised" License
176 stars 48 forks source link

Error with running Q-Chem job #1185

Closed kumaranu closed 11 months ago

kumaranu commented 11 months ago

Details about the quacc environment

What is the issue?

I am getting the following error:

2023-11-09 14:18:27,692 INFO Started executing jobs locally
2023-11-09 14:18:27,694 INFO Starting job - ts_job (b68f6e78-e3be-4bb7-9320-b476c43e5b36)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Traceback (most recent call last):
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/custodian.py", line 9, in <module>
    from quacc import SETTINGS
  File "/global/home/users/kumaranu/quacc/src/quacc/__init__.py", line 10, in <module>
    from quacc.settings import QuaccSettings
  File "/global/home/users/kumaranu/quacc/src/quacc/settings.py", line 15, in <module>
    from quacc.calculators.vasp import presets as vasp_presets
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/vasp/__init__.py", line 3, in <module>
    from quacc.calculators.vasp.vasp import Vasp
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/vasp/vasp.py", line 14, in <module>
    from quacc.calculators.vasp import custodian
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/vasp/custodian.py", line 8, in <module>
    from custodian import Custodian
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/custodian.py", line 9, in <module>
    from quacc import SETTINGS
ImportError: cannot import name 'SETTINGS' from partially initialized module 'quacc' (most likely due to a circular import) (/global/home/users/kumaranu/quacc/src/quacc/__init__.py)
2023-11-09 14:18:35,872 INFO ts_job failed with exception:
Traceback (most recent call last):
  File "/global/home/users/kumaranu/quacc/src/quacc/runners/ase.py", line 180, in run_opt
    dyn.run(fmax=fmax, steps=max_steps, **run_kwargs)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 338, in run
    return Dynamics.run(self)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 218, in run
    for converged in Dynamics.irun(self):
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 186, in irun
    self.optimizable.get_forces()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 30, in get_forces
    return self.atoms.get_forces()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/atoms.py", line 791, in get_forces
    forces = self._calc.get_forces(self)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/abc.py", line 30, in get_forces
    return self.get_property('forces', atoms)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 503, in get_property
    self.calculate(atoms, [name], system_changes)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 906, in calculate
    self.execute()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 936, in execute
    raise CalculationFailed(msg)
ase.calculators.calculator.CalculationFailed: Calculator "qchem" failed with command "python /global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/custodian.py 32" failed in /global/home/users/kumaranu/.scratch/quacc-tmp-2023-11-09-22-18-27-695511-ezq9lz_9 with error code 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/jobflow/managers/local.py", line 102, in _run_job
    response = job.run(store=store)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/jobflow/core/job.py", line 583, in run
    response = function(*self.function_args, **self.function_kwargs)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/monty/dev.py", line 100, in decorated
    return _callable(*args, **kwargs)
  File "/global/home/users/kumaranu/quacc/src/quacc/recipes/qchem/ts.py", line 144, in ts_job
    return _base_opt_job(
  File "/global/home/users/kumaranu/quacc/src/quacc/recipes/qchem/core.py", line 554, in _base_opt_job
    dyn = run_opt(atoms, copy_files=copy_files, **opt_flags)
  File "/global/home/users/kumaranu/quacc/src/quacc/runners/ase.py", line 183, in run_opt
    raise RuntimeError(msg) from err
RuntimeError: Calculation failed. Read the full traceback above. If needed, the logfiles are at /global/home/users/kumaranu/.scratch/quacc-tmp-2023-11-09-22-18-27-695511-ezq9lz_9

2023-11-09 14:18:35,874 INFO Finished executing jobs locally
Traceback (most recent call last):
  File "/global/home/users/kumaranu/Documents/tests/quacc_tests/example.py", line 15, in <module>
    result = responses[job1.uuid][1].output
KeyError: 'b68f6e78-e3be-4bb7-9320-b476c43e5b36'

How can we easily reproduce the issue?

# Paste your *minimal* example code here
import jobflow as jf
from ase import Atoms
from fireworks import LaunchPad
from quacc.recipes.qchem.ts import ts_job
from quacc.recipes.qchem.ts import irc_job
from quacc.recipes.qchem.core import freq_job
from jobflow.managers.fireworks import flow_to_workflow

symbols = ['H', 'O', 'H']
coordinates = [(0, 0, 0), (0, 0, 1), (0, 1, 0)]
atoms = Atoms(symbols=symbols, positions=coordinates)
job1 = ts_job(atoms, 0, 1, method="wb97x", basis="6-31G*")
flow = jf.Flow([job1])
responses = jf.run_locally(flow)
result = responses[job1.uuid][1].output
print(result)
kumaranu commented 11 months ago

@Andrew-S-Rosen I was not able to add quacc_settings above because I am not sure how to access that information. I am using the default settings from settings.py, I believe. Thanks.

Andrew-S-Rosen commented 11 months ago

@kumaranu:

Thank you for your issue report. I was able to reproduce your issue with the following example:

quacc set WORKFLOW_ENGINE local # just to make sure it's not jobflow
from ase import Atoms
from quacc.recipes.qchem.ts import ts_job

symbols = ['H', 'O', 'H']
coordinates = [(0, 0, 0), (0, 0, 1), (0, 1, 0)]
atoms = Atoms(symbols=symbols, positions=coordinates)
ts_job(atoms, 0, 1)

I do not yet know the cause but will investigate.

kumaranu commented 11 months ago

Hi @Andrew-S-Rosen, I am using the current version of the code. It eliminated the circular import error I mentioned earlier, but, it led to some other problems.

Here is what I see. Please let me know if there is anything that you see right away or do I need to put it with more details.

(quacc) [kumaranu@n0003 quacc_tests]$ python prod26_qchem.py 
2023-11-28 12:20:29,646 INFO Started executing jobs locally
2023-11-28 12:20:29,649 INFO Starting job - ind005_nov8d_ts (133ad2ff-3224-4b7d-baa3-bad3a09bf14c)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
SLURM_CPUS_ON_NODE not in environment
/bin/sh: qchem: command not found
Traceback (most recent call last):
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/qchem_custodian.py", line 107, in <module>
    run_custodian(qchem_cores=int(sys.argv[1])) if len(  # skipcq: PYL-W0106
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/monty/dev.py", line 100, in decorated
    return _callable(*args, **kwargs)
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/qchem_custodian.py", line 103, in run_custodian
    c.run()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/custodian/custodian.py", line 383, in run
    self._run_job(job_n, job)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/custodian/custodian.py", line 492, in _run_job
    has_error = self._do_check(self.handlers)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/custodian/custodian.py", line 648, in _do_check
    if h.check():
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/custodian/qchem/handlers.py", line 65, in check
    self.outdata = QCOutput(self.output_file).data
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/pymatgen/io/qchem/outputs.py", line 59, in __init__
    with zopen(filename, mode="rt", encoding="ISO-8859-1") as f:
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/monty/io.py", line 45, in zopen
    return open(filename, *args, **kwargs)  # pylint: disable=R1732
FileNotFoundError: [Errno 2] No such file or directory: 'mol.qout'
2023-11-28 12:20:38,421 INFO ind005_nov8d_ts failed with exception:
Traceback (most recent call last):
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/jobflow/managers/local.py", line 102, in _run_job
    response = job.run(store=store)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/jobflow/core/job.py", line 583, in run
    response = function(*self.function_args, **self.function_kwargs)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/monty/dev.py", line 100, in decorated
    return _callable(*args, **kwargs)
  File "/global/home/users/kumaranu/quacc/src/quacc/recipes/qchem/ts.py", line 115, in ts_job
    return base_opt_fn(
  File "/global/home/users/kumaranu/quacc/src/quacc/recipes/qchem/_base.py", line 114, in base_opt_fn
    dyn = run_opt(atoms, copy_files=copy_files, **opt_flags)
  File "/global/home/users/kumaranu/quacc/src/quacc/runners/ase.py", line 181, in run_opt
    dyn.run(fmax=fmax, steps=max_steps, **run_kwargs)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 338, in run
    return Dynamics.run(self)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 218, in run
    for converged in Dynamics.irun(self):
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 186, in irun
    self.optimizable.get_forces()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 30, in get_forces
    return self.atoms.get_forces()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/atoms.py", line 791, in get_forces
    forces = self._calc.get_forces(self)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/abc.py", line 30, in get_forces
    return self.get_property('forces', atoms)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 503, in get_property
    self.calculate(atoms, [name], system_changes)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 906, in calculate
    self.execute()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 936, in execute
    raise CalculationFailed(msg)
ase.calculators.calculator.CalculationFailed: Calculator "qchem" failed with command "python /global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/qchem_custodian.py 32" failed in /global/home/users/kumaranu/.scratch/quacc-tmp-2023-11-28-20-20-29-651539-gcktq4zt with error code 1

2023-11-28 12:20:38,423 INFO Finished executing jobs locally
Traceback (most recent call last):
  File "/global/home/users/kumaranu/Documents/tests/quacc_tests/prod26_qchem.py", line 103, in <module>
    add_to_launchpad(index, atoms, tag, RUN=run)
  File "/global/home/users/kumaranu/Documents/tests/quacc_tests/prod26_qchem.py", line 76, in add_to_launchpad
    result = responses[job2.uuid][1].output
KeyError: '6268d01c-f83c-4e41-9f12-1bacaae2393d'
Andrew-S-Rosen commented 11 months ago

@kumaranu: Looks like you don't have the qchem executable in your PATH.

kumaranu commented 11 months ago

@Andrew-S-Rosen : Thanks for correcting me on that.

I did not have the qchem module loaded there.

Now, I have loaded the module and you can see that it asks for the input files when I type in the qchem command on the terminal.

Nevertheless, it still led to a different error as shown below:

(quacc) [kumaranu@n0003 quacc_tests]$ 
(quacc) [kumaranu@n0003 quacc_tests]$ qchem
You are running Q-Chem version: 6.0.2
QCSCRATCH /tmp should NOT be the same!
unset QCLOCALSCR ...
Error: input file not specified !

Usage: qchem [ -save ] [ -archive ] [ -mpi|-gpu ] [ -nt nthreads ] [ -np nprocs ] input output scratchDir

(quacc) [kumaranu@n0003 quacc_tests]$ 
(quacc) [kumaranu@n0003 quacc_tests]$ 
(quacc) [kumaranu@n0003 quacc_tests]$ 
(quacc) [kumaranu@n0003 quacc_tests]$ python prod26_qchem.py
2023-11-28 12:31:51,653 INFO Started executing jobs locally
2023-11-28 12:31:51,656 INFO Starting job - ind005_nov8d_ts (eb099417-1b49-40b3-8c51-552d22b70aa4)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
SLURM_CPUS_ON_NODE not in environment
Error in Q-Chem run part 1
cp: cannot stat '/tmp/scratch/*': No such file or directory
Error in the Q-Chem run
ERROR:custodian.custodian:QChemErrorHandler
Error in Q-Chem run part 1
cp: cannot stat '/tmp/scratch/*': No such file or directory
Error in the Q-Chem run
ERROR:custodian.custodian:QChemErrorHandler
Error in Q-Chem run part 1
cp: cannot stat '/tmp/scratch/*': No such file or directory
Error in the Q-Chem run
ERROR:custodian.custodian:QChemErrorHandler
Error in Q-Chem run part 1
cp: cannot stat '/tmp/scratch/*': No such file or directory
Error in the Q-Chem run
ERROR:custodian.custodian:QChemErrorHandler
Error in Q-Chem run part 1
cp: cannot stat '/tmp/scratch/*': No such file or directory
Error in the Q-Chem run
ERROR:custodian.custodian:QChemErrorHandler
ERROR:custodian.custodian:Max errors per job reached: 5.
Traceback (most recent call last):
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/qchem_custodian.py", line 107, in <module>
    run_custodian(qchem_cores=int(sys.argv[1])) if len(  # skipcq: PYL-W0106
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/monty/dev.py", line 100, in decorated
    return _callable(*args, **kwargs)
  File "/global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/qchem_custodian.py", line 103, in run_custodian
    c.run()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/custodian/custodian.py", line 383, in run
    self._run_job(job_n, job)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/custodian/custodian.py", line 532, in _run_job
    raise MaxCorrectionsPerJobError(msg, True, self.max_errors_per_job, job)
custodian.custodian.MaxCorrectionsPerJobError: Max errors per job reached: 5.
2023-11-28 12:32:03,332 INFO ind005_nov8d_ts failed with exception:
Traceback (most recent call last):
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/jobflow/managers/local.py", line 102, in _run_job
    response = job.run(store=store)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/jobflow/core/job.py", line 583, in run
    response = function(*self.function_args, **self.function_kwargs)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/monty/dev.py", line 100, in decorated
    return _callable(*args, **kwargs)
  File "/global/home/users/kumaranu/quacc/src/quacc/recipes/qchem/ts.py", line 115, in ts_job
    return base_opt_fn(
  File "/global/home/users/kumaranu/quacc/src/quacc/recipes/qchem/_base.py", line 114, in base_opt_fn
    dyn = run_opt(atoms, copy_files=copy_files, **opt_flags)
  File "/global/home/users/kumaranu/quacc/src/quacc/runners/ase.py", line 181, in run_opt
    dyn.run(fmax=fmax, steps=max_steps, **run_kwargs)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 338, in run
    return Dynamics.run(self)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 218, in run
    for converged in Dynamics.irun(self):
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 186, in irun
    self.optimizable.get_forces()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/optimize/optimize.py", line 30, in get_forces
    return self.atoms.get_forces()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/atoms.py", line 791, in get_forces
    forces = self._calc.get_forces(self)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/abc.py", line 30, in get_forces
    return self.get_property('forces', atoms)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 503, in get_property
    self.calculate(atoms, [name], system_changes)
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 906, in calculate
    self.execute()
  File "/global/home/users/kumaranu/.conda/envs/quacc/lib/python3.9/site-packages/ase/calculators/calculator.py", line 936, in execute
    raise CalculationFailed(msg)
ase.calculators.calculator.CalculationFailed: Calculator "qchem" failed with command "python /global/home/users/kumaranu/quacc/src/quacc/calculators/qchem/qchem_custodian.py 32" failed in /global/home/users/kumaranu/.scratch/quacc-tmp-2023-11-28-20-31-51-657530-pfeidx41 with error code 1

2023-11-28 12:32:03,333 INFO Finished executing jobs locally
Traceback (most recent call last):
  File "/global/home/users/kumaranu/Documents/tests/quacc_tests/prod26_qchem.py", line 103, in <module>
    add_to_launchpad(index, atoms, tag, RUN=run)
  File "/global/home/users/kumaranu/Documents/tests/quacc_tests/prod26_qchem.py", line 76, in add_to_launchpad
    result = responses[job2.uuid][1].output
KeyError: '8f1c9a38-fb0c-419c-8620-114e6a444439'
(quacc) [kumaranu@n0003 quacc_tests]$

Another piece of info: I have species the environment variables in my .bashrc file:

 28 export QCSCRATCH=/tmp
 29 export MODULEPATH=${MODULEPATH}:/clusterfs/mp/modules
 30 export PROJ=/global/home/groups/lr_mp/smblau/kumaranu
Andrew-S-Rosen commented 11 months ago

@kumaranu: Thanks for the follow up. Unfortunately, I am not too familiar with the Q-Chem setup right now. Your best bet is to ask @samblau until I have some time to look at this further. I'm trying to make the recipes easier to use in #1123, but it isn't ready to merge yet.

kumaranu commented 11 months ago

@Andrew-S-Rosen, After talking with @samblau, I did a couple more things to get rid of the errors.

  1. I was trying to run the tests locally using the run_locally command. @samblau suggested I run them on compute nodes instead.
  2. I still got some errors after running them on compute nodes. @samblau suggested I update Pymatgen and custodian both. I did that and it worked out!

Thanks to both of you for your help.