qffbuddy error: No module named 'idlelib'

[isolated-matrix]

While pbqff seems to be working now, I have come across an error trying to use qffbuddy. When I try entering the command qffbuddy -h, I am given the following error:

Traceback (most recent call last):
  File "/usr/local/bin/qffbuddy", line 14, in <module>
    from idlelib.tooltip import Hovertip
ModuleNotFoundError: No module named 'idlelib'

After a bunch of reading, and realising I did not have python3-tk installed (it is called py-tkinter in MacPorts), it looks like the error is something to do with not being able to find the location of idlelib. I have tried adding it to my PATH variable (export PATH="$PATH:/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/idlelib"), but that did not help. Is there another path variable I should be using?

[ntBre]

Oh no, Python packaging issues! I think PYTHONPATH is the variable you need to set. I thought IDLE and tkinter were both in the standard library, though, so it's weird that they can't be found.

[isolated-matrix]

Sorry I did not reply earlier!

Turns out adding a $PYTHONPATH fixed that issue (I did not have one set before, for some reason), but I came up against another error.

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/io.py", line 55, in <module>
ImportError: cannot import name 'text_encoding' from 'io' (unknown location)

I have been trying to figure out what might have caused this and how I can fix it. I have found a few possible answers (e.g., this one and this one), but I have been working on other things and have not looked into it too much yet. I will let you know what I come up with, though!

[ntBre]

No problem at all!

Did you install Python just for pbqff/qffbuddy, or did you have a working installation before? To me this looks like something is very wrong with your Python installation, especially the first two lines. I guess at least one person on stackoverflow agrees with me

I had same error but not on windows and I solved it by rebuilding Python, so I suggest you re-install Python

I kinda hate to recommend this, but we use conda environments at work (or really mamba/micromamba, which are much faster). qffbuddy shouldn't actually require any third-party dependencies, which is what conda usually helps with, but it would also be a way to install a self-contained Python version if your system one is doing something weird.

[isolated-matrix]

Looks like I'm not the only one with this issue - someone else has found it for another program, but it doesn't look like they found a solution...

I reinstalled the Macports version of Python, and updated the Miniconda version of Python, but I've still not had any success. Either way, it looks like the problem is not on your end!

I guess I have to figure out how to interface pbqff with Molpro on our HPC without a GUI... Wish me luck!

[ntBre]

Good luck! Feel free to ask any questions you run into. I've been meaning to improve the documentation for a while. The PDF manual and some of the test input files might help. The qffbuddy source contains some additional examples of chemistry program templates and also queue templates. Hopefully your HPC is running either PBS or Slurm? There's also a Local queue option, which can run the jobs on a single node with bash, but PBS and Slurm are the distributed queues it currently supports.

[isolated-matrix]

Thankfully our HPC uses Slurm.

In that case, I do have a few questions... The main things I am unsure about are how to use the templates, and how to specify the geometry in the pbqff input:

• How do I tell pbqff to use a particular job script as a template? And how do I tell it where in the script the input file name should be written?
• Do I need a Molpro input file template? How can I indicate to pbqff where the geometry should go? (I assume I set the number of processors and RAM to be used as part of the template?)
• I would be running this with Molpro, but all your examples are for MOPAC, I think, so I am not sure whether the geometry field of the pbqff input should look like Molpro's geometry input or using the style in the test files.

[ntBre]

This is the Molpro template from qffbuddy, adapted to be pasted into a pbqff.toml input file:

template = """memory,1,g
gthresh,energy=1.d-12,zero=1.d-22,oneint=1.d-22,twoint=1.d-22;
gthresh,optgrad=1.d-8,optstep=1.d-8;
nocompress;

geometry={
{{.geom}}
basis={
default,cc-pVTZ-f12
}
set,charge={{.charge}}
set,spin=0
hf,accuracy=16,energy=1.0d-10
{CCSD(T)-F12,thrden=1.0d-12,thrvar=1.0d-10}
{optg,grms=1.d-8,srms=1.d-8}

pbqff=energy(2)
show[1,f20.12],pbqff
"""

All of the {{.field}} parts are filled in by pbqff. You don't need to read this documentation or anything, but this is inspired by the Go template library. The geometry handling is a little weird for Molpro. You might notice that the template doesn't have a closing brace } for the geometry block. That allows you to put in either a Z-matrix, in which case pbqff adds the brace between the geometry and the parameter values, or an XYZ, where pbqff will add the brace after the last line. pbqff also replaces {{.charge}} with the charge specified as a TOML value, but you can also just set the charge literally in the template. The only other critical part are the last two lines. pbqff only extracts the energy if it's called pbqff. For most methods other than CCSD(T)-F12, you just write pbqff=energy (you likely know this, but the F12 energy is a vector in Molpro with the F12a and F12b values, which is why you need energy(2) for F12b).

Once you have some templates you like, you can also supply the template as a path to a file as in:

template = { file = "/path/to/template.file" }

Almost nobody in my lab uses this, though.

The job script is very similar, except it goes in the queue_template field of the TOML file:

queue_template = """#!/bin/bash
#SBATCH --job-name={{.filename}}
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH -o {{.filename}}.out
#SBATCH --no-requeue
#SBATCH --mem=8gb
"""

Apparently the qffbuddy example for this is actually outdated because it says {{.basename}} instead of {{.filename}} (I haven't run pbqff on a Slurm cluster in a while). I'm also noticing that I've hard-coded /home/qc/bin/molpro2020.sh 1 1 {f}.inp as the command for each single-point energy calculation that gets written to the Slurm script. Unless your Molpro command happens to have this exact path and filename, I'm going to have to push a fix. What I've been doing for other programs is having pbqff write something like $MOLPRO_CMD {f}.inp, so a full template once I fix this would look something like this:

queue_template = """#!/bin/bash
#SBATCH --job-name={{.filename}}
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH -o {{.filename}}.out
#SBATCH --no-requeue
#SBATCH --mem=8gb

export MOLPRO_CMD=/path/to/molpro
"""

Does that help? I've been intending to write some of this in a better form of documentation than just the manual page, so let me know of anything else confusing that I can try to include there!

Like I said, I'll also have to push a change to include the MOLPRO_CMD part. I'll comment here again once that's done, hopefully this evening.

[isolated-matrix]

That's very helpful, thank you! I will give this all a try today and see if I can get it working.

One thing I would mention, though, is that both of the HPCs I have used so far don't use a Molpro command (MOLPRO_CMD=/path/to/molpro), exactly, they have the programs set up as modules. For example, apart from setting up scratch directories, my Molpro submission script only says:

module purge
module add Molpro/mpp-2022.2.2.linux_x86_64_mpipr
module add NBO

export NBOEXE=${EBNBOROOT}/bin/nbo7.i8.exe

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

molpro -n ${SLURM_NTASKS} -t ${SLURM_CPUS_PER_TASK} -d ${TMPDIR} -I${INTDIR} -W${WFUDIR} ${INPUT}

Would that work for this kind of calculation? I'm sure we can find out the directory to the Molpro binaries, but I'm not sure the calculation would run if we did it that way.

[ntBre]

I think that should be okay. You should be able to stuff all of this into MOLPRO_CMD="molpro -n ${SLURM_NTASKS} -t ${SLURM_CPUS_PER_TASK} -d ${TMPDIR} -I${INTDIR} -W${WFUDIR}" (I left off ${INPUT} assuming that's the input file). And the other lines can just go below the #SBATCH directives. Our normal PBS scripts look very similar with the module imports, followed by exporting some variables. The important thing is that it's okay to append the input filename to the $MOLPRO_CMD, which you could get around with a shell function if you really had to, but assuming that's what $INPUT is, it should work!

I still have to make the change that will allow $MOLPRO_CMD at all, but I hope to get around to that tonight or at least this week.

[ntBre]

Okay, just pushed a commit (ntBre/psqs@f22b342) that I think should resolve this. However, in a bad sign for my test coverage in psqs, the change didn't cause any existing tests to fail, so please let me know if you run into any issues!

[isolated-matrix]

I have just taken a look at the qffbuddy files and I think I know why they're not working. The qffbuddy file just contains:

brent=/ddn/home1/r2518
host=$(hostname)
case $host in
    sequoia)
        module load python/3.7
        ;;
    maple)
        module load python37
        ;;
esac
python3.7 $brent/bin/qffbuddy.py --pbqff $brent/bin/rpbqff "$@"

and I don't have Python 3.7 installed, nor do I have a /ddn/home1/r2518 directory.

[isolated-matrix]

I got qffbuddy to work! I just needed to run the Python script directly... I should have tried that before, sorry!

[ntBre]

Oh wow, that makes a lot of sense! I thought the Makefile was installing the qffbuddy.py script, I didn't realize it was installing this wrapper shell script. Clearly parts of the project are very closely tied to our infrastructure!