local (especially Jupyter Notebook) dev / debug?

[m-k-S]

Hi, I'm very interested in using Psiflow to set up MD experiments but I'm trying to get it running on my local machine before plugging it into e.g. the HPC cluster I have access to. However, I have a lot of issues doing this. For example, when I try to run the H2 static/dynamic example, I get:

RuntimeError                              Traceback (most recent call last)
Cell In[8], [line 63](vscode-notebook-cell:?execution_count=8&line=63)
     [54](vscode-notebook-cell:?execution_count=8&line=54)     return frequencies[-1] * second / (100 * _c)
     [58](vscode-notebook-cell:?execution_count=8&line=58) geometry = Geometry.from_data(
     [59](vscode-notebook-cell:?execution_count=8&line=59)     numbers=np.ones(2),
     [60](vscode-notebook-cell:?execution_count=8&line=60)     positions=np.array([[0, 0, 0], [0.8, 0, 0]]),
     [61](vscode-notebook-cell:?execution_count=8&line=61)     cell=None,
     [62](vscode-notebook-cell:?execution_count=8&line=62) )
---> [63](vscode-notebook-cell:?execution_count=8&line=63) mace = MACEHamiltonian.mace_mp0()
     [65](vscode-notebook-cell:?execution_count=8&line=65) dynamic = frequency_dynamic(geometry, mace)
     [66](vscode-notebook-cell:?execution_count=8&line=66) static = frequency_static(geometry, mace)

File ~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:416, in MACEHamiltonian.mace_mp0(cls, size)
    [411](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:411) urls = dict(
    [412](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:412)     small="https://github.com/ACEsuit/mace-mp/releases/download/mace_mp_0/2023-12-10-mace-128-L0_energy_epoch-249.model",  # 2023-12-10-mace-128-L0_energy_epoch-249.model
    [413](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:413)     large="https://github.com/ACEsuit/mace-mp/releases/download/mace_mp_0/2023-12-03-mace-128-L1_epoch-199.model",
    [414](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:414) )
    [415](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:415) assert size in urls
--> [416](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:416) parsl_file = psiflow.context().new_file("mace_mp_", ".pth")
    [417](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:417) urllib.request.urlretrieve(
    [418](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:418)     urls[size],
    [419](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:419)     parsl_file.filepath,
    [420](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:420) )
    [421](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/hamiltonians.py:421) return cls(parsl_file, {})
...
    [598](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:598)     if cls._context is None:
--> [599](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:599)         raise RuntimeError("No ExecutionContext is currently loaded")
    [600](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:600)     return cls._context

RuntimeError: No ExecutionContext is currently loaded

And when I try to run the basic geometry example, I get:

AssertionError                            Traceback (most recent call last)
Cell In[7], [line 7](vscode-notebook-cell:?execution_count=7&line=7)
      [3](vscode-notebook-cell:?execution_count=7&line=3) from psiflow.geometry import Geometry
      [6](vscode-notebook-cell:?execution_count=7&line=6) # a simple H2 molecule in vacuum
----> [7](vscode-notebook-cell:?execution_count=7&line=7) geometry = Geometry.from_string(
      [8](vscode-notebook-cell:?execution_count=7&line=8) '''
      [9](vscode-notebook-cell:?execution_count=7&line=9)     2
     [10](vscode-notebook-cell:?execution_count=7&line=10)     H 0.0 0.0 0.0
     [11](vscode-notebook-cell:?execution_count=7&line=11)     H 0.0 0.0 0.8
     [12](vscode-notebook-cell:?execution_count=7&line=12) ''')
     [14](vscode-notebook-cell:?execution_count=7&line=14) # the same H2 molecule using ase Atoms
     [15](vscode-notebook-cell:?execution_count=7&line=15) atoms = ase.Atoms(
     [16](vscode-notebook-cell:?execution_count=7&line=16)             numbers=[1, 1, 1],
     [17](vscode-notebook-cell:?execution_count=7&line=17)             positions=[[0, 0, 0], [0, 0, 0.8]],
     [18](vscode-notebook-cell:?execution_count=7&line=18)             pbc=False,
     [19](vscode-notebook-cell:?execution_count=7&line=19)             )

File ~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:184, in Geometry.from_string(cls, s, natoms)
    [180](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:180) else:
    [181](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:181)     lines = s.rstrip().split(
    [182](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:182)         "\n"
    [183](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:183)     )  # i-PI nonperiodic starts with empty -> rstrip!
--> [184](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:184) assert len(lines) == natoms + 1
    [185](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:185) comment = lines[0]
    [186](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/geometry.py:186) comment_dict = key_val_str_to_dict_regex(comment)

AssertionError:

Are Psiflow scripts not intended to be developed locally? Is there any guide on how to get it working in a Jupyter notebook? I am running Python 3.10.15, a fresh Conda virtual env and installed Psiflow via pip.

[svandenhaute]

They are intended to be developed locally, but running them in a Jupyter notebook is perhaps nontrivial since Parsl requires to wrap the whole execution inside a context manager, and psiflow therefore also adopts that structure.

def main():
    # stuff
    pass

if __name__ == '__main__':
    with psiflow.load():
        main()

From your error, it looks like the psiflow.load() command was not executed in the notebook?

[m-k-S]

Sorry, yes, that was an oversight on my part. And ok, that makes sense re: Parsl. Is there any way to pass the contents of an equivalent YAML file to the main psiflow object in a Jupyter Notebook?

Also, even with the psiflow.load() context, I still get the error:

AssertionError                            Traceback (most recent call last)
Cell In[40], [line 56](vscode-notebook-cell:?execution_count=40&line=56)
     [53](vscode-notebook-cell:?execution_count=40&line=53)     frequencies = compute_frequencies(hessian, minimum).result()
     [54](vscode-notebook-cell:?execution_count=40&line=54)     return frequencies[-1] * second / (100 * _c)
---> [56](vscode-notebook-cell:?execution_count=40&line=56) with psiflow.load():
     [57](vscode-notebook-cell:?execution_count=40&line=57)     geometry = Geometry.from_data(
     [58](vscode-notebook-cell:?execution_count=40&line=58)         numbers=np.ones(2),
     [59](vscode-notebook-cell:?execution_count=40&line=59)         positions=np.array([[0, 0, 0], [0.8, 0, 0]]),
     [60](vscode-notebook-cell:?execution_count=40&line=60)         cell=None,
     [61](vscode-notebook-cell:?execution_count=40&line=61)     )
     [62](vscode-notebook-cell:?execution_count=40&line=62)     mace = MACEHamiltonian.mace_mp0()

File ~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:586, in ExecutionContextLoader.load(cls, psiflow_config)
    [584](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:584) assert len(sys.argv) == 2
    [585](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:585) path_config = psiflow.resolve_and_check(Path(sys.argv[1]))
--> [586](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:586) assert path_config.exists()
    [587](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:587) assert path_config.suffix in [".yaml", ".yml"], (
    [588](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:588)     "the execution configuration needs to be specified"
    [589](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:589)     " as a YAML file, but got {}".format(path_config)
    [590](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:590) )
    [591](https://file+.vscode-resource.vscode-cdn.net/Users/muon/Documents/materials/curie/~/miniforge3/envs/md/lib/python3.10/site-packages/psiflow/execution.py:591) with open(path_config, "r") as f:

AssertionError:

And this is with the following config file, note gpu explicitly marked as false. I believe the assertion expects gpu to always be set to True?

retries: 0
ModelEvaluation:
  gpu: false
  use_threadpool: true
  max_simulation_time: 0.4```

[svandenhaute]

Yes, you can just pass the contents of the YAML file as a dictionary to the psiflow.load() call. In your case this would have to be:

import psiflow

if __name__ == '__main__':
    config_dict = {
        'retries': 0,
        'ModelEvaluation': {
            'gpu': False,
            'use_threadpool': True,
            'max_simulation_time': 0.4,
        },
        'ModelTraining': {
            'gpu': True,
            'use_threadpool': True,
        },
    }
    with psiflow.load(config_dict):
        print('test')

At the moment, users are forced to also specify ModelTraining with gpu: true -- which obviously doesn't make sense if no training is required. I'll fix this asap.

Does it raise any errors when you use the above config in your Jupyter environment? I'll talk to the people at Parsl to ask what the recommended way is for users to execute workflows in notebooks. We should put effort in making this more straightforward.