search

pyro-kinetics / pyrokinetics

Python library to run and analyse gyrokinetics simulations
https://pyrokinetics.readthedocs.io/en/latest/#
GNU Lesser General Public License v3.0
25 stars 6 forks source link

Freeqdsk v0.5.0 breaking automatic GEQDSK file type determination #386

Closed spmarsden closed 2 months ago

spmarsden commented 2 months ago

The same code running on the same version of pyro is broken in a fresh venv installation. I presume due to an update in a dependency.

The code:

#!.venv/bin/python

from pathlib import Path
import numpy as np
from pyrokinetics import Pyro
from pyrokinetics.diagnostics import Diagnostics

from jetto_tools import binary

def calculate_gamma(run_dir:Path):

    # Validate the input.
    run_dir = Path(run_dir)

    # Open the needed input files.
    jsp = binary.read_binary_file(run_dir / 'jetto.jsp')
    xrho = jsp['XRHO'][-1]

    # Load up pyro object.
    pyro = Pyro(
        eq_file         = run_dir / "jetto.eqdsk_out",
        kinetics_file   = run_dir / "jetto.jsp",
        kinetics_type   = "JETTO",
        kinetics_kwargs = {"time_index": -1},
    )

    # Select code as CGYRO
    pyro.gk_code = "CGYRO"

    # Calculate the gamma values as a function of XRHO.
    gamma = np.empty(len(xrho))
    for i, x in enumerate(xrho):
        pyro.load_local_geometry(psi_n=x)
        gamma[i] = Diagnostics(pyro).ideal_ballooning_solver()

    # Save the gamma values.
    np.savetxt(run_dir / 'jsp_gamma.csv', gamma, delimiter=',')

    # Calculate the maximum gamma value.
    max_gamma = np.max(gamma)
    with open(run_dir / 'gamma_max.txt', 'w') as f:
        f.write(str(max_gamma))

if __name__ == '__main__':
    calculate_gamma("/common/cmg/smars/jetto/runs/run-step/spr-045/jetto-26/002-genetic-fit/Chromosome_chromosome__0.85_0.40_0.03_0.50_0.00_0.27_0.10_0.50_0.25_0.50_0.80_0.40")

The error:

Traceback (most recent call last):
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/factory.py", line 37, in type
    return self._registered_types[key]
KeyError: '/common/cmg/smars/jetto/runs/run-step/spr-045/jetto-26/002-genetic-fit/Chromosome_chromosome__0.85_0.40_0.03_0.50_0.00_0.27_0.10_0.50_0.25_0.50_0.80_0.40/jetto.eqdsk_out'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 136, in _chain_exceptions
    _chain_exceptions(*excs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 140, in _chain_exceptions
    raise exc
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 160, in type
    return super().type(key)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/factory.py", line 39, in type
    raise KeyError(
KeyError: "'/common/cmg/smars/jetto/runs/run-step/spr-045/jetto-26/002-genetic-fit/Chromosome_chromosome__0.85_0.40_0.03_0.50_0.00_0.27_0.10_0.50_0.25_0.50_0.80_0.40/jetto.eqdsk_out' is not recognised as a type of FileReader"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 136, in _chain_exceptions
    _chain_exceptions(*excs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 138, in _chain_exceptions
    raise exc from chain
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 163, in type
    return super().type(self._infer_file_type(key))
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 191, in _infer_file_type
    raise FileInferenceException(excs)
pyrokinetics.file_utils.FileInferenceException: Could not infer file type. The following exceptions were raised:
+ 'Pyrokinetics' raised --> ValueError("did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:\nhttps://docs.xarray.dev/en/stable/getting-started-guide/installing.html\nhttps://docs.xarray.dev/en/stable/user-guide/io.html")
+ 'GEQDSK' raised --> AttributeError("'GEQDSKFile' object has no attribute 'keys'")
+ 'TRANSP' raised --> OSError(-51, 'NetCDF: Unknown file format')
+ 'GACODE' raised --> ValueError('EquilibriumReaderGACODE could not find /common/cmg/smars/jetto/runs/run-step/spr-045/jetto-26/002-genetic-fit/Chromosome_chromosome__0.85_0.40_0.03_0.50_0.00_0.27_0.10_0.50_0.25_0.50_0.80_0.40/jetto.eqdsk_out')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/./run-pyro.py", line 50, in <module>
    calculate_gamma("/common/cmg/smars/jetto/runs/run-step/spr-045/jetto-26/002-genetic-fit/Chromosome_chromosome__0.85_0.40_0.03_0.50_0.00_0.27_0.10_0.50_0.25_0.50_0.80_0.40")
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/./run-pyro.py", line 23, in calculate_gamma
    pyro = Pyro(
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/pyro.py", line 198, in __init__
    self.load_global_eq(eq_file, eq_type, **eq_kwargs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/pyro.py", line 1504, in load_global_eq
    self.eq = read_equilibrium(self.eq_file, eq_type, **kwargs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/equilibrium/equilibrium.py", line 1054, in read_equilibrium
    return Equilibrium.from_file(path, file_type=file_type, **kwargs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 245, in from_file
    reader = cls._factory(str(path) if file_type is None else file_type)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/factory.py", line 66, in __call__
    return self.create(key, *args, **kwargs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/factory.py", line 45, in create
    return self.type(key)(*args, **kwargs)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 174, in type
    _chain_exceptions(RuntimeError(msg), infer_error, key_error)
  File "/home/smars/notebook/projects/step/concept-generation/concepts/spr-045/jetto-26/fit-gaussians/.venv/lib/python3.9/site-packages/pyrokinetics/file_utils.py", line 138, in _chain_exceptions
    raise exc from chain
RuntimeError: Unable to determine the type of 'Equilibrium' from the input '/common/cmg/smars/jetto/runs/run-step/spr-045/jetto-26/002-genetic-fit/Chromosome_chromosome__0.85_0.40_0.03_0.50_0.00_0.27_0.10_0.50_0.25_0.50_0.80_0.40/jetto.eqdsk_out'. If you supplied a file name, please check the exceptions raised above from each Equilibrium reader, and try fixing any reported issues. It may also help to specify the file type explicitly.

The old working venv:

appdirs==1.4.4
asttokens==2.4.1
Cerberus==1.3.5
certifi==2024.7.4
cftime==1.6.4
charset-normalizer==3.3.2
cleverdict==1.9.2
click==8.1.7
cloudpickle==3.0.0
contourpy==1.2.1
cycler==0.12.1
dask==2024.7.0
decorator==5.1.1
defusedxml==0.7.1
docker==7.1.0
exceptiongroup==1.2.2
executing==2.0.1
f90nml==1.4.4
flexcache==0.3
flexparser==0.3.1
fonttools==4.53.1
fortranformat==2.0.0
freeqdsk==0.4.0
fsspec==2024.6.1
h5py==3.11.0
idna==3.7
idspy-dictionaries==34000.2.0
idspy_toolkit==0.6.5
importlib_metadata==8.0.0
importlib_resources==6.4.0
ipython==8.18.1
jedi==0.19.1
jetto_tools==1.8.15
kiwisolver==1.4.5
locket==1.0.0
matplotlib==3.9.1
matplotlib-inline==0.1.7
netCDF4==1.7.1.post1
numpy==1.26.4
packaging==24.1
pandas==2.2.2
parso==0.8.4
partd==1.4.2
path==16.14.0
pexpect==4.9.0
pillow==10.4.0
Pint==0.24.3
pint-xarray==0.4
prominence==0.20.2
prompt_toolkit==3.0.47
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.18.0
PyJWT==2.8.0
pyloidal==0.2.0
pyparsing==3.1.2
pyrokinetics==0.7.0
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
scipy==1.13.1
simplejson==3.19.2
six==1.16.0
stack-data==0.6.3
toml==0.10.2
toolz==0.12.1
traitlets==5.14.3
typing_extensions==4.12.2
tzdata==2024.1
urllib3==1.26.14
wcwidth==0.2.13
xarray==2024.6.0
xmltodict==0.13.0
xrft==1.0.1
zipp==3.19.2

The new broken venv:

appdirs==1.4.4
asttokens==2.4.1
Cerberus==1.3.5
certifi==2024.8.30
cftime==1.6.4
charset-normalizer==3.3.2
cleverdict==1.9.2
click==8.1.7
cloudpickle==3.0.0
contourpy==1.3.0
cycler==0.12.1
dask==2024.8.0
decorator==5.1.1
defusedxml==0.7.1
docker==7.1.0
exceptiongroup==1.2.2
executing==2.1.0
f90nml==1.4.4
flexcache==0.3
flexparser==0.3.1
fonttools==4.53.1
fortranformat==2.0.0
freeqdsk==0.5.0
fsspec==2024.9.0
h5py==3.11.0
idna==3.10
idspy-dictionaries==34000.2.0
idspy_toolkit==0.6.5
importlib_metadata==8.5.0
importlib_resources==6.4.5
ipython==8.18.1
jedi==0.19.1
jetto_tools==1.8.15
kiwisolver==1.4.7
locket==1.0.0
matplotlib==3.9.2
matplotlib-inline==0.1.7
netCDF4==1.7.1.post2
numpy==1.26.4
packaging==24.1
pandas==2.2.3
parso==0.8.4
partd==1.4.2
path==17.0.0
pexpect==4.9.0
pillow==10.4.0
Pint==0.24.3
pint-xarray==0.4
prominence==0.20.2
prompt_toolkit==3.0.47
ptyprocess==0.7.0
pure_eval==0.2.3
Pygments==2.18.0
PyJWT==2.9.0
pyloidal==0.2.0
pyparsing==3.1.4
pyrokinetics==0.7.0
python-dateutil==2.9.0.post0
pytz==2024.2
PyYAML==6.0.2
requests==2.32.3
scipy==1.13.1
simplejson==3.19.3
six==1.16.0
stack-data==0.6.3
toml==0.10.2
toolz==0.12.1
traitlets==5.14.3
typing_extensions==4.12.2
tzdata==2024.1
urllib3==1.26.14
wcwidth==0.2.13
xarray==2024.7.0
xmltodict==0.13.0
xrft==1.0.1
zipp==3.20.2
ZedThree commented 2 months ago

Ah, this is my fault, I made a breaking change in freeqdsk == 0.5.0! Fix incoming

LiamPattinson commented 2 months ago

Just raised a PR to fix it. If anybody is running into this problem on Pyrokinetics installed from PyPI and would rather not use the latest version on GitHub, you can fix it by specifying the old version explicitly:

pip install pyrokinetics freeqdsk==0.4.0

We should probably make a new release soon with updated dependencies, and some of them that are specified as >= 0.x or ~= 0.x should probably be ~= 0.x.0 to avoid this sort of thing happening again.