Support older versions of numpy, scipy

[ZedThree]

Add shims and fall-back functionality to support numpy==1.20.3 and scipy==1.7.3.

• Use shgo global optimiser for SciPy < 1.9. This is a little slower than direct but does still seem to work
• Fall back to np.ndarray for some type hints for numpy < 1.21. This is less specific, but makes no difference at runtime
• For interpolated magnetic fields, fall back to explicit finite differences for poloidal flux derivatives for scipy < 1.9, using the MagneticField base class implementations. This means we have to pass delta_R/Z back to the EFIT fields, but if the user doesn't set them, they'll still get sensible values.

@quinntpratt I've checked this all works in a python 3.9 virtual environment, but maybe you want to check this works on iris?

[quinntpratt]

@ZedThree yes I'll try this out on iris and report back.

[quinntpratt]

@ZedThree I tried it out but I get: ModuleNotFoundError: No module named 'freeqdsk' I'm not sure how to properly "install" it... I can't use pip and it's not part of conda.

There's probably some way I can clone the repo from github and just move things around so scotty can find it.

I think the best path forward is for me to work on making scotty its own module file on omega so we can have our own env. including non-conda things like freeqdsk.

[ZedThree]

Apologies, I'm not very familiar with conda -- can you not pip install . from the scotty directory? It should be happy with the versions of numpy and scipy in the conda environment, and just install freeqdsk (and then scotty).

[quinntpratt]

@ZedThree my apologies, I'm not too familiar with pip, but my understanding is that both pip and conda are python package managers (though conda can manage more than python...) and I don't think they're interoperable. I was told not to use pip to do anything on these clusters where everything is managed by conda because it will confuse the python versions, paths, etc.

I have a mostly working conda env. on the omega cluster for scotty where I handled freeqdsk by cloning the git repo and then running conda develop <path-to-freeqdsk> which makes things like from freeqdsk import <> work in a conda env.

I think iris is basically a lost-cause unless we adhere to the omfit python env. (i.e. no freeqdsk without resorting to hacks... and I don't know what hacks I would need... maybe it's something as simple as cloning freeqdsk directly into the scotty repo so that something like from freeqdsk import <> to work).

[valerian-chen]

Shame it doesn't work on Iris. Changes look good though, should I merge into master or is this still a work in progress?

[ZedThree]

@quinntpratt It is definitely possible to still use pip alongside conda, the issue is that they don't know about each other, so it is possible to clobber packages installed with the other tool. Though, if you have a conda environment active and use python -m pip then there's no problem getting the python version mixed up.

I believe best practice is to have an environment.yml file with pip dependencies explicitly listed.

Here's a couple of pages with some more details:

• https://www.anaconda.com/blog/using-pip-in-a-conda-environment
• https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment

So I think a minimal way to get something set up would be:

conda env export > environment.yml

to make a copy of the existing environment, and then add to the end of the dependencies list:

# change the name from the existing environment
name: scotty
dependencies:
  # packages already installed by conda
  ...
  - pip
  - pip:
    /path/to/local/scotty
    # or to install github version:
    git+https://github.com/beam-tracing/Scotty.git

Then you can do:

conda env create -f environment.yml

which should make a new environment, reusing all the pre-installed conda packages, and only installing scotty and freeqdsk with pip. This works locally for me, although of course I probably have a very different set up than iris.

@valerian-chen These changes should enable using older versions of Numpy/Scipy while still using the more recent features when available, so I think they will still be useful, even if they don't help for iris/omega.