Add support for SunPy in Carnets with scipy

[danseaton]

SunPy adds functionality to astropy (already supported by carnets with scipy), especially working with solar system coordinates and solar image data. But it is not installable because it is not pure python. Would you consider adding support for SunPy? Thanks!

[Cadair]

:wave: SunPy developer here.

We have been looking into this and it looks like we would need to package up multidict, yarl and sunpy itself at minimum.

We would be happy to work with you on this if you let us know what needs to be done.

[jeplowman]

Another SunPy user here. I've been looking into this as well, and talking with Cadair about it. Looks to me like there are a couple of facets to it:

1. In the big picture, webasm projects like Pyodide allow a full Python stack to run in a browser, even on iPads. This includes packages requiring compilation, although it does require providing a meta.yaml file to Pyodide (https://pyodide.org/en/stable/development/meta-yaml.html). There currently isn't one for SunPy, although that may soon change. Perhaps Carnets could take advantage of this?

2. I think SunPy itself is pure Python and it doesn't appear to have many non-pure Python dependencies. yarl and multidict are the only two that pyodide's micropip finds, and those can be installed in pure Python mode by setting environment variables: %env MULTIDICT_NO_EXTENSIONS=1 %pip install multidict %env YARL_NO_EXTENSIONS=1 %pip install yarl Rebooting Kernel may be necessary after these. In principle, it should then be possible to install SunPy in carnets using %pip install --no-dependencies sunpy However, it fails with "ModuleNotFoundError: No module named 'extension_helpers'". I tried doing a %pip install extension_helpers, which appeared to have completed successfully, but got the same ModuleNotFoundError when I tried to install sunpy again. extension_helpers is not going into site-packages due to Carnets limitations, so perhaps that's part of the issue?

[holzschu]

Hi, thanks for the help. Carnets already defines MULTIDICT_NO_EXTENSIONS and YARL_NO_EXTENSIONS, so installing multidict and yarl works without problems. Same for installing aiohttp, which is what sunpy needs.

But you are right, pip install sunpy fails, even with --no-dependencies. Looking at the output, it seems that pip tries to install dependencies, even with the --no-dependencies flag (it says at the top "Installing build dependencies: started"). The error message is what happens when pip tries to compile a package without a compiler present (the line just above extension_helper missing is compile(...)).

I'm going to explore a bit.

[holzschu]

I was wrong, the problem does not occur in "Installing build dependencies", it occurs in "Getting requirements to build wheel" (which explains why --no-dependencies does not help). Even with --debug, I could not get pip to tell me what these requirements are.

[holzschu]

I've tracked it down to the following issue: pip tries to install all the dependencies listed in pyproject.toml in a separate sub-directory in ~/tmp, then run the install from this sub-directory. I think it's for security reasons, but it's not very convenient in the current situation. That's why it says "extension_helpers" is not installed, even though it is: it was not installed in this sub-directory. And once we get past "extension_helpers", it will try to re-install numpy, which is definitely not going to work.

So it looks like I will have to tweak pip or at least convince it to look at already installed packages.

Update: running !pip install sunpy --no-build-isolation gets past this dependencies roadblock, but then stops because it cannot compile the sunpy.io._pyana extension, which is written in C. I don't know how essential the _pyana extension is for sunpy, or whether it would be possible to run sunpy without it. It seems that it is the only module in sunpy that is written in C.

[Cadair]

The pyana extension isn't very important (it's an archaic file format), but we don't seem to provide a hook for ignoring it on non-windows platforms (it doesn't work on windows).

I can write a PR to sunpy which provides you with a env var to skip compiling it. Out of interest how does Carnets handle C extensions in general? Presumably you have some support for them?

[holzschu]

Having an environment variable to ignore pyana would be great, and would solve this issue. It would also help with PyIodide, by making it possible to install sunpy with micropip.

To answer your question about C extensions: Carnets supports them if they were built during the development time, but cannot support them in user-installed packages. The blocking point here is Apple security rules: all executable binaries (including dynamic libraries) must be present when the app is evaluated before release. So, for all modules that have non-skippable C extensions (like numpy, scipy, astropy...), these are compiled and shipped with the app as separate dynamic libraries.

There is a bit more to it, because dynamic libraries have to be encapsulated inside frameworks, so I rewrote a bit of the code in charge of loading extensions (we're not loading "library.so", but "library.framework/library"), but that's the key idea. The consequence is that the user can install pure Python packages, but if a package has a C library, it has to be installed by the developer, which takes more time.

[Cadair]

Makes sense, thanks for the info.

I think that pyiodide only looks for pure python wheels rather than python-only tarballs?

[holzschu]

Hi, some follow-up on this issue: the current TestFlight version for Carnets-with-scipy has all the code required so that "!pip install sunpy" works: https://testflight.apple.com/join/UbNIjkjp

(or at least, it does in my tests, but I'm interested in feedback before release). Carnets will have the same, but it hasn't been approved for TestFlight yet.

[danseaton]

I'm downloading the TestFlight version and will give this a try.

[danseaton]

I was able to get SunPy installed, but it seems like even basic functionality still might not be compatible as it is presently implemented. Even just trying:

import sumpy.map

fails because it seems to be trying to generate directories in ways that are not allowed. I don't know enough about what's happening on the backend to be able to assess whether there's a quick fix or not.

[holzschu]

The good news: I think I see where the problem is. The bad news: I don't know how to fix it without patching. I'm going to run some tests, and maybe submit a PR to sunpy (but I'll install sunpy with Carnets in the meantime, for tests). Edit: I may be able to make it work by creating a configuration file and placing it in the right place. Editx2: I have it working by creating the right configuration file in ~/Library/Application Support/sunpy. Well, at least for "import sunpy.map" and "import sunpy.data.sample".

[danseaton]

Good news! If you can push changes to a testflight, I can test additional SunPy features more extensively.

[holzschu]

Thank you very much for that. Build 210 is now out on TestFlight. When you import sunpy, it will create a sunpy directory in "On my iPad/Carnets" and also one in "~/Library/Application Support/". The latter contains the config file and most of the data you download, but is not visible from the Files app.

[danseaton]

I haven't rigorously tested everything, but most features seem to be working fine for me. I have downloaded data, done some analysis, used the sample data, all with no issue. I had to install a couple of packages required to retrieve data (zeep, drms) but that worked without a problem. So thus far things look good.

I wonder if other SunPy users/developers here (@jeplowman, @Cadair?) might have time to take a brief look and see if they identify any problems I have missed. Could somebody run the key package tests, maybe?

[Cadair]

I don't have an iOS device I can test this on, but it sounds like it's going well :)

[holzschu]

Thank you very much for your help with this.