conda support

[usdk1]

Hi, I would like to use Carnets to use a programmatic CAD tool on the ipad (https://github.com/bernhard-42/jupyter-cadquery). To run cadquery some packages need to be installed. Is there any chance to get conda up and running?

[holzschu]

conda is, to the best of my knowledge, incompatible with the restrictions on iOS, as it tries to duplicate the entire Python environment. You can have virtual environments in the latest version fo Carnets, with "!python -m venv newEnv", followed by "!run newEnv/bin/activate.py". These virtual environments are limited to user-installed packages.

The good news is that conda is not necessary to run jupyter-cadquery, it's just a way to isolate the packages required by the installation from the rest of the installation.

The bad news is that jupyter-cadquery requires jupyterlab, which is not installed in Carnets at the moment.

The good news is that I am, currently, working on the next major release of Carnets, which will have jupyterlab.

[bernhard-42]

@holzschu You are right, conda is not needed for Jupyter-Cadquery. Pip does the trick.

The real reason why I use conda in the installation guide is CadQuery. It uses a C++ based opencascade kernel wrapped into Python via the module github.com/CadQuery/OCP. This module is basically a big .so file for Intel CPUs - which for sure will not be digested by iOS (let alone Apple allowing to load additional native libs at runtime, so ARM support in OCP alone wouldn't help ...).

And even pip doesn't like several 100MB big artifacts, that's why CadQuery is published as conda package.

To my knowledge, Carnet would need to compile OCP into the app (as is done for numpy, scipy, ...). And I am not sure you would like the effort and the significant increase of size of your app (the osx intel .so lib has 260MB)

Additionally, Jupyter-CadQuery is based on cad-viewer-widget which is a prebuild ipywidgets extension. Besides the Jupyterlab support, Carnet would need to support Jupyterlab extensions (at least prebuilt ones) - plus the native part of CadQuery compiled into it.

cc @usdk1

[holzschu]

@bernhard-42 thanks for the detailed summary. You are absolutely correct in your analysis of Carnets. The next big step for Carnets is support for Jupyterlab (and prebuilt jupyterlab extensions), which will remove one obstacle. The "effort and significant increase of the app" will remain, and these are big obstacles (also, Apple has placed a limit to the number of embedded frameworks, so I'm more careful when adding new packages).

Anyway, for @usdk1 question, the short answer is "not in the near future".

[jakirkham]

Would be curious to understand how pip is allowed while conda isn't

[holzschu]

Pip has a separation between “system-packages” and “user-installed packages”. iOS allows the user to install additional text files, but not binary files. This fits nicely with pip model: packages that contain binary files are installed with the app, and the user can install any additional pure-Python packages since they are text files. Also, binary files can be used only if they are in a protected area that cannot be changed by the user (all of this is for security reasons).

Conda duplicates the entire set of Python files in a new place. That means that all binary files are now outside of the protected area and unusable.

[jakirkham]

Thanks for the context! 🙏

Sorry for the likely naïve questions

What constitutes binary? Would WebAssembly be allowed? How about WAT?

[holzschu]

That is a very good question.

Binary, in that context, means an Arm64 binary that will be executed directly by the processor (so something linked with the iOS SDK, that has access to the system functions). Either an executable or a dynamic library (Python modules use tons of dynamic libraries). These are checked by the system before being loaded, and can only be loaded if: they are in the restricted place, and they are signed with a developer certificate (both conditions must be true).

Anything else can be added by the user, including WebAssembly files or x86 binaries. The problem is that these do not have access, by design, to the system functions (the interpreter does, but that's different). For speed, Python inside Carnets and a-Shell is cross-compiled to Arm64, and all the modules are cross-compiled to Arm64 as well. The good point is that it's really fast. The bad point is that I haven't found a way to add binary modules to the app.

In theory, one could make a "WebAssembly Carnets", where all Python code is running in WebAssembly, and that would provide the ability to add more modules at runtime (assuming one also includes clang).