sage --pip install --editable /path/to/<foo> is not visible in Jupyter unless Jupyter can access that directory (e.g., via full disk permissions)

[posita]

It was possible (if you worked around the lack of OpenSSL) on 9.1 to do something like …/SageMath-9.1.app/…/sage --pip install --editable <foo>, which would allow you to import <foo> in your Jupyter notebooks and receive updates as one hacked on the underlying library. This technique is very useful for development.

It appears that functionality is not longer working. After installing the recommended helper tools, /usr/local/bin/sage --pip install --editable <foo> appears to work (installing foo into ~/.sage/lib/python3.9/site-packages/<foo>.egg-link). However, trying to access it in Jupyter will result in something like:

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-c28bce4280cc> in <module>
----> 1 import <foo>

ModuleNotFoundError: No module named '<foo>'

Replace <foo> with your favorite library not already included with Sage. If you can't think of anything, tinyec has no dependencies and will stand in well to reproduce the above symptoms.

From a shell:

$ Python 3.9.2 (default, Jun 10 2021, 21:07:27)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tinyec
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tinyec'
>>> ^D
$ git clone https://github.com/alexmgr/tinyec.git
$ cd tinyec
$ sage --pip install --editable .
…
$ find ~/.sage/local -name '*tinyec*'
…/.sage/local/lib/python3.9/site-packages/tinyec.egg-link
$ sage --python
Python 3.9.2 (default, Jun 10 2021, 21:07:27)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tinyec ; print(tinyec.__file__)
…/tinyec/tinyec/__init__.py
>>> ^D

From Jupyter:

import tinyec

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-c28bce4280cc> in <module>
----> 1 import tinyec

ModuleNotFoundError: No module named 'tinyec'

Note that if you do not install the library via link to a local directory (i.e., do not use --editable), it will work:

From a shell:

$ sage --pip uninstall tinyec
Found existing installation: tinyec 0.3.1
Uninstalling tinyec-0.3.1:
  Would remove:
    …/.sage/local/lib/python3.9/site-packages/tinyec.egg-link
Proceed (y/n)? y
  Successfully uninstalled tinyec-0.3.1
$ sage --pip install tinyec
…

From Jupyter:

import tinyec

☝️ That works, perplexingly.

I bring this up because it appears to be a regression. (This technique worked for SageMath-9.1.app.)

[posita]

Okay, I think I figured out the root cause. From Jupyter:

import os
os.listdir("</path/to>/tinyec")

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-4-07f88aceeb75> in <module>
      1 import os
----> 2 os.listdir("</path/to>/tinyec")

PermissionError: [Errno 1] Operation not permitted: '</path/to>/tinyec'

I realized that I was running sage … from my terminal app, which has been granted full disk access, so that might explain why that worked. I granted full disk access to SageMath-9-3.app, but that did not appear to change things. I also looked into granting full disk access to the python3.9 binary inside SageMath-9-3.app, but it is not selectable:

[NathanDunfield]

First, thanks very much for the detailed report and initial diagnosis!

When you start Jupyter via SageMath-9-3.app, select as the notebook folder something that contains tinyec as a subfolder. Jupyter and its kernels are only able to access files that are below that "notebook root" directory, so I think this will solve your problem.

If that doesn't work, you could start the Jupyter notebook server manually in a Terminal.app window by sage -n. It should then inherit the full disk permissions.

However, I didn't test either of these, so let us know if either fix your problem.

[posita]

You beat me to the sage -n technique. That does indeed work (likely having inherited permissions from the terminal). I hadn't tried the root bit yet. Unfortunately, that technique doesn't work where the notebook and the linked development lib are in different locations. I started reading this to see if I could figure out why I couldn't grant full disk perms to the internal python3.9 binary. I suspect it's a code signing issue, but probably won't pursue too much further (at least not right away) since I have a work-around with sage -n.

[NathanDunfield]

I hadn't tried the root bit yet. Unfortunately, that technique doesn't work where the notebook and the linked development lib are in different locations.

You can set the notebook root directory all the way up to / by selecting Macintosh HD in the file dialog selector in SageMath.app. Then every file on your system will be below that and Jupyter should have full access do your disk. Of course, you then have to navigate in the Jupyter file browser to the actual directory where your notebook lives, which is a bit annoying.

[NathanDunfield]

By the way, what version of macOS are you running? I couldn't immediately duplicate what you're seeing in Catalina (10.15).

[posita]

Interesting. I'm running 10.15.7 (19H1217).

[culler]

I suspect that the reason the SageMath app cannot be directly given full disk access is that its main executable is actually a shell script which execs python3 to run itself as a python script. As a wild guess, I think that giving /bin/sh full disk access might cause the app to inherit full disk access.

[posita]

I suspect that the reason the SageMath app cannot be directly given full disk access is that its main executable is actually a shell script which execs python3 to run itself as a python script.

Yeah, that's why I looked into granting full disk access to the embedded python3.9 binary in particular:

$ file -h SageMath-9-3.app/Contents/Frameworks/Sage.framework/Versions/9.3/local/bin/python3.9
SageMath-9-3.app/Contents/Frameworks/Sage.framework/Versions/9.3/local/bin/python3.9: Mach-O 64-bit executable x86_64

(See my earlier comment with a screenshot.) Although, upon some additional digging, I don't know if this is a code signing issue:

$ codesign -d -v --entitlements - SageMath-9-3.app/Contents/Frameworks/Sage.framework/Versions/9.3/local/bin/python3.9
Executable=…/SageMath-9-3.app/Contents/Frameworks/Sage.framework/Versions/9.3/local/bin/python3.9
Identifier=python3
Format=Mach-O thin (x86_64)
CodeDirectory v=20500 size=627 flags=0x10000(runtime) hashes=9+7 location=embedded
Signature size=9052
Timestamp=Jun 22, 2021 at 14:39:32
Info.plist=not bound
TeamIdentifier=A3LBEGBB69
Runtime Version=11.3.0
Sealed Resources=none
Internal requirements count=1 size=168
qq<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
</dict>
</plist>

☝️ I'm not sure how to interpret that. (Code signing details are a bit opaque to me.)

As a wild guess, I think that giving /bin/sh full disk access might cause the app to inherit full disk access.

My guess is that this would effectively open up full disk access to (almost) everything (or certainly more than what is intended).

---

UPDATE: I was able to add the embedded python3.9 binary to the Full Disk Access portion of the System Preferences panel via drag-on-drop, but that didn't solve the problem.

[culler]

I saw that. But did you try granting permission to /bin/sh?

I think that might actually work whereas granting access to python would not work because the system is not smart enough to recognize that the main executable eventually gets run by python after it forks. It can only detect that the main executable initially is run by bin/sh (before the fork).

On Sun, Jun 27, 2021 at 12:04 PM Matt Bogosian @.***> wrote:

I suspect that the reason the SageMath app cannot be directly given full disk access is that its main executable is actually a shell script which execs python3 to run itself as a python script. As a wild guess, I think that giving /bin/sh full disk access might cause the app to inherit full disk access.

Yeah, that's why I looked into granting full disk access to the embedded python3.9 binary in particular:

$ file -h SageMath-9-3.app/Contents/Frameworks/Sage.framework/Versions/9.3/local/bin/python3.9 SageMath-9-3.app/Contents/Frameworks/Sage.framework/Versions/9.3/local/bin/python3.9: Mach-O 64-bit executable x86_64

(See my earlier comment <#m_-8297335885466527522_issuecomment-869073868> with a screenshot.)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/3-manifolds/Sage_macOS/issues/15#issuecomment-869194982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ6CP33AMH73E4CVIA5LGTTU5KZLANCNFSM47LZIGLQ .

[culler]

Incidentally, the disable-library-validation entitlement is used in order to allow loading unsigned python extension modules. Without that entitlement, Apple says that the "[Hardened Runtime] feature prevents a program from loading frameworks, plug-ins, or libraries unless they’re either signed by Apple or signed with the same Team ID as the main executable".

[posita]

Interesting. Can one change that parameter without resigning?

[culler]

As far as I know, nothing can be changed without resigning.

On Mon, Jun 28, 2021 at 12:27 PM Matt Bogosian @.***> wrote:

Interesting. Can one change that parameter without resigning?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/3-manifolds/Sage_macOS/issues/15#issuecomment-869872129, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ6CP673REFURWKK6WVBX3TVCWGRANCNFSM47LZIGLQ .

[culler]

If you want to experiment with code signing (experimentation being essentially the only way to find out anything about how it works) you can create a self-signed certificate.

[culler]

I retested this using your tinyec example, and it now works fine to import tinyec. The difference is that the main executable is no longer a shell script. That has been the case for a few releases now. I am sorry that I forgot to recheck this issue when I made that change.

I think I can close this now, though.