reticulate:::initialize_python() kills R outside R studio

courtiol commented 4 years ago

Hi all, sorry if the issue exists already (could not spot it).

On 2 different computers on MacOS managed by 2 different users (me and @lizimai), reticulate:::initialize_python() creates R to crash (=kill) with R from r.app, R in the console, R in RStudio accessed through devtools menu (e.g. through test() or check()) but NOT in the R console inside RStudio.

Example in RStudio:

In R from terminal (using Rscript here but also using R properly):

Running the problematic call directly, I could verify this behaviour using the same arguments everywhere. Indeed, this

.Call(reticulate:::`_reticulate_py_initialize`,
      "/Users/alex/Library/r-miniconda/envs/r-reticulate/bin/python",
      "/Users/alex/Library/r-miniconda/envs/r-reticulate/lib/libpython3.6m.dylib",
      "/Users/alex/Library/r-miniconda/envs/r-reticulate:/Users/alex/Library/r-miniconda/envs/r-reticulate", "",
      TRUE,
      TRUE,
      "")

crashes everywhere except in the R console within RStudio.

I replicate this with both the current devel version (1.15-9000) of reticulate and the CRAN version.

Any idea what could be going on?

courtiol commented 4 years ago

Doing some Rcheology shows that the problem disappears with version of reticulate < 1.14 (1.10, 1.11, 1.12, 1.13 works, but not 1.14 or 1.15).

kevinushey commented 4 years ago

If you haven't already, it would be worth trying to reinstall both Rcpp and reticulate together.

courtiol commented 4 years ago

I did try, also with older versions of Rcpp and devel... no luck. Only downgrading reticulate to 1.13 or earlier does the job. Rcpp does not seem to be the issue. Yet, I have used reticulate until a week ago without issues and I usually have all my packages up to date. I don't get it...

courtiol commented 4 years ago

I have installed R 4.0 and took the opportunity to reinstall all pkgs from scratch -> same issue... :-( @kevinushey, is there anything I could try? If it helps you, I am happy you to investigate the case directly on my laptop (I discovered that Zoom makes this easy...).

skeydan commented 4 years ago

Salut Alexandre, if you want, we can try this later today (like, 4pm or so?), although I'm of course not @kevinushey :-) (but at least we're in the same time zone)

LMK if you'd like to (if so mail to my first name @ rstudio.com)

courtiol commented 4 years ago

@skeydan I managed to make gdb work, but then when running reticulate:::initialize_python() and running bt after the crash I get something not very informative:

[New Thread 0x262f of process 39180]

Thread 2 received signal SIGKILL, Killed.
0x0000000100059430 in ?? ()
(gdb) bt
#0  0x0000000100059430 in ?? ()
#1  0x00007ffeefbe9e80 in ?? ()
#2  0x00000001000205ac in ?? ()
#3  0x00007ffeefbea460 in ?? ()
#4  0x0000000000000004 in ?? ()
#5  0x000000010009e268 in ?? ()
#6  0x0000000107da83b0 in ?? ()
#7  0x00007ffeefbeabf0 in ?? ()
#8  0x0000000000000000 in ?? ()

Any idea?

skeydan commented 4 years ago

yeah... you should have seen several messages saying don't have debug symbols for ... when starting lldb. Can you install all those (I remember these included R, but there will be others as well) and try again?

courtiol commented 4 years ago

Hmmm... would love to but i have no idea: it tells me

GNU gdb (GDB) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin19.3.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /Library/Frameworks/R.framework/Resources/bin/exec/R...

warning: `/Volumes/Builds/Simon/R4/high-sierra-x86_64/R-4.0.0/src/main/Rmain.o': can't open to read symbols: No such file or directory.
(No debugging symbols found in /Library/Frameworks/R.framework/Resources/bin/exec/R)
(gdb)

And I am not surprise he does not find the file with the symbols because I am not on Simon Urbanek computer. This famous Simon who shows up here is the one that compiled the R binary for MacOS. Does it mean I need to compile R from source to solve the issue and is there a tweak to add debugging symbols post-hoc? Sorry, I am new to gdb (I am not using lldb by the way, but gdb)...

skeydan commented 4 years ago

hm, I see ... well... then perhaps it's best to ask @kevinushey how other Mac users do it (obtain the debug version) :-)

kevinushey commented 4 years ago

I think you need to use lldb rather than gdb on macOS. Note that you either need to disable SIP to get that working (https://www.imore.com/how-turn-system-integrity-protection-macos) or install a non-notarized version of R 4.0.0. Unfortunately I can't find links for that on https://cran.r-project.org/bin/macosx/...

courtiol commented 4 years ago

Thanks @kevinushey & @skeydan, I can make lldb work (yay!) but then the issue disappear because... ... !!! disabling SIP in itself solves this reticulate issue !!! :-(

I tested a couple of times and it seems reproducible:

when I deactivate SIP, the issue disappear.
when I reactivate SIP, the issue reappear.

You may be able to solve that in reticulate since old version do work...

Otherwise, do you think using a non-notarized version of R is the cleanest workaround? If so, how do I create that, by compiling R from source with some special flags?

I wonder also if changing some security setting specifically for R could help. I tried, but it did not work. Yet if RStudio can do it, they must be a way...

In any case, if your SIP is off, perhaps this is why you did not see this on your system.

In hindsight that explains why I had no problem before, despite already using the version 1.14 which is now problematic, an MacOS update must have triggered the problem.

Here are some info on my test system (I can provide more upon request):

MacBook Pro (Retina, 13-inch, Early 2015)
R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.4
reticulate 1.15
Rcpp 1.0.4.6

Ping @lizimai

kevinushey commented 4 years ago

Yikes!! That is not expected. I wonder if R or Python are built / notarized with restrictions that prevent us from using Python in the way we do?

I'll have to re-enable SIP and see if there's anything we can do here. Indeed, the simplest workaround would be to use a non-notarized version of R, either built from sources yourself or an official non-notarized build from CRAN. However, I'm not sure if those are available yet :-/

I will try to find time to see if I can reproduce this issue with SIP enabled with R 4.0.0.

vjcitn commented 4 years ago

A non-notarized version of R is at http://mac.r-project.org/high-sierra/R-4.0-branch/x86_64/R-4.0-branch.tar.gz, mentioned at https://cran.r-project.org/bin/macosx/RMacOSX-FAQ.html 10.17

vjcitn commented 4 years ago

Only with that version have we been able to run python modules obtained from a miniconda installation. This has been an issue for the Bioconductor basilisk package by @LTLA.

courtiol commented 4 years ago

Many thanks for the tip @vjcitn (I will try installing this R version, despite using MacOS 10.15 and not 10.13), but I am hoping that @kevinushey will find a more user-friendly solution. I am optimistic because in RStudio's console it does work, so there seem to be a way to make it work with a notarized version of R.

kevinushey commented 4 years ago

I vaguely recall it being possible to see the actual issue logged in Console.app, although I forget exactly how to find it (filtering through the various logs on macOS is kind of challenging). But it should at least tell you which entitlement was missing.

courtiol commented 4 years ago

It must be that log file: crash_log_reticulate.txt

kevinushey commented 4 years ago

Thanks! Copying the relevant bits of the crash log:

Process:               R [940]
Path:                  /Library/Frameworks/R.framework/Versions/4.0/Resources/bin/exec/R
Identifier:            R
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        zsh [530]
Responsible:           iTerm2 [416]
User ID:               501

Date/Time:             2020-04-29 21:28:17.503 +0200
OS Version:            Mac OS X 10.15.4 (19E287)
Report Version:        12
Anonymous UUID:        4941C822-484D-9E28-E671-F3E72E19DFCE

Time Awake Since Boot: 520 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (Code Signature Invalid)
Exception Codes:       0x0000000000000032, 0x00000001106c3000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 0x2

kernel messages:

VM Regions Near 0x1106c3000:
    MALLOC_LARGE           00000001106a5000-00000001106c3000 [  120K] rw-/rwx SM=PRV  
--> mapped file            00000001106c3000-00000001106c4000 [    4K] r--/r-x SM=PRV  
    MALLOC_LARGE           00000001106c6000-00000001106e7000 [  132K] rw-/rwx SM=PRV  

Application Specific Information:
dyld: in dlopen()
/Users/alex/Library/r-miniconda/envs/r-reticulate/lib/libmkl_intel_thread.dylib

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   dyld                            0x000000011c2f6430 bcmp + 208
1   dyld                            0x000000011c2bd5ac ImageLoaderMachO::validateFirstPages(linkedit_data_command const*, int, unsigned char const*, unsigned long, long long, ImageLoader::LinkContext const&) + 104
2   dyld                            0x000000011c2c3446 ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, encryption_info_command const*, ImageLoader::LinkContext const&) + 310
3   dyld                            0x000000011c2bc638 ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) + 158
4   dyld                            0x000000011c2a7e23 dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) + 728
5   dyld                            0x000000011c2afa7f dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) + 1422
6   dyld                            0x000000011c2af48f dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) + 185
7   dyld                            0x000000011c2af1f9 dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) + 1435
8   dyld                            0x000000011c2ae8b3 dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) + 183
9   dyld                            0x000000011c2a79bb dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, unsigned int&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) + 487
10  dyld                            0x000000011c2a7578 dyld::load(char const*, dyld::LoadContext const&, unsigned int&) + 185
11  dyld                            0x000000011c2b3d3d dlopen_internal + 419
12  libdyld.dylib                   0x00007fff7320fd8a dlopen + 171
13  libmkl_rt.dylib                 0x0000000111f76b50 mkl_sdl_load_function + 912
14  libmkl_rt.dylib                 0x0000000111f73329 MKL_Get_Version + 41
< ... >

In other words, the application is crashing because the code signature in the version of Intel MKL being loaded is invalid.

The version I have in my installation appears valid:

$ codesign -dv --verbose=4 libmkl_intel_thread.dylib
Executable=/Users/kevinushey/opt/anaconda3/pkgs/mkl-2019.4-233/lib/libmkl_intel_thread.dylib
Identifier=libmkl_intel_thread.dylib
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=227434 flags=0x0(none) hashes=11366+2 location=embedded
VersionPlatform=1
VersionMin=658432
VersionSDK=658432
Hash type=sha1 size=20
CandidateCDHash sha1=c053e187127be4c4c960bef56ff48e39a49da6a9
CandidateCDHashFull sha1=c053e187127be4c4c960bef56ff48e39a49da6a9
Hash choices=sha1
CMSDigest=e6ede1882aab91aa98810691d47c464a627185153a46e5ba30255f704224a81d
CMSDigestType=2
Page size=4096
CDHash=c053e187127be4c4c960bef56ff48e39a49da6a9
Signature size=8590
Authority=Developer ID Application: Intel Corporation Apps (Z3L495V9L4)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Apr 11, 2019 at 11:07:01 AM
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=2 size=176

Can you share the output of:

codesign -dv --verbose=4 /Users/alex/Library/r-miniconda/envs/r-reticulate/lib/libmkl_intel_thread.dylib

for posterity?

kevinushey commented 4 years ago

You might also need to ensure that Conda / the MKL package is up to date.

courtiol commented 4 years ago

alex@alexandres-mbp Desktop % codesign -dv --verbose=4 /Users/alex/Library/r-miniconda/envs/r-reticulate/lib/libmkl_intel_thread.dylib
Executable=/Users/alex/Library/r-miniconda/envs/r-reticulate/lib/libmkl_intel_thread.dylib
Identifier=libmkl_intel_thread.dylib
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=227434 flags=0x0(none) hashes=11366+2 location=embedded
VersionPlatform=1
VersionMin=658432
VersionSDK=658432
Hash type=sha1 size=20
CandidateCDHash sha1=c053e187127be4c4c960bef56ff48e39a49da6a9
CandidateCDHashFull sha1=c053e187127be4c4c960bef56ff48e39a49da6a9
Hash choices=sha1
CMSDigest=e6ede1882aab91aa98810691d47c464a627185153a46e5ba30255f704224a81d
CMSDigestType=2
Page size=4096
CDHash=c053e187127be4c4c960bef56ff48e39a49da6a9
Signature size=8590
Authority=Developer ID Application: Intel Corporation Apps (Z3L495V9L4)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Apr 11, 2019 at 20:07:01
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=2 size=176

courtiol commented 4 years ago

I only have MKL and Conda through r-miniconda as automatically installed by RStudio... I tried that to update, but it did not update Conda or MKL nor improved the situation:

> reticulate::miniconda_update()
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/alex/Library/r-miniconda

  added / updated specs:
    - conda

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libffi-3.2.1               |       h0a44026_6          43 KB
    ncurses-6.2                |       h0a44026_1         749 KB
    sqlite-3.31.1              |       h5c1f38d_1         2.4 MB
    tqdm-4.45.0                |             py_0          58 KB
    ------------------------------------------------------------
                                           Total:         3.3 MB

The following packages will be UPDATED:

  libffi                                   3.2.1-h475c297_4 --> 3.2.1-h0a44026_6
  ncurses                                    6.2-h0a44026_0 --> 6.2-h0a44026_1
  openssl                                 1.1.1f-h1de35cc_0 --> 1.1.1g-h1de35cc_0
  sqlite                                  3.31.1-ha441bb4_0 --> 3.31.1-h5c1f38d_1
  tqdm                                          4.44.1-py_0 --> 4.45.0-py_0

Downloading and Extracting Packages
sqlite-3.31.1        | 2.4 MB    | ########## | 100% 
libffi-3.2.1         | 43 KB     | ########## | 100% 
ncurses-6.2          | 749 KB    | ########## | 100% 
tqdm-4.45.0          | 58 KB     | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

kevinushey commented 4 years ago

The codesign output looks the same as what I've got as well. :-/

Do you have the latest version of Xcode / Apple command line tools installed?

courtiol commented 4 years ago

I think so: I have XCode Version 11.4.1 (11E503a) which seems very recent. Cannot you replicate the issue with SIP on?

LTLA commented 4 years ago

I'll just mention that nomkl'ing the Conda installation seems to solve the issue, at least on the Bioconductor build system as well as @vjcitn's machine; this is basilisk's current workaround.

webbedfeet commented 4 years ago

I've also found this same issue running on Mac OS Catalina. I am running bookdown on a set of Rmd files that have python chunks, and it works fine within RStudio but not using Rscript or in R running from a terminal. In fact, reticulate:::initialize_python() causes non-RStudio R to crash. If I disable the SIP, it works. Of note, I do have the notarized version of R installed.

I'm running: R version 3.6.3 reticulate v. 1.15 Anaconda Python with python 3.7 Xcode Version 11.5 (11E608c)

slowkow commented 4 years ago

I found this issue by searching for the libmkl_intel_thread.dylib message in the Crash Logs from the macOS Console app.

I noticed that running mv ~/miniconda3{,.bak} successfully prevents R from finding MKL, so I do not get the Killed: 9 error anymore.

This worked for me to avoid getting more of these Killed: 9 errors when I'm building various R packages (thanks to Cami Williams).

conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service

This thread might help to learn more about what is going on:

https://github.com/ContinuumIO/anaconda-issues/issues/6423

mufernando commented 3 years ago

I think I'm running into the same issue. Reticulate works fine in RStudio but when I tried running with Rscript/R it fails:

> library(reticulate)
> use_condaenv("greatapes")
use_python("/opt/anaconda3/envs/greatapes/bin/python")
> use_python("/opt/anaconda3/envs/greatapes/bin/python")
> np <- reticulate::import("numpy", convert=T)
[1]    5016 killed     R

vjcitn commented 3 years ago

sessionInfo()? haven't seen this in quite some time.

mufernando commented 3 years ago

I found this issue by searching for the libmkl_intel_thread.dylib message in the Crash Logs from the macOS Console app.

I noticed that running mv ~/miniconda3{,.bak} successfully prevents R from finding MKL, so I do not get the Killed: 9 error anymore.

This worked for me to avoid getting more of these Killed: 9 errors when I'm building various R packages (thanks to Cami Williams).
conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service
This thread might help to learn more about what is going on:

ContinuumIO/anaconda-issues#6423

I just did what this person recommended and it worked!

aaronjfisher commented 3 years ago

I found this issue by searching for the libmkl_intel_thread.dylib message in the Crash Logs from the macOS Console app.

I noticed that running mv ~/miniconda3{,.bak} successfully prevents R from finding MKL, so I do not get the Killed: 9 error anymore.

This worked for me to avoid getting more of these Killed: 9 errors when I'm building various R packages (thanks to Cami Williams).
conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service
This thread might help to learn more about what is going on:

ContinuumIO/anaconda-issues#6423

I have been having the same issue, no bugs in Rstudio, but crashes when loading conda versions of python in the terminal or R GUI.

This fix does work for me too, but the reason I'm using python is for PyTorch, which depends on mkl, mkl-service. Is there a way that I can do a similar fix, but still be able to use PyTorch? (I'm not an expert on these kinds of things, so I apologize if the answer is hiding in plains sight.)

Thanks so much regardless!

rstudio / reticulate

reticulate:::initialize_python() kills R outside R studio #758