py_run_file_impl() crashing since .v1.27

[DavorJ]

> reticulate::py_config()

python:         [conda_path]/x64/envs/[conda_env]/python.exe
libpython:      [conda_path]/x64/envs/[conda_env]/python39.dll
pythonhome:     [conda_path]/x64/envs/[conda_env]
version:        3.9.17 (main, Jul  5 2023, 21:22:06) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          [conda_path]/x64/envs/[conda_env]/Lib/site-packages/numpy
numpy_version:  1.25.0

NOTE: Python version was forced by RETICULATE_PYTHON

The RETICULATE_PYTHON is pointing to python.exe in a conda environment. After that reticulate::source_python('file.py') is called and R crashes. The file can be empty. This happens since v1.27 and above (tested till v1.30). The crash happens both on console, as in RStudio. v1.26 works fine.

> sessionInfo()

R version 3.6.3 (2020-02-29)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=Dutch_Belgium.1252  LC_CTYPE=Dutch_Belgium.1252    LC_MONETARY=Dutch_Belgium.1252
[4] LC_NUMERIC=C                   LC_TIME=Dutch_Belgium.1252    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] assertthat_0.2.1

loaded via a namespace (and not attached):
 [1] reticulate_1.26    tidyselect_1.1.1   xfun_0.31          shinyjs_2.0.0      ks_1.13.5         
 [6] purrr_0.3.4        lattice_0.20-38    colorspace_2.0-2   vctrs_0.3.8        generics_0.1.1    
[11] testthat_3.1.1     viridisLite_0.4.0  htmltools_0.5.2    plotly_4.10.0      pracma_2.3.8      
[16] utf8_1.2.2         rlang_1.0.4        later_1.3.0        pillar_1.6.4       glue_1.6.2        
[21] DBI_1.1.1          xgboost_1.4.1.1    bit64_4.0.5        HDInterval_0.2.2   matrixStats_0.62.0
[26] lifecycle_1.0.1    stringr_1.4.0      munsell_0.5.0      gtable_0.3.0       htmlwidgets_1.5.4 
[31] mvtnorm_1.1-3      knitr_1.36         fastmap_1.1.0      fansi_0.5.0        Rcpp_1.0.7        
[36] KernSmooth_2.23-16 DT_0.20            renv_0.17.3        scales_1.1.1       jsonlite_1.7.2    
[41] fs_1.5.2           bit_4.0.4          ggplot2_3.3.5      png_0.1-8          digest_0.6.28     
[46] stringi_1.6.1      dplyr_1.0.7        grid_3.6.3         cli_3.3.0          tools_3.6.3       
[51] magrittr_2.0.1     lazyeval_0.2.2     tibble_3.1.5       tidyr_1.1.4        crayon_1.4.1      
[56] pkgconfig_2.0.3    shrtcts_0.1.2      ellipsis_0.3.2     Matrix_1.2-18      data.table_1.14.8 
[61] xml2_1.3.3         roxygen2_7.1.2     httr_1.4.2         rstudioapi_0.13    R6_2.5.1          
[66] mclust_5.4.10      intervals_0.15.4   compiler_3.6.3    

I have traced it down to py_run_file_impl(). Here is a compare between v1.26 and v1.27. I don't immediately see what is causing the crash:

[t-kalinowski]

Is there any output before the session crashes? (perhaps there might be some additional output if you run this outside the RStudio IDE?)

Can you reproduce this outside of conda? (i.e., if you install python from www.python.org or using reticulate::install_python())

[DavorJ]

Is there any output before the session crashes? (perhaps there might be some additional output if you run this outside the RStudio IDE?)

No, nothing.

Can you reproduce this outside of conda? (i.e., if you install python from www.python.org or using reticulate::install_python())

I have only a conda installation available currently and I tried multiple python versions and changing MKL to openblas for numpy, without effect.

[t-kalinowski]

I can't reproduce this locally. Can you provide any additional information?

• What is the file path? Maybe the path has some special characters and there is an issue with encoding?

• Is this a conda binary incompatability issue? Can you please try and see if this issue still happens with a conventional Python binary from www.python.org/downloads or reticulate::install_python()?

[genobobeno]

I'm having the same problem on Windows 10. I'm running R 4.1.2.

> library(reticulate)
> reticulate::use_python("C:/Users/eugen/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe")
> py_config()
python:         C:/Users/eugen/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe
libpython:      C:/Users/eugen/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python39.dll
pythonhome:     C:/Users/eugen/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13
version:        3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]

NOTE: Python version was forced by use_python() function

I've already attempted to remediate this unknown "fatal error" crash (with no details coming from anywhere) by updating reticulate to the dev version, updating my Rstudio, and running reticulate::install_python() and forcing the new installation to be sourced by reticulate as shown above. I've also installed numpy and pandas using py_install() but somehow, it's not finding numpy.

The moment I attempt to "source_python()", there is a bomb, fatal crash of R. I also have traced the failure to the py_run_file_impl().

The file I'm attempting to source is a simple import and declaration of a function:

import pandas as pd
from datetime import datetime
from datetime import timedelta
import numpy as np

def add_columns(col_config_file,df_input):

   df_csv_columns = pd.read_csv(col_config_file)
   csv_col_list = df_csv_columns['column_name'].tolist()

   s = set(df_input.columns)
   new_cols = [x for x in csv_col_list if x not in s]
   if len(new_cols) > 0:
      df_input[new_cols] = pd.DataFrame([[np.nan] * len(new_cols)], index=df_input.index)

   return df_input

[t-kalinowski]

What is the name of the python file?

[genobobeno]

What is the name of the python file?

add_columns.py

[genobobeno]

I've now removed all Python installations, and attempted another reticulate driven install of miniconda, created a conda environment, installed numpy, scipy, and pandas, and then attempted source_python() and the exact same blow up

[t-kalinowski]

what is the output from sessionInfo() ?

[t-kalinowski]

I tried again but am unable to reproduce:

[t-kalinowski]

What is the file path? Maybe the path has some special characters and there is an issue with encoding?

Can you please provide the full file path when you do normalizePath("add_columns.py")?

[genobobeno]

"C:/Documents and Settings/eugen/Documents/JobRepo/Bitscopic/pre_covid19_clustering_workflow/python/add_columns.py"

Still bombs

Normalize path gives:

"C:\\Users\\eugen\\Documents\\JobRepo\\Bitscopic\\pre_covid19_clustering_workflow\\python\\add_columns.py"

It would be really nice if I could figure out how to monitor what is breaking. The logs aren't showing any errors. It just crashes. There is code that I haven't touched or edited at all that is bombing since I updated Rstudio. With all the people that work on python, it blows my mind how terrible the error handling is. Nevermind the terrible difficulty it is to manage python errors while working with Rstudio. I have no clue how to troubleshoot this except to blow away my entire computer's software installations and reinstall everything and start from scratch... in fact, I've already tried this with the python install, uninstalling miniconda, destroying all folders in my documents, updating to the dev version of reticulate, allowing reticulate to manage the re-install of miniconda. Still, source_python() bombs everytime.

[t-kalinowski]

what is the output from sessionInfo() ?

[genobobeno]

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.30-9000

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.9       rstudioapi_0.14  magrittr_2.0.3   rappdirs_0.3.3   tidyselect_1.2.0 lattice_0.20-45  R6_2.5.1         rlang_1.0.6     
 [9] fansi_1.0.4      dplyr_1.0.10     tools_4.1.2      grid_4.1.2       png_0.1-8        utf8_1.2.2       cli_3.6.0        DBI_1.1.3       
[17] withr_2.5.0      assertthat_0.2.1 httpcode_0.3.0   tibble_3.1.8     lifecycle_1.0.3  Matrix_1.5-3     purrr_1.0.1      tidyr_1.3.0     
[25] vctrs_0.5.2      curl_5.0.0       crul_1.3         glue_1.6.2       compiler_4.1.2   pillar_1.8.1     generics_0.1.3   jsonlite_1.8.7  
[33] pkgconfig_2.0.3 

[t-kalinowski]

Can you please update Rcpp? (probably a good idea to update any other outdated packages as well... update.packages())

[t-kalinowski]

I thought it might be related to the UNWIND_PROTECT changes with Rcpp 1.0.10. But after rolling back Rcpp to 1.0.7, I still can't reproduce the issue.

[genobobeno]

Don't know what to say. I updated Rcpp. When the rsession restarts by invoking Rstudio, the following error shows up:

Error in python_config_impl(python) : Error 103 occurred running C:/Users/eugen/Documents/.virtualenvs/r-reticulate/Scripts/python.exe: Calls: do.call ... <Anonymous> -> <Anonymous> -> python_config -> python_config_impl In addition: Warning message: In system2(command = python, args = shQuote(script), stdout = TRUE, : running command '"C:/Users/eugen/Documents/.virtualenvs/r-reticulate/Scripts/python.exe" "C:/Users/eugen/Documents/R/win-library/4.1/reticulate/config/config.py"' had status 103

Still can't do anything without a bomb. Still have no way of figuring out what is causing this bomb. I can begin a console session, I can call the reticulate library, I can tell reticulate to use the r-reticulate conda environment, and I can successfully run py_config() ... I can even run python code. But the moment I attempt to source a file, BOOM! BOMB! I guess I have to do my favorite thing and blow away my entire machine and re-install everything from top to bottom. Fantastic.

[t-kalinowski]

Before you do all that, can you please

1) Reinstall the development version of reticulate 2) recreate the r-reticulate virtual environment

update.packages(check = TRUE, ask = FALSE)
remotes::install_github("rstudio/reticulate", force = TRUE)
reticulate::virtualenv_create(force = TRUE)

[xiaodfeng]

I came across similar situation. I guess the error comes from the UNC paths. See the errors below: In addition: Warning messages: 1: In normalizePath(path.expand(path), winslash, mustWork) : path[1]="\\nki-home": The specified path is invalid 2: In normalizePath(path.expand(path), winslash, mustWork) : path[1]="\\nki-home": The specified path is invalid

[t-kalinowski]

I tried a local UNC path and still can't reproduce the error. Note, afaik, normalizePath() is not called in py_run_file(), only path.expand() is. @xiaodfeng the warning you're seeing must be coming from somewhere else.

[xiaodfeng]

Thank you for the quick reply. I tried the commands that you have suggested above, the details are shown below:

update.packages(check = TRUE, ask = FALSE)

It works for updating the packages.

remotes::install_github("rstudio/reticulate", force = TRUE)

It updated the reticulate into version ‘1.31.0.9000’.

reticulate::install_python()

Cloning into 'C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv'...
Already up to date.
:: [Info] ::  Mirror: https://www.python.org/ftp/python
:: [Info] ::  Scanned 200 pages and found 650 installers.
+ "C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/bin/pyenv" install 3.9.13
:: [Info] ::  Mirror: https://www.python.org/ftp/python
:: [Downloading] ::  3.9.13 ...
:: [Downloading] ::  From https://www.python.org/ftp/python/3.9.13/python-3.9.13-amd64.exe
:: [Downloading] ::  To   C:\Users\XA250~1.FEN\AppData\Local\R-RETI~1\R-RETI~1\pyenv\PYENV-~1\install_cache\python-3.9.13-amd64.exe
:: [Installing] ::  3.9.13 ...
:: [Info] :: completed! 3.9.13
[1] "C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe"

reticulate::virtualenv_create(force = TRUE)

Using Python: C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe
Creating virtual environment "~/.virtualenvs/r-reticulate" ... 
+ "C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe" -m venv "\\nki-home\users\x.feng\Documents/.virtualenvs/r-reticulate"
Done!
Installing packages: pip, wheel, setuptools
+ "\\nki-home\users\x.feng\Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -m pip install --upgrade pip wheel setuptools
Error: Error installing package(s): "pip", "wheel", "setuptools"`

reticulate::source_python("flights.py")

`Error in system2(command = python, args = shQuote(script), stdout = TRUE,  : 
'CreateProcess' failed to run '
 "D:/Program Files/R/R-4.1.2/library/reticulate/config/config.py"' "\\nki-home/users/x.feng/Documents/.virtualenvs/r-reticulate/Scripts/python.exe"
Error: Installation of Python not found, Python bindings not loaded.
Please create a default virtual environment with `reticulate::virtualenv_create('r-reticulate')`.`

sessionInfo()

R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8             here_1.0.1             lattice_0.20-45        png_0.1-8              ps_1.7.5               prettyunits_1.1.1      withr_2.5.0            crayon_1.5.2          
 [9] rprojroot_2.0.3        rappdirs_0.3.3         grid_4.1.2             R6_2.5.1               jsonlite_1.8.4         rlang_1.1.0            cli_3.6.1              curl_5.0.0            
[17] remotes_2.4.2.1        callr_3.7.3            Matrix_1.3-4           reticulate_1.31.0.9000 desc_1.4.2             tools_4.1.2            compiler_4.1.2         processx_3.8.1        
[25] pkgbuild_1.4.2 

[xiaodfeng]

It is worth to mention that the reticulate::repl_python() and reticulate::import() function works, only the reticulate::source_python() function will crash the system. See details below:

Miniconda has been successfully installed at "C:/Users/x.feng/AppData/Local/r-miniconda". [1] ``"C:/Users/x.feng/AppData/Local/r-miniconda"

reticulate::repl_python()

Python 3.9.18 (C:/Users/x.feng/AppData/Local/r-miniconda/envs/r-reticulate/python.exe) Reticulate 1.31.0.9000 REPL -- A Python interpreter in R. Enter 'exit' or 'quit' to exit the REPL and return to R.

print("hello, world")

hello, world

[t-kalinowski]

It looks like creating the virtualenv_create() call failed, but the reason is unclear. I tried locally on a windows machine with a UNC path and it succeeded without error, so I don't think it's due to the UNC path. Every subsequent error is due to the first error, where these system command failed. You can try in cmd.exe running these two commands and seeing if they succeed, just to rule out any other reticulate related bugs.

"C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe" -m venv "\\nki-home\users\x.feng\Documents/.virtualenvs/r-reticulate"

"\\nki-home\users\x.feng\Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -m pip install --upgrade pip wheel setuptools

[xiaodfeng]

The suggested two commands do not work. See details below.

D:\Program Files>"C:/Users/x.feng/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python.exe" -m venv "\\nki-home\users\x.feng\Documents/.virtualenvs/r-reticulate"
The system cannot find the path specified.
D:\Program Files>"\\nki-home\users\x.feng\Documents/.virtualenvs/r-reticulate/Scripts/python.exe" -m pip install --upgrade pip wheel setuptools
The system cannot find the path specified.

Thus, I re-install the r-reticulate environment using conda, it works. See details below.

D:\Program Files>"C:/Users/x.feng/AppData/Local/r-miniconda/envs/r-reticulate/python.exe" -m pip install --upgrade pip wheel setuptools
Requirement already satisfied: pip in c:\users\x.feng\appdata\local\r-miniconda\envs\r-reticulate\lib\site-packages (23.2.1)
Requirement already satisfied: wheel in c:\users\x.feng\appdata\local\r-miniconda\envs\r-reticulate\lib\site-packages (0.41.2)
Requirement already satisfied: setuptools in c:\users\x.feng\appdata\local\r-miniconda\envs\r-reticulate\lib\site-packages (68.1.2)
Collecting setuptools
  Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
  Downloading setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Downloading setuptools-68.2.2-py3-none-any.whl (807 kB)
   ---------------------------------------- 807.9/807.9 kB 3.2 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 68.1.2
    Uninstalling setuptools-68.1.2:
      Successfully uninstalled setuptools-68.1.2
Successfully installed setuptools-68.2.2

However, reticulate::source_python() function still crashes the system