package fails to load if python dependencies can't be installed on Windows

[Phenomniverse]

Hi Ethan,

Not sure what has happened because this was working previously, but now I'm not able to load the chromConverter package.

From the error message I gather that there are some issues with the python package 'python-lzf' installation process, looks like it needs an install of Microsoft Visual C++, and also I also see this warning:

SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

I'll try manually installing the build tools for Microsoft Visual C++ and see if that resolves the issue, but in the mean time, here's the full output of library(chromConverter):

> library(chromConverter)
Configuring package 'chromConverter': please wait ...

C:\users>CALL "C:\Users\regan\AppData\Local\r-miniconda\condabin\activate.bat" "C:\Users\regan\AppData\Local\r-miniconda\envs\r-reticulate" 

C:\users>conda.bat activate "C:\Users\regan\AppData\Local\r-miniconda\envs\r-reticulate" 

(r-reticulate) C:\users>"C:/Users/regan/AppData/Local/r-miniconda/envs/r-reticulate/python.exe" -m pip install --upgrade --no-user "aston" "numpy" "pandas" "rainbow-api" "scipy" 
Collecting aston
  Using cached Aston-0.7.1-py3-none-any.whl (74 kB)
Requirement already satisfied: numpy in c:\users\regan\appdata\local\r-miniconda\envs\r-reticulate\lib\site-packages (1.23.5)
Collecting pandas
  Using cached pandas-1.5.2-cp38-cp38-win_amd64.whl (11.0 MB)
Collecting rainbow-api
  Using cached rainbow_api-1.0.1-py3-none-any.whl (21 kB)
Collecting scipy
  Using cached scipy-1.9.3-cp38-cp38-win_amd64.whl (39.8 MB)
Requirement already satisfied: pytz>=2020.1 in c:\users\regan\appdata\local\r-miniconda\envs\r-reticulate\lib\site-packages (from pandas) (2022.6)
Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting python-lzf
  Using cached python-lzf-0.2.4.tar.gz (9.3 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting matplotlib
  Using cached matplotlib-3.6.2-cp38-cp38-win_amd64.whl (7.2 MB)
Collecting lxml
  Using cached lxml-4.9.1-cp38-cp38-win_amd64.whl (3.6 MB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.4.4-cp38-cp38-win_amd64.whl (55 kB)
Collecting pillow>=6.2.0
  Using cached Pillow-9.3.0-cp38-cp38-win_amd64.whl (2.5 MB)
Collecting fonttools>=4.22.0
  Using cached fonttools-4.38.0-py3-none-any.whl (965 kB)
Collecting pyparsing>=2.2.1
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting contourpy>=1.0.1
  Using cached contourpy-1.0.6-cp38-cp38-win_amd64.whl (163 kB)
Collecting packaging>=20.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Building wheels for collected packages: python-lzf
  Building wheel for python-lzf (setup.py): started
  Building wheel for python-lzf (setup.py): finished with status 'error'
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [5 lines of output]
      running bdist_wheel
      running build
      running build_ext
      building 'lzf' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-lzf
  Running setup.py clean for python-lzf
Failed to build python-lzf
Installing collected packages: python-lzf, six, scipy, pyparsing, pillow, lxml, kiwisolver, fonttools, cycler, contourpy, python-dateutil, packaging, aston, pandas, matplotlib, rainbow-api
  Running setup.py install for python-lzf: started
  Running setup.py install for python-lzf: finished with status 'error'
  error: subprocess-exited-with-error

  × Running setup.py install for python-lzf did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      running install
      C:\Users\regan\AppData\Local\r-miniconda\envs\r-reticulate\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_ext
      building 'lzf' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> python-lzf

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
Error: package or namespace load failed for ‘chromConverter’:
 .onLoad failed in loadNamespace() for 'chromConverter', details:
  call: NULL
  error: Error installing package(s): "\"aston\"", "\"numpy\"", "\"pandas\"", "\"rainbow-api\"", "\"scipy\""
In addition: Warning message:
In shell(fi, intern = intern) :
  'C:\Users\regan\AppData\Local\Temp\RtmpINwiN5\file2c3c7541260c.bat' execution failed with error code 1

[Phenomniverse]

The generic build tools available via the link in the error message above ("Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/) doesn't fix the problem. But the build tools installer gives a lot of options for what kind of toolset you want to install. I went for a minimal install because other options use a lot of disk space and I have no idea what specifically the python-lzf package actually needs.

[ethanbass]

Hmm. That's not good. Did you update your computer or anything like that? It's strange that this would just suddenly happen (I haven't changed anything on the python side recently). I'm assuming you restarted your R session after installing the Microsoft Build Tools? If not, this would be something to try. You might also want to try removing the reticulate package and reinstalling everything in case it's corrupted somehow?

I'm pretty unhappy with the fact that the whole package errors out like this if something goes wrong with the python installation, but as far as I've been able to figure out there isn't any way of having python modules as suggested dependencies (at least for CRAN packages).The only way I know of would be to split the python parts into their own separate package and have that other package as a suggested dependency in chromConverter, but that also doesn't seem ideal.

[ethanbass]

This seems like a pretty authoritative resource: https://wiki.python.org/moin/WindowsCompilers#Which_Microsoft_Visual_C.2B-.2B-_compiler_to_use_with_a_specific_Python_version_.3F

It sounds like the trick may be to install the Windows SDK? I think this is a separate checkbox in the build tools installer, but to be honest I am somewhat mystified by Windows things, being only a reluctant and infrequent Windows user.

[ethanbass]

I think I may actually have a solution for the issue with the whole package erroring out due to python dependency problems (25235d1a489fc5bbdfa0e2863489875e755ab4be ). Would be great if you can test it for me. It won't actually help with getting the python stuff to load, but I'm hoping it can allow the package to still load without them. I just wrapped everything in the .onLoad function that imports the python modules in try.

[Phenomniverse]

I'm assuming you restarted your R session after installing the Microsoft Build Tools?

Yes I did a system reboot, but it didn't help.

It sounds like the trick may be to install the Windows SDK?

That's an extra 15.7GB of installation which is a lot on my 450GB hdd. I'm reluctant to try that unless its really, really necessary.

I think I may actually have a solution for the issue with the whole package erroring out due to python dependency problems (https://github.com/ethanbass/chromConverter/commit/25235d1a489fc5bbdfa0e2863489875e755ab4be ). Would be great if you can test it for me. It won't actually help with getting the python stuff to load, but I'm hoping it can allow the package to still load without them. I just wrapped everything in the .onLoad function that imports the python modules in try.

This doesn't seem to have worked, but just to confirm, when you reference a commit like that, will that be incorporated into a fresh install from devtools::install_github("https://github.com/ethanbass/chromConverter/")? Or is there some other procedure to test this change? I did a fresh install, and I'm seeing the same issue still. I think try() can be a bit finicky, I know I've had trouble implementing it in the past.

[ethanbass]

Yes, it should be incorporated with the fresh install, since I added it to the master branch. Just to double check, did you refresh your R session after reinstalling? I've found that sometimes the packages get cached or something and you could still be loading the old version after reinstalling if you haven't loaded a new R session. It's a little hard for me to troubleshoot the try thing myself, because I haven't thought of a way to make the python installation fail on my computer. I thought that it would work though.

[ethanbass]

I guess I might have to go uninstall the weird build tools from the windows computer in the lab. Not sure when i'll be able to get around to that though

[Phenomniverse]

Ok I tried a new r session, I'm still getting error messages, in fact there are more of them, but it does seem to have loaded the chromConverter namespace, so maybe it is working. Let me play around a bit more and I'll confirm. Here's some extra error messages that library(chromConverter) is throwing:

Error : Error installing package(s): "\"aston\"", "\"numpy\"", "\"pandas\"", "\"rainbow-api\"", "\"scipy\""
In addition: Warning message:
In shell(fi, intern = intern) :
  'C:\Users\regan\AppData\Local\Temp\Rtmpa0OZdx\file14f05f33374.bat' execution failed with error code 1
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'aston'

Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'pandas'

Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'rainbow'

Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'rainbow'

[Phenomniverse]

Confirmed, its working again at least as far as I am using it, which at the moment is just as: read_chroms("FID1A.CH",parser="chromconverter",format_in="chemstation_fid")

[ethanbass]

Ah good news! The new error messages are just because it's trying to load all the individual python modules but can't (because they didn't install). Previously, it was erroring out completely on the installation step.

I'm going to go ahead and silence the module not found errors, since they wouldn't really be needed for troubleshooting. I don't think I should silence the python installation errors because they're probably pretty important for anyone trying to troubleshoot why python isn't installing correctly.

[Phenomniverse]

Is there a way that I can pre-install the requisite python packages directly through python and make sure they're in the right environment for R to be able to find and use them?

[ethanbass]

Maybe try conda env list in the terminal and see if there's a chromConverter environment. If so you should be able to activate it and install whatever you want in there. Theoretically, chromConverter should create its own environment through miniconda if everything is working correctly, but reticulate can be kind of confusing in how it handles the environments since it's behavior is apparently somewhat dependent on how you have python configured.

By the way, the python package that's failing to install (python-lzf) is a dependency of rainbow-api

[ethanbass]

Actually it kind of seems like everything is just getting installed into the "r-reticulate" environment on my computer (and maybe on yours as well judging by the paths in the output you posted in your first message). So maybe try opening that one from the terminal and see if you can get rainbow-api to install somehow or if it errors out with the message about Visual C++

[ethanbass]

Nevermind, i'm pretty sure it is installing in the chromConverter environment on my computer as I originally thought. In conclusion however, it is very confusing, especially if you have (as I do) multiple versions of conda on your computer.

[Phenomniverse]

I did a fresh install of r-studio on a linux virtual machine, and installation of chromConverter went very smoothly on that. The biggest problem was getting devtools to install (it has a lot of system dependencies) but once I figured out that you can install devtools (and all its dependecies) from linux terminal directly with sudo apt-get install r-cran-devtools everything else worked perfectly. I'm running kali linux, but ubuntu or any other debian based linux distro should work equally well. This doesn't solve the windows problem and it will add extra difficulty for getting my data across to my virtual machine, but it bodes well for any linux users.

[ethanbass]

Nice. This is very good to know. Devtools is not an actual dependency by the way. You can also use remotes::install_github to install from GitHub (not sure if this is easier to install on Linux or not). Or manually download/clone the GitHub repository and just use install.packages to load it from your hard drive.

Once i feel like i've arrived at some kind of stopping point with the current round of development I will also push updates back to CRAN. They limit you to one update every 2-3 months, so I don't want to push an update there with a bunch of bugs that I'm going to fix a day or two later.

[ethanbass]

I think I'm going to close this for now, since it seems like I've solved the main issue of the package failing to load entirely when the python stuff fails. I opened another issue in #8 about the python environment (sometimes) failing to be configured on Windows, but I'm not sure what I'll really be able to do here other than maybe trying to document the actual requirements better