Create post-link script to install Julia deps

MilesCranmer commented 2 years ago

Attempts to add a post-link script to install PySR's julia dependencies.

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Addresses #38.

cc @ngam @mkitti

conda-forge-linter commented 2 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

ngam commented 2 years ago

Could you add this line to conda-forge.yml?

azure:
  store_build_artifacts: true

This would allow us to see what actually gets packaged (by downloading the zipped files from the CI).

ngam commented 2 years ago

Anyway, it is working as intended I believe, the key portion of the CI starts on line 845

Maybe a statement should be printed and a way to cancel...

Executing transaction: ...working...   Installing known registries into `~/feedstock_root/build_artifacts/pysr_1660455210381/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/share/julia`
  No Changes to `~/feedstock_root/build_artifacts/pysr_1660455210381/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/share/julia/environments/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/Project.toml`
  No Changes to `~/feedstock_root/build_artifacts/pysr_1660455210381/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/share/julia/environments/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/Manifest.toml`

ngam commented 2 years ago

The install/build script(s) for pysr deleted the following files (from dependencies) from the prefix:
['lib/julia/libmbedtls.so', 'lib/julia/libnghttp2.so', 'lib/julia/libmbedx509.so', 'lib/julia/libcurl.so', 'lib/julia/libopenblas64_.so', 'lib/julia/libmbedcrypto.so']
This will cause the post-link checks to mis-report. Please try not to delete and files (DSOs in particular) from the prefix
WARNING:conda_build.build:The install/build script(s) for pysr deleted the following files (from dependencies) from the prefix:
['lib/julia/libmbedtls.so', 'lib/julia/libnghttp2.so', 'lib/julia/libmbedx509.so', 'lib/julia/libcurl.so', 'lib/julia/libopenblas64_.so', 'lib/julia/libmbedcrypto.so']
This will cause the post-link checks to mis-report. Please try not to delete and files (DSOs in particular) from the prefix

I wonder what exactly this means...

ngam commented 2 years ago

Btw, we need to edit your reqs --- something like this is enough I believe or along the lines of what isuruf suggested: https://github.com/conda-forge/staged-recipes/pull/17647/files

requirements:
  host:
    - pip
    - python >=3.7
  run:
    - pyjulia >=0.5.7
    - julia <=1.7.1
    - numpy
    - pandas
    - python >=3.7
    - scikit-learn >=1.0.0
    - sympy

conda-forge-linter commented 2 years ago

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

The top level meta key azure is unexpected

conda-forge-linter commented 2 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

MilesCranmer commented 2 years ago

Thanks, all suggestions implemented. Let's see if it works

MilesCranmer commented 2 years ago

Since the deps are changing, do I need to merge a new version of PySR (i.e., creating a v0.10.1), or is updating the build number enough?

ngam commented 2 years ago

build number is enough

ngam commented 2 years ago

My one last concern is that we need to get someone from @conda-forge/core to approve this to avoid misunderstanding and trouble. Maybe @isuruf? I would give them like a week to respond.

If no response, my opinion is: You're the maintainer here, and this is your package after all, so you should feel empowered to make whatever changes that you think are appropriate and beneficial.

MilesCranmer commented 2 years ago

Sounds good to me. If there's no response this week I will plan to merge this next weekend.

Also: I relaxed the Julia version requirement to 1.7.*.

ngam commented 2 years ago

Hm... @MilesCranmer

[ Info: Trying to import PyCall...
┌ Error: `import PyCall` failed
│   exception =
│    ArgumentError: Package PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] is required but does not seem to be installed:
│     - Run `Pkg.instantiate()` to install all recorded dependencies.
│    
│    Stacktrace:
│     [1] _require(pkg::Base.PkgId)
│       @ Base ./loading.jl:1089
│     [2] require(uuidkey::Base.PkgId)
│       @ Base ./loading.jl:1013
│     [3] top-level scope
│       @ ~/feedstock_root/build_artifacts/pysr_1660496397213/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib/python3.10/site-packages/julia/install.jl:36
│     [4] include(mod::Module, _path::String)
│       @ Base ./Base.jl:418
│     [5] exec_options(opts::Base.JLOptions)
│       @ Base ./client.jl:292
│     [6] _start()
│       @ Base ./client.jl:495

What went wrong between 1.7.1 and 1.7.2+?

MilesCranmer commented 2 years ago

@ngam that issue by itself is fine - that's how the normal PyJulia install process works - it installs PyCall.jl if it can't import it.

However, the other issues look to be new: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=551157&view=logs&j=656edd35-690f-5c53-9ba3-09c10d0bea97&t=e5c8ab1d-8ff9-5cae-b332-e15ae582ed2d&l=683 from julia >1.7.1

ngam commented 2 years ago

@conda-forge-admin, please rerender

ngam commented 2 years ago

@ngam that issue by itself is fine - that's how the normal PyJulia install process works - it installs PyCall.jl if it can't import it.

However, the other issues look to be new: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=551157&view=logs&j=656edd35-690f-5c53-9ba3-09c10d0bea97&t=e5c8ab1d-8ff9-5cae-b332-e15ae582ed2d&l=683 from julia >1.7.1

I think the issue is with pycall still:

  [8e850b90] + libblastrampoline_jll
  [8e850ede] + nghttp2_jll
    Building Conda ─→ `~/feedstock_root/build_artifacts/pysr_1660496397213/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/share/julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/6e47d11ea2776bc5627421d59cdcc1296c058071/build.log`
    Building PyCall → `~/feedstock_root/build_artifacts/pysr_1660496397213/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/share/julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/1fc929f47d7c151c839c5fc1375929766fb8edcc/build.log`
ERROR: LoadError: Error building `PyCall`: 

DirectoryNotACondaEnvironmentError: The target directory exists, but it is not a conda environment.
Use 'conda create' to convert the directory to a conda environment.
  target directory: /home/conda/feedstock_root/build_artifacts/pysr_1660496397213/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p

[ Info: Running `conda install -y numpy` in root environment

ngam commented 2 years ago

I think the issue here is that your package is trying to make another conda env for itself. Currently, what we are doing is stacking envs. You have a conda env, right? Then a julia env gets built within it inside $CONDA_PREFIX/share/julia/ so there is no longer a need to create a new conda env and instead one can work within the existing conda env in the hierarchical structure.

I think this makes sense as the change from julia 1.7.1 to 1.7.3 exposed this issue. In your helper function, it may help to adjust it so that it can make use of this new feature we introduced --- i.e., all you need to do is ask julia to install your deps directly. What does PyCall do exactly in this case btw?

MilesCranmer commented 2 years ago

pysr.install() is doing the following:

Install PyCall.jl using PyJulia's install() command. Code
Pkg.activate("pysr-0.10.0", shared=true) - (This should create a new env). Code
Pkg.add("SymbolicRegression"). Code
Pkg.instantiate(). Code

ngam commented 2 years ago

pysr.install() is doing the following:

Install PyCall.jl using PyJulia's install() command. Code

Pkg.activate("pysr-0.10.0", shared=true) - (This should create a new env). Code

Pkg.add("SymbolicRegression"). Code

Pkg.instantiate(). Code

Yes, I am not sure what's causing the error. I got it to work fine with a separate step after the fact (not post-link, i.e. like what you had before in the tests):

     Cloning git-repo `https://github.com/JuliaParallel/ClusterManagers.jl`
   Resolving package versions...
    Updating `~/.Mambaforge-Linux-x86_64/envs/py310/share/julia/environments/pysr-0.10.0/Project.toml`
  [34f1f09b] + ClusterManagers v0.4.2 `https://github.com/JuliaParallel/ClusterManagers.jl#14e7302`
    Updating `~/.Mambaforge-Linux-x86_64/envs/py310/share/julia/environments/pysr-0.10.0/Manifest.toml`
  [34f1f09b] + ClusterManagers v0.4.2 `https://github.com/JuliaParallel/ClusterManagers.jl#14e7302`
Precompiling project...
  82 dependencies successfully precompiled in 205 seconds (8 already precompiled)
/home/ngam/Repos/PySR/pysr/julia_helpers.py:39: UserWarning: It is recommended to restart Python after installing PySR's dependencies, so that the Julia environment is properly initialized.
  warnings.warn(

ngam commented 2 years ago

@mkitti could you have a look at what went wrong here? I am not entirely sure...

conda-forge / pysr-feedstock

Create post-link script to install Julia deps #41