Difficulty Packaging Scripts Folder with scikit-build-core

ahnaf-tahmid-chowdhury commented 8 months ago

Description

I am currently in the process of migrating from scikit-build to scikit-build-core for packaging my project, specifically the PyNE repository. Previously, with scikit-build, I utilized a method to package the scripts folder containing all the executables. However, I am encountering difficulty replicating this functionality with scikit-build-core.

Steps to Reproduce

Attempt to package the scripts folder containing executables using scikit-build-core.
Encounter issues due to differences in methods or functionality between scikit-build and scikit-build-core.
Unable to find a suitable alternative or workaround within the documentation or resources available.

Expected Behavior

I expect to be able to package the scripts folder containing executables seamlessly using scikit-build-core, similar to how it was done with scikit-build.

Current Approach

Here's the method I was using previously with scikit-build:

# Collect scripts
scripts = [os.path.join("scripts", f) for f in os.listdir("scripts")]
scripts = [
    s
    for s in scripts
    if (os.name == "nt" and s.endswith(".bat"))
    or (os.name != "nt" and not s.endswith(".bat"))
]

# Collect extension
extension = ["*.dll", "*.so", "*.dylib", "*.pyd", "*.pyo"]

# Setup configuration
setup(
    packages=[
        "pyne",
        "pyne.dbgen",
        "pyne.apigen",
        "pyne.xs",
        "pyne.transmute",
        "pyne.gui",
        "pyne.cli",
        "pyne.fortranformat",
    ],
    package_data={
        "lib": extension,
        "pyne": [
            "*.pxd",
            "*.json",
            "*.inp",
        ]
        + extension,
        "pyne.xs": ["*.pxd"] + extension,
        "pyne.gui": ["*.pyw"],
        "pyne.dbgen": ["*.html", "*.csv", "abundances.txt", "mass.mas16", "*.dat"],
    },
    scripts=scripts,
    cmake_args=cmake_args,
    cmake_install_dir=".",
)

Additional Information

Using scikit-build-core version 0.8.2.
Repository: PyNE with scikit-build-core
Repository: PyNE with scikit-build
Any insights, guidance, or examples on how to achieve similar packaging of the scripts folder with scikit-build-core would be greatly appreciated.

henryiii commented 8 months ago

You can write to SKBUILD_SCRIPTS_DIR from CMake. https://scikit-build-core.readthedocs.io/en/latest/cmakelists.html#install-directories

LecrisUT commented 8 months ago

I am wondering if OP means static (shell-like) scripts. But I guess in that case as well, you can add a install() to copy those withing CMake.

You can write to SKBUILD_SCRIPTS_DIR

Worth aliasing SKBUILD_BIN_DIR to it as well, to mimic GnuInstalldir?

ahnaf-tahmid-chowdhury commented 8 months ago

Here is my new pyproject.toml that I am currently working on

[build-system]
requires = [
    "scikit-build-core",
    "cython<3",
    "numpy",
]
build-backend = "scikit_build_core.build"

# Project Metadata
[project]
name = "pyne-toolkit"
dynamic = ["version"]
authors = [{ name = "PyNE Team", email = "pyne-dev@googlegroups.com" }]
license = { file = "license.txt" }
description = "PyNE: The Nuclear Engineering Toolkit"
readme = "readme.rst"
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Intended Audience :: End Users/Desktop",
    "Intended Audience :: Science/Research",
    "License :: OSI Approved :: BSD License",
    "Natural Language :: English",
    "Operating System :: OS Independent",
    "Topic :: Scientific/Engineering",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
]
requires-python = ">=3.8"
dependencies = [
    "progress",
    "tables",
    "numpy",
    "scipy",
    "matplotlib",
    "jinja2",
    "future",
]

# Optional Dependencies
[project.optional-dependencies]
test = ["pytest"]
docs = ["sphinx"]

# Project URLs
[project.urls]
"Homepage" = "https://pyne.io"
"Bug Tracker" = "https://github.com/pyne/pyne/issues"
"Source Code" = "https://github.com/pyne/pyne"

# Scikit-Build Configuration
[tool.scikit-build]
cmake.verbose = true
logging.level = "INFO"
wheel.packages = ["pyne"]
wheel.exclude = ["CMakeLists.txt", "*.pyx"]

# Get version from CMakeLists
[tool.scikit-build.metadata.version]
provider = "scikit_build_core.metadata.setuptools_scm"

[tool.setuptools_scm]

# Add the version to the package
[[tool.scikit-build.generate]]
path = "pyne/_version.py"
template = '''
__version__ = "${version}"
'''

# Add the version to the header
[[tool.scikit-build.generate]]
path = "src/pyne_version.h"
location = "source"
template = '''
#ifndef PYNE_VERSION_HEADER
#define PYNE_VERSION_HEADER

#define PYNE_VERSION "${version}"

#endif // PYNE_VERSION_HEADER
'''

It appears that I'm unable to configure the scripts from this toml file and must handle this from the CMake side.

Lately, I've observed that whenever I install a wheel, it places everything into the site-packages directory, including the include, lib, and share directories. Is this behavior a result of using scikit-build-core? I prefer them to be installed into the environment directory as it's more neutral.

LecrisUT commented 8 months ago

it places everything into the site-packages directory, including the include, lib, and share directories. Is this behavior a result of using scikit-build-core?

Afaik that's the pythonic way of doing it. It allows the python scripts to find the content more easily, e.g. with `importlib.

One way I go around this is to have the cmake build be buildable both as stand-alone and as part of a python sdist archive. That way when installed by the user/system it installs to whatever CMAKE_INSTALL_PREFIX you specify, e.g. ~/.local, and when installed via pip install it goes through skbuild logic and installs to site-package paths.

henryiii commented 8 months ago

You can install to SKBUILD_DATA_DIR, that's the "base" directory. SKBUILD_PLATLIB_DIR is the site-packages dir, and is the default install location (if you install to .). There's also SKBUILD_HEADERS_DIR if we are being complete here. :)

Do keep in mind, having a wheel with stuff that doesn't go into site-packages, especially general data dir stuff, is somewhat surprising - someone pip installing into a global Python environment will install to their system folders, which might not be expected. It also might be a little odd on Windows, where you might expect a different structure. (This is the Pythonic comment from @LecrisUT above.)

PS: there is an experimental way to redirect the default location for installs to one of these other folders from TOML.

henryiii commented 8 months ago

It appears that I'm unable to configure the scripts from this toml file and must handle this from the CMake side.

Yes, currently the only auto-copy is for packages. This is helpful for editable installs, where knowing about the packages from Python helps in avoiding making copies when editable installing. But the other directories don't support redirection, so there's no benefit to copying them from Python vs. CMake. Often, you might be writing out binaries to SKBUILD_SCRIPT_DIR, but you can glob a directory and install script files from CMake as well.

We mostly use TOML to configure things that can't be done from CMake.

ahnaf-tahmid-chowdhury commented 8 months ago

Thank you very much for all the support! It seems like SKBUILD_SCRIPT_DIR did the trick. It sets the path to ${CMAKE_INSTALL_PREFIX}/bin on Unix and ${CMAKE_INSTALL_PREFIX}/Scripts for Windows.

I agree with the Pythonic way, as @LecrisUT mentioned. However, I am currently facing some difficulties. Previously, I was using CMake environment variables to configure different directory paths, such as CMAKE_INSTALL_LIBDIR, CMAKE_INSTALL_FULL_LIBDIR, CMAKE_INSTALL_INCLUDEDIR, etc., as they automatically configure these path names for different operating systems. This meant that I didn't need to bother with them. But now it seems I may not be able to use them, or I don't know how to do this with scikit-build-core.

Yes, I know I can use SKBUILD_DATA_DIR/lib for libraries, but it works on Unix; Windows uses a different logic. Are there any environment variables available for scikit-build that are similar to LIBDIR, SHAREDSTATEDIR, DOCDIR, etc.?

My plan is to store the include folder in site-packages/pyne/core/include, the lib folder in pyne/core/lib, and so on.

henryiii commented 8 months ago

If you want it relative to site-packages, then CMAKE_INSTALL_*DIR should just work if you set wheel.install-dir to "pyne", since CMAKE_INSTALL_PREFIX is set to site-packages/wheel.install-dir directory already. Is that what you meant?

ahnaf-tahmid-chowdhury commented 8 months ago

Isn't wheel.install-dir experimental?

henryiii commented 8 months ago

No, only setting it to an absolute value (like /data or /headers) is.

ahnaf-tahmid-chowdhury commented 8 months ago

It's worth mentioning that scikit-build-core is incredibly powerful and packed with numerous features. Thank you, everyone, for all the support. I am closing this issue now since I have figured out how to configure different projects with this package. However, I have a request to make. The documentation for this should be made more clear by adding detailed instructions, as it was difficult for me (a beginner) to grasp.

LecrisUT commented 8 months ago

Well documentation is always difficult to manage. I liked how packit added an example section where it's basically a series of tldr configurations. @ahnaf-tahmid-chowdhury do you have some ideas on how to make these more navigable, and a list of actions that you think should be included?

My list are:

minimal configuration
adding CMake options (to pyproject.toml)
installing binaries/scripts
installing additional files
adding CMake options to pip install from git/PyPI

ahnaf-tahmid-chowdhury commented 8 months ago

I think for now, the tests already have different examples available. So, if I plan to proceed, I would first add some more instructions to the install-directories for clearer understanding. These instructions would explain how variables work and may include a section on how to configure this with GnuInstalldir. I'll also try to demonstrate an example code.

For example, here is how I have manually configured a value for GnuInstalldir:

  # By default, scikit-build installs everything to ${SKBUILD_PLATLIB_DIR} which is site-packages/ or dist-packages/ dir.
  # That means, CMAKE_INSTALL_BINDIR will be available in ${SKBUILD_PLATLIB_DIR}/${CMAKE_INSTALL_BINDIR}
  # So, to set the bin directory to the root environment (${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR} or ${SKBUILD_SCRIPTS_DIR})
  set (CMAKE_INSTALL_BINDIR ${SKBUILD_SCRIPTS_DIR})

LecrisUT commented 8 months ago

Regarding setting CMAKE_INSTALL_*. Such variables should not be set by the project. Instead define install(DESTINATION). The former is meant for the end-user to be able to override.

Maybe one issue with the documentation is that we are assuming the developer has a rather high familiarity with CMake and instead we should be more explicit with the examples and provide more links to further reference

ahnaf-tahmid-chowdhury commented 8 months ago

Ah, I think I have forgotten to add some more comments.

# Now install your executable package as you install generally
add_executable(your_executable)
---
install(
    TARGETS your_executable
    RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}   # prefix/bin
    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}   # site-packages/${install-dir}/lib
    ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
)

ahnaf-tahmid-chowdhury commented 8 months ago

I can be wrong. But it seems this is the best way for the developers who are just migrating to scikit-build-core from a pure CMake project.

scikit-build / scikit-build-core