(fix) fix pyproject.toml

[wizyoung]

Summary

In https://github.com/linkedin/Liger-Kernel/pull/218, I fixed the tool.setuptools.packages.find field and tested it only in editable mode with pip install -e .. However, in production mode with pip install ., only the env_report.py file is copied to the Python site-packages directory. To fix this, adding "liger_kernel.*" to the include list will ensure that setuptools correctly includes all subpackages within liger_kernel.

Testing Done

• Hardware Type:
• [ ] run make test to ensure correctness
• [ ] run make checkstyle to ensure code style
• [ ] run make test-convergence to ensure convergence

[ByronHsu]

even without https://github.com/linkedin/Liger-Kernel/pull/218, i can install correctly. i think it should work even before #218. Otherwise, the CI will fail? Maybe we are on different environment? cc @yundai424

[lancerts]

even without #218, i can install correctly. i think it should work even before #218. Otherwise, the CI will fail? Maybe we are on different environment? cc @yundai424

use pip install liger-kernel

>>> from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'LigerFusedLinearCrossEntropyLoss' from 'liger_kernel.transformers' (/home/lancerts/miniconda3/envs/wsl-dev/lib/python3.11/site-packages/liger_kernel/transformers/__init__.py)

but install from source this works fine. pytest in the source folder also passed. So something is broken in the install.

[ByronHsu]

@lancerts that is because https://github.com/linkedin/Liger-Kernel/pull/210 is merged after v0.2.1. Different issue

[lancerts]

@lancerts that is because #210 is merged after v0.2.1. Different issue

Oh, i thought it was in v021... my bad :P

[wizyoung]

my env:

Operating System: Linux-5.10.0-1.0.0.28-x86_64-with-glibc2.27
Python version: 3.9.19
PyTorch version: 2.3.1+cu121
CUDA version: 12.1
Triton version: 2.3.1
Transformers version: 4.45.0.dev0

Actually 0.2.1 works fine in my env, but the nightly version fails to import. After examining the pyproject.toml file and debugging for a while, I discovered that the tool.setuptools.packages.find blocked file copying. To reproduce it again:

❯ ipython                                                                                                                                            14s
iPython 3.9.19 | packaged by conda-forge | (main, Mar 20 2024, 12:50:21) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import liger_kernel

In [2]: from liger_kernel.transformers.rope import liger_rotary_pos_emb
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[2], line 1
----> 1 from liger_kernel.transformers.rope import liger_rotary_pos_emb

❯ ls /root/miniconda3/lib/python3.9/site-packages/liger_kernel                                                                                     2m 0s
env_report.py  __pycache__

[ByronHsu]

cc @AndreSlavescu ^^ can you take a look

[S1ro1]

Joining here, having weird issues with the editable install as well, can't find a minimaln reproduction tho. Something is off though.

[AndreSlavescu]

ls /root/miniconda3/lib/python3.9/site-packages/liger_kernel 

Looking into it

[AndreSlavescu]

Yeah I think the change should fix the issue with nightly and main release on pypi. Package structure can't be determined in current state, which seems to be an issue.

[AndreSlavescu]

Env Settings:

python 3.9

Tested on following pypi packages: liger-kernel main release liger-kernel-nightly release

Tested on source build: both with change and without change work fine.

This is my small script for repro:

import os
import sys
import importlib

def test_liger_kernel_imports(version):
    print(f"liger_kernel version: {version}")

    try:
        import liger_kernel
        print("imported liger_kernel successfully")
        print(f"liger_kernel.__file__: {liger_kernel.__file__}")
        print(f"liger_kernel.__path__: {liger_kernel.__path__}")
        print(f"sys.modules['liger_kernel']: {sys.modules['liger_kernel']}")

        modules_to_test = [
            "liger_kernel.transformers.rope",
            "liger_kernel.transformers.swiglu",
            "liger_kernel.transformers.layer_norm",
        ]

        for module in modules_to_test:
            try:
                imported_module = importlib.import_module(module)
                print(f"imported {module} successfully")
                print(f"{module}.__file__: {imported_module.__file__}")
            except ImportError as e:
                print(f"Failed to import {module}: {e}")

        print("\nPackage structure:")
        if liger_kernel.__file__ is not None:
            package_path = os.path.dirname(liger_kernel.__file__)
            for root, dirs, files in os.walk(package_path):
                level = root.replace(package_path, '').count(os.sep)
                indent = '    ' * level
                print(f"{indent}{os.path.basename(root)}/")
                sub_indent = ' ' * 4 * (level + 1)
                for file in files:
                    print(f"{sub_indent}{file}")
        else:
            print(f"cant build package structure: liger_kernel.__file__ is {liger_kernel.__file__}")

    except ImportError as e:
        print(f"Failed to import liger_kernel: {e}")

    print("\npython path:")
    for path in sys.path:
        print(path)

if __name__ == "__main__":
    print(f"python version: {sys.version}")
    print(f"working dir: {os.getcwd()}")
    test_liger_kernel_imports("stable")
    print("\n==================================\n")
    test_liger_kernel_imports("nightly")

[lancerts]

Joining here, having weird issues with the editable install as well, can't find a minimaln reproduction tho. Something is off though.

same here.