search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
698 stars 180 forks source link

Make `bigcode_eval` pip installable #231

Closed shehrozek-cerebras closed 1 week ago

shehrozek-cerebras commented 2 months ago

Is it possible to make this package pip installable? Currently, when I pin some commit in my requirements.txt file,

bigcode_eval @ git+https://github.com/bigcode-project/bigcode-evaluation-harness.git@a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e

I see the following error:

ERROR: Could not find a version that satisfies the requirement bigcode-eval (unavailable) (from versions: none)
ERROR: No matching distribution found for bigcode-eval (unavailable)

Is this because the setup.py doesn't have a package name specified? https://github.com/bigcode-project/bigcode-evaluation-harness/blob/a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e/setup.py#L30-L37

I couldn't find the published package on PyPi either.

Perhaps the error is due to mismatching Python version. I'm using python3.8 in my environment, but it's unclear what the required Python version is for bigcode. Is it possible to make the requirement explicit and streamline the package installation process? Thanks.

shehrozek-cerebras commented 2 months ago

From the pip install logs, I also see this:

Collecting bigcode-eval@ git+https://github.com/bigcode-project/bigcode-evaluation-harness.git (from -r /tmp/tmpj0nno88m (line 126))
  Cloning https://github.com/bigcode-project/bigcode-evaluation-harness.git to /tmp/pip-install-2cfwowhe/bigcode-eval_e7397c9bd97643f09a0059e0caf044ae
  Running command git clone --filter=blob:none --quiet https://github.com/bigcode-project/bigcode-evaluation-harness.git /tmp/pip-install-2cfwowhe/bigcode-eval_e7397c9bd97643f09a0059e0caf044ae
  Resolved https://github.com/bigcode-project/bigcode-evaluation-harness.git to commit a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
  WARNING: Generating metadata for package bigcode-eval produced metadata for project name unknown. Fix your #egg=bigcode-eval fragments.

It seems that the call to setup must have name specified (see similar issue), i.e.

 setup( 
     name="bigcode_eval", 
     description="A framework for the evaluation of autoregressive code generation language models.", 
     long_description=readme, 
     license="Apache 2.0", 
     packages=find_packages() , 
     install_requires=requirements, 
     extras_require={"ds1000": ds1000_requirements}, 
 )