search

Quansight / ragna

RAG orchestration framework ⛵️
https://ragna.chat
BSD 3-Clause "New" or "Revised" License
177 stars 22 forks source link

PoC for Ragna meta package #439

Open pmeier opened 2 months ago

pmeier commented 2 months ago

This is an alternative to #405. With this we have a new folder meta-package that contains a pyproject.toml and a setup.py:

Here is how you do a development install

$ # install ragna-base
$ pip install -e .
$ # either install the meta package
$ pip install -e ./meta-package
$ # or just install the optional requirements
$ pip install -r requirements-optional.txt

To release you can do

$ # build ragna-base
$ python -m build
$ # build ragna
$ python -m build --sdist --wheel --outdir ./dist ./meta-package

I don't know why I explicitly need to set --sdist --wheel for the meta-package build. If I don't, the build errors. This seems to be a bug in the build package. Will look into it.

arjxn-py commented 2 months ago

I also noticed that currently, it is essential to build ragna-base before ragna & that's just because so that pip can locate ragna-base dependency.

Would we want to do the same on the CI workflow as well? (However more experimentation is yet to be done)

arjxn-py commented 2 months ago

I did some workaround on a branch on my fork with this one as the base. And also opened a PR on the fork to run workflow.

You can also download & test build artifacts from here:

Currently, one cannot install ragna-dist directly without installing ragna-base-dist

arjxn-py commented 2 months ago

I think the failure here can be used as a breaking point which should prevent us from going ahead to publish the meta-package until & unless pip has the access to the dependent ragna-base version.

pmeier commented 2 months ago

However I noticed one thing that the version that is being checked in setup.py looks locally something like 0.3.0.dev31+g7743725.d20240704 & say we've published ragna-base==0.3.0 on pip. Would it still be going to give error saying ERROR: Could not find a version that satisfies the requirement ragna==0.3.0.dev31+g7743725.d20240704?

This has nothing to do with ragna-base being published. If you want to install ragna locally, and there is really no need for it, you'll first have to install ragna-base. For this to work, both packages have to be build on the same commit.

I also noticed that currently, it is essential to build ragna-base before ragna & that's just because so that pip can locate ragna-base dependency.

Would we want to do the same on the CI workflow as well? (However more experimentation is yet to be done)

We'll never install ragna in CI other than in one dedicated workflow that makes sure the wheel we build is correct.

I think the failure here can be used as a breaking point

This is just a failing docker build, because I haven't updated the Dockerfile yet. Let me do that today.

which should prevent us from going ahead to publish the meta-package until & unless pip has the access to the dependent ragna-base version.

Yes, of course. We'll only ever publish them together. The first release for both of them will be v0.3.0 assuming we can get this done by then.

arjxn-py commented 2 months ago

This has nothing to do with ragna-base being published. If you want to install ragna locally, and there is really no need for it, you'll first have to install ragna-base. For this to work, both packages have to be build on the same commit.

Sorry to ask again, I just want to confirm that when releasing & publishing wheels, For example v0.3.0, ragna would have ragna-base==0.3.0 as a dependency instead of something like ragna==0.3.0.dev31+g7743725.d20240704 as this extra version identifier string makes it hard for pip or conda to identify the package.

We'll never install ragna in CI other than in one dedicated workflow that makes sure the wheel we build is correct.

Right, that makes sense. But just to make a note again that installation of wheels will only pass either if ragna-base is already installed in the environment or that specific version of ragna-base could be found on Pypi. It's not an issue but a checkpoint type of a thing.

This is just a failing docker build, because I haven't updated the Dockerfile yet. Let me do that today.

😬 I reffered to the another linked failure here on my fork, this is the one I refer to as checkpoint.

pmeier commented 2 months ago

Sorry to ask again, I just want to confirm that when releasing & publishing wheels, For example v0.3.0, ragna would have ragna-base==0.3.0 as a dependency instead of something like ragna==0.3.0.dev31+g7743725.d20240704 as this extra version identifier string makes it hard for pip or conda to identify the package.

I see where the confusion is coming from now. We are using setuptools_scm as versioning tool.

https://github.com/Quansight/ragna/blob/9776ec5b06b573ee0b672e07e23c13cd7e0e5fb3/pyproject.toml#L1-L6

https://github.com/Quansight/ragna/blob/9776ec5b06b573ee0b672e07e23c13cd7e0e5fb3/pyproject.toml#L67-L70

It looks at the current state of git and produces a version number from it. So your string ragna==0.3.0.dev31+g7743725.d20240704 translates to

You can check the version it produces with python -m setuptools_scm. If you commit everything in your local copy, the .d... part will disappear. And finally, if you add a git tag, e.g. git tag v1.2.3, you'll see that the output is 1.2.3 (make sure to delete the tag again with git tag -d v1.2.3 to avoid future clashes).

Long story short: since we'll add a git tag before each release, all the extra version information that we have during development will be gone.

But just to make a note again that installation of wheels will only pass either if ragna-base is already installed in the environment or that specific version of ragna-base could be found on Pypi.

I'm aware, but again this is not an issue. For development, we'll never install ragna and for published packages, the corresponding ragna-base will be available.

It's not an issue but a checkpoint type of a thing.

I don't understand what you mean by "checkpoint".

arjxn-py commented 2 months ago

Long story short: since we'll add a git tag before each release, all the extra version information that we have during development will be gone.

That makes sense, thanks a lot on confirming this.

I don't understand what you mean by "checkpoint".

I mean just a checkpoint to see corresponding ragna-base availability. Please see this error.

This looks promising, Please do let me know if I can assist you with anything related 🚀

pmeier commented 2 months ago

I don't know why I explicitly need to set --sdist --wheel for the meta-package build. If I don't, the build errors. This seems to be a bug in the build package. Will look into it.

The failure means that the wheel cannot be build by the sdist, i.e. the sdist is broken. This needs to be fixed before moving forward.

pmeier commented 2 months ago

@arjxn-py If you are still eager to help with this, could you check why we cannot build a wheel from the sdist on this branch, i.e. find out what is broken with the sdist? From the error we get when running python -m build --output-dir ./dist ./meta-package is seems the pyproject.toml file is missing. We might just need to add a MANIFEST.in file and link the pyproject.toml there.