`open3d` v.s `open3d-cpu`

tdejager commented 2 months ago

Comment:

Hey 👋 !

We've been running into the following issue with the pixi project, that is showcased by this pixi.toml file: https://github.com/pablovela5620/dn-splatter/blob/main/pixi.toml

Background

So as far as I know this is the current situation for open3d w.r.t pypi to conda:

open3d

For pypi:

open3d is provided for all platforms (where the linux ones have gpu support)
open3d-cpu is provided for linux only and has started being a thing from 0.17 onwards.

Conda:

open3d is just one package where:
- open3d-cpu is provided for linux
- open3d is provided for osx

So as far as I can see there is no cuda version of open3d on conda at all.

Also, although open3d uses the open3d-cpu dist info and related directories. The actual installed code is still in the open3d namespace. So if you've installed the -cpu variant one can still do import open3d. For the conda version the same holds.

The problem

With pixi we try to combine pypi and conda dependencies into one environment we do this by.

First solving the conda packages, and python (needed for building sdists)
During the pypi solve we provide the locked conda packages to the solver and if a requirements requires a pypi dependency that we have previously solved we say "hey we got this one already".

Because of this we want an accurate conda -> pypi mapping. This is why we have https://github.com/prefix-dev/parselmouth @nichmor can tell you more about this if there is interest.

In the mapping we map:

osx as open3d -> open3d
linux as open3d -> open3d-cpu

Which for linux is a problem, because all pyp requirements want open3d and we say, conda provides open3d-cpu which makes pixi install both dependencies.

Questions

So part of the problem is that open3d is kind-of making strange use of packages because it uses the open3d namespace but has a open3d-cpu distinfo. So what was is the name of the actual package!? 😄

Should we maybe make a open3d-cpu for linux only? So that it maps to the pypi name?
Is there a reason why the CUDA support is disabled for this package? Were there some problems with this?

Tobias-Fischer commented 2 months ago

Hi @tdejager - I’m a bit confused. We only provide an open3d package on conda-forge, there is no open3d-cpu; where did you see this?

Re cuda: I haven’t tried enabling this. Feel free to send a PR that enables support :)

tdejager commented 2 months ago

Hi @Tobias-Fischer thanks for answering :)

Yeah, I think it has gone a bit unnoticed, but the latest path.json has this part regarding the .dist-info:

    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/INSTALLER",
      "path_type": "hardlink",
      "sha256": "d0edee15f91b406f3f99726e44eb990be6e34fd0345b52b910c568e0eef6a2a8",
      "size_in_bytes": 5
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/LICENSE.txt",
      "path_type": "hardlink",
      "sha256": "0cbb89e9cd94b28a25db7226f70621e43993124605b39f6de713720a5630d274",
      "size_in_bytes": 1109
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/METADATA",
      "path_type": "hardlink",
      "sha256": "13546704ef1d30501cc33c1875589074f0962dba0bf0403753b23ff915315974",
      "size_in_bytes": 3909
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/RECORD",
      "path_type": "hardlink",
      "sha256": "0ade9ad6f7c3f60afe2f5df36eb655f389761936bd6d8a44eea3ed95b693686f",
      "size_in_bytes": 37654
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/REQUESTED",
      "path_type": "hardlink",
      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "size_in_bytes": 0
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/WHEEL",
      "path_type": "hardlink",
      "sha256": "746646485d369a708e34eb264b7df840c1220d9096171417b4034f94985da3b0",
      "size_in_bytes": 114
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/direct_url.json",
      "path_type": "hardlink",
      "sha256": "029b06fbebd9ea313648519c01a87033a970432546b5680eb498a38ac57a9c39",
      "size_in_bytes": 380
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/entry_points.txt",
      "path_type": "hardlink",
      "sha256": "8fb2c5441471c993b4113b3ffa2c79a1fee7f900a8dfa1e9c8ebe2a268202484",
      "size_in_bytes": 141
    },
    {
      "_path": "lib/python3.10/site-packages/open3d_cpu-0.18.0+b128edd.dist-info/top_level.txt",
      "path_type": "hardlink",
      "sha256": "af1612b25697e2bfa874ab708a450a175905af42cfa3d64064794513d58a84c1",
      "size_in_bytes": 7
    }

Again, this should be for the linux version only. You can see the setup.py where they are doing this: https://github.com/isl-org/Open3D/blob/172367ea241e36236af886e3649a09e8401c7503/python/setup.py#L149

Again this begs the question, what is the actual name of the package, because the python files are still located in the open3d namespace.

tdejager commented 2 months ago

W.r.t cuda I'll give it a try later 👍

Tobias-Fischer commented 2 months ago

Thanks @tdejager - I wasn’t aware of this open3d-cpu stuff in the .dist-info. If it causes trouble for your mapping, I’d be happy to accept a PR that changes the setup.py so it’s always just open3d. I guess as you said currently we don’t really notice as the Python files are still in the open3d namespace regardless.

tdejager commented 2 months ago

Yeah, so actually it's more of a problem of what is correct, I think ideally we would:

Have a cuda version on linux thats called open3d and a cpu variant (with a different buildstring I guess) that installs the open3d-cpu .dist-info things.
Keep the rest as-is so just open3d that is actually the cpu version as well but 🤷

Hopefully, a cuda version is then automatically selected if you support the virtual package for it. It otherwise selects the cpu variant. For the conda ecosystem, it might make less sense to have an explicit -cpu actual package? Not sure how you feel about this :)

An aside is that I still don't know how correct it is, the question being what is the name of your package if the .dist-info/METADATA says open3d-cpu and you use the namespace open3d. I would say open3d-cpu but don't know for sure.

tdejager commented 2 months ago

We could copy tensorflow and have a specific -cpu filename + buildstring. Altough, it would then make sense to use the -cpu for osx64 and osx64-arm as well, although the .conda would contain the regular .dist-info.

traversaro commented 4 weeks ago

We experienced something similar in opencv with opencv-python, opencv-python-headless, opencv-contrib-python, opencv-contrib-python-headless (can't add a link as I am on the phone but it should be easy to find it). We ended up adding pip metadata for all these packages, not sure if in this case it make sense to install metadata so that pip list returns both open3d and open3d-cpu .

conda-forge / open3d-feedstock