Repeatedly breaking fbapipublicfiles (central list)

[aaroncsolomon]

@bottler Issue: fbapipublicfiles download links for builds are not working

Status: I previously reported this issue when pt3d bumped from 0.7.0 to 0.7.1 at https://github.com/facebookresearch/pytorch3d/issues/1366 - it's quite annoying that the previous versions are deprecated during release. As I and others noted on the previous issue, these pre-builds are quite useful because not everyone uses conda.

Request: don't delete/remove access to existing wheels on fbapipublicfiles

[bottler]

We are right in the middle of releasing 0.7.2. I have had to pull the wheels down.

[aaroncsolomon]

Yeah, I think I don't understand why the wheels need to be pulled down during new releases - the wheels are named with the semantic version number - why can't pytorch3d leave the wheels under the install link (since the wheels contain the semantic version in their name) and maintain a latest wheel to help automate installation?

@BotScutters

[aaroncsolomon]

@bottler the current approach means that our builds + tests break whenever you do a release because the links break - can you just leave the link with a deprecation warning?

[thathert]

We are right in the middle of releasing 0.7.2. I have had to pull the wheels down.

@bottler how do y'all recommend external folks use this library? Since the linux wheels aren't pushed to pypi, we have been depending on the ones placed in fbapipublicfiles. If the standard deployment practice makes the previous version unavailable then this will break our build every time there is a release.

[bottler]

The wheels on fbaipublicfiles are there for a particular purpose - supporting the tutorials. We don't have a proper archive and distribution system for wheels. With the current system, it doesn't make sense to leave old ones up because when we stop supporting a particular dependency combination a pip install would leave the user with a non-latest version. These wheels are not reliable. If you have a downstream CI or automated process can it not use conda? Or save the wheel you are using?

Release 0.7.2 has just happened. The wheels available are cuda 11.3 with PyTorch 1.11.0 and 1.12.0, and cuda 11.6 with PyTorch 1.13.0. These are all with Python 3.8, 3.9 and 3.10.

I plan to add cuda 11.6 with pytorch 1.13.1 soon. I intended to have cuda 11.3 with pytorch 1.12.1 as well (which is a combination we previously had) but I am having trouble building that.

[thathert]

If you have a downstream CI or automated process can it not use conda?

Unfortunately not since conda and pip/virtualenv don't seem to play nicely together. This is the first library we've come across in ~2 years that has a conda-only approach to distribution.

Or save the wheel you are using?

Yep, I uploaded the wheels to our internal pypi mirror and updated our requirements to switch to this after the build broke so that we'll be able to opt-in to the next release.

We don't have a proper archive and distribution system for wheels. With the current system, it doesn't make sense to leave old ones up because when we stop supporting a particular dependency combination a pip install would leave the user with a non-latest version. These wheels are not reliable.

Understood. If you copied these Linux wheels to pypi like y'all are already doing for the Mac wheels though, you would have a proper archive and distribution system that people could use to pip install particular versions. I think it would make your library much more accessible to other non-conda folks like us.

[bottler]

Understood. If you copied these Linux wheels to pypi like y'all are already doing for the Mac wheels though, you would have a proper archive and distribution system that people could use to pip install particular versions. I think it would make your library much more accessible to other non-conda folks like us.

Copy to pypi how? pypi only allows each version to have only one package per architecture/python_version, right? So how to account for the builds for different cuda and pytorch versions? That's exactly the kind of problem conda is designed to solve. For mac we have no cuda in the equation and we pick only one pytorch version. Doing the same for linux won't please many people.

I plan to add cuda 11.6 with pytorch 1.13.1 soon. I intended to have cuda 11.3 with pytorch 1.12.1 as well (which is a combination we previously had) but I am having trouble building that.

These got done. The combinations available now are

py310_cu113_pyt1110
py310_cu113_pyt1120
py310_cu113_pyt1121
py310_cu116_pyt1130
py310_cu116_pyt1131
py38_cu113_pyt1110
py38_cu113_pyt1120
py38_cu113_pyt1121
py38_cu116_pyt1130
py38_cu116_pyt1131
py39_cu113_pyt1110
py39_cu113_pyt1120
py39_cu113_pyt1121
py39_cu116_pyt1130
py39_cu116_pyt1131

[thathert]

Copy to pypi how? pypi only allows each version to have only one package per architecture/python_version, right? So how to account for the builds for different cuda and pytorch versions?

Ahh, I see your point @bottler. I remember pip installing pytorch but forgot that they have you use their mirror at download.pytorch.org which allows them to have 1-index-per-cuda-version to get around this limitation of pypi. Any possibility y'all can piggy-back on this service like some of the other pytorch libraries or stand up something similar?

Doing the same for linux won't please many people.

Sure it will, just upload the version that matches our cuda and pytorch versions and you'll already have one group of happy customers 😉. Kidding aside, not a huge deal for us -- we luckily only need two combinations (one for 3.8 and one for 3.9) so copying wheels to our internal mirror will be an okay compromise moving forward. Appreciate the discussion.

[bottler]

I'm releasing 0.7.3, which has wheel builds for cu117 and cu118. I haven't removed previous builds. So we now have

packaging/wheels/py310_cu113_pyt1110/pytorch3d-0.7.2-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu113_pyt1120/pytorch3d-0.7.2-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu113_pyt1121/pytorch3d-0.7.2-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu116_pyt1130/pytorch3d-0.7.2-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu116_pyt1131/pytorch3d-0.7.2-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu117_pyt1131/pytorch3d-0.7.2-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu117_pyt1131/pytorch3d-0.7.3-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu117_pyt200/pytorch3d-0.7.3-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu118_pyt200/pytorch3d-0.7.3-cp310-cp310-linux_x86_64.whl
packaging/wheels/py38_cu113_pyt1110/pytorch3d-0.7.2-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu113_pyt1120/pytorch3d-0.7.2-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu113_pyt1121/pytorch3d-0.7.2-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu116_pyt1130/pytorch3d-0.7.2-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu116_pyt1131/pytorch3d-0.7.2-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu117_pyt1131/pytorch3d-0.7.3-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu117_pyt200/pytorch3d-0.7.3-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu118_pyt200/pytorch3d-0.7.3-cp38-cp38-linux_x86_64.whl
packaging/wheels/py39_cu113_pyt1110/pytorch3d-0.7.2-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu113_pyt1120/pytorch3d-0.7.2-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu113_pyt1121/pytorch3d-0.7.2-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu116_pyt1130/pytorch3d-0.7.2-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu116_pyt1131/pytorch3d-0.7.2-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu117_pyt1131/pytorch3d-0.7.3-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu117_pyt200/pytorch3d-0.7.3-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu118_pyt200/pytorch3d-0.7.3-cp39-cp39-linux_x86_64.whl

[bottler]

Builds have been added for versions 0.7.4 and 0.7.5 (just now). Currently

packaging/wheels/py310_cu117_pyt201/pytorch3d-0.7.4-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu118_pyt201/pytorch3d-0.7.4-cp310-cp310-linux_x86_64.whl
packaging/wheels/py38_cu117_pyt201/pytorch3d-0.7.4-cp38-cp38-linux_x86_64.whl
packaging/wheels/py38_cu118_pyt201/pytorch3d-0.7.4-cp38-cp38-linux_x86_64.whl
packaging/wheels/py39_cu117_pyt201/pytorch3d-0.7.4-cp39-cp39-linux_x86_64.whl
packaging/wheels/py39_cu118_pyt201/pytorch3d-0.7.4-cp39-cp39-linux_x86_64.whl
packaging/wheels/py310_cu118_pyt210/pytorch3d-0.7.5-cp310-cp310-linux_x86_64.whl

[bottler]

Just added

packaging/wheels/py310_cu118_pyt211/pytorch3d-0.7.5-cp310-cp310-linux_x86_64.whl

[bottler]

Just added

packaging/wheels/py310_cu121_pyt210/pytorch3d-0.7.5-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu121_pyt211/pytorch3d-0.7.5-cp310-cp310-linux_x86_64.whl
packaging/wheels/py310_cu121_pyt212/pytorch3d-0.7.5-cp310-cp310-linux_x86_64.whl

[bottler]

py310_cu121_pyt221 now added

[bottler]

Just added py310_cu121_pyt230/pytorch3d-0.7.6-cp310-cp310-linux_x86_64.whl

[bottler]

Just added py310_cu121_pyt231/pytorch3d-0.7.6-cp310-cp310-linux_x86_64.whl

[bottler]

py310_cu121_pyt240/pytorch3d-0.7.8-cp310-cp310-linux_x86_64.whl py310_cu121_pyt241/pytorch3d-0.7.8-cp310-cp310-linux_x86_64.whl