HolyWu / vs-dpir

DPIR function for VapourSynth
MIT License
32 stars 3 forks source link

Question regarding dependencies and other projects #42

Open Selur opened 4 months ago

Selur commented 4 months ago

Motivated that I got vs-dpir working with your help, I wanted to add your other projects, but can into a few problems. (Sorry, this is a bit longer, but I wanted to be sure you know what I did so you might be able to see what I missed or did wrong.)

What I did was

So far so good.

Next thing I tried was, installing vs-realesrgan, by calling:

I know, that cuart64_12.dll comes in https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_551.78_windows.exe

I hoped using: python pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121 could help, but that ended with:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torch-tensorrt 2.4.0.dev0+8915f846d requires torch<2.4.0,>=2.3.0, but you have torch 2.4.0.dev20240513+cu121 which is incompatible.

So that did not work, next I tried vsrife:

Next I tried installing vsfemasr:

Last but not least, I tried BasicVSR++:

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.



vsfemasr and vsrife: both seem to not know where to find the dlls, but not knowing what they are looking to I don't know what I should install or could install to my path.
vsrealesrgan: no clue how to fix it
basicvsr++: from what I see, the problem is with mmvc is not supporting Python 3.12 so far through precomiled files.

Got any hints/tips/… I could try to get vsfemasr, vsrife, vsrealesrgan and basicvsr++ working?
HolyWu commented 4 months ago

Why did you pip install --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu121 after pip install torch_tensorrt-2.4.0.dev0+8915f846d-cp312-cp312-win_amd64.whl? The README didn't tell you to do that!

Selur commented 4 months ago

iirc I installed that since something was missing. Will try again and report back.

Selur commented 4 months ago

Motivated that I got vs-dpir working with your help, I wanted to add your other projects, but can into a few problems. (Sorry, this is a bit longer, but I wanted to be sure you know what I did so you might be able to see what I missed or did wrong.)

What I did was

So far so good.

Then I tried 'vsrealesrgan':

Then I tried 'vsrife':

Then I tried 'vsfemasr':

Last I tried BasicVSR++:

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.



=> not using  "--pre torch-tensorrt" changed the output.
HolyWu commented 3 months ago

Your tensorrt python package was not properly installed. Normally installing tensorrt 10 from PyPI will also install tensorrt-cu12_bindings and tensorrt-cu12_libs automatically. I have updated the README to specifically install them as well.

Besides, some other projects (vs-femasr, vs-gmfss_fortuna, vs-realesrgan) depend on torch_tensorrt_fx_only package. That package is outdated and uses the same directory name as torch_tensorrt package. Hence, some files could get overwritten by old files if you installed the aforementioned projects after torch_tensorrt package had been installed.

Selur commented 3 months ago

!Thanks!, using:

btw.

HolyWu commented 3 months ago

Huh? It's already on PyPI a few hours ago.

Selur commented 3 months ago

Strange, just triggered it again:

F:\Vapoursynth>python -m pip install -U vsdpir
Requirement already satisfied: vsdpir in f:\vapoursynth\lib\site-packages (4.0.0)
Collecting vsdpir
  Using cached vsdpir-4.1.0-py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: numpy in f:\vapoursynth\lib\site-packages (from vsdpir) (1.26.4)
Requirement already satisfied: requests in f:\vapoursynth\lib\site-packages (from vsdpir) (2.31.0)
INFO: pip is looking at multiple versions of vsdpir to determine which version is compatible with other requirements. This could take a while.
Requirement already satisfied: torch>=2.3.0 in f:\vapoursynth\lib\site-packages (from vsdpir) (2.3.0)
Requirement already satisfied: tqdm in f:\vapoursynth\lib\site-packages (from vsdpir) (4.66.4)
Requirement already satisfied: vapoursynth>=66 in f:\vapoursynth\lib\site-packages (from vsdpir) (68)
Requirement already satisfied: vstools in f:\vapoursynth\lib\site-packages (from vsdpir) (3.1.0)
Requirement already satisfied: filelock in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (3.14.0)
Requirement already satisfied: typing-extensions>=4.8.0 in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (4.11.0)
Requirement already satisfied: sympy in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (1.12)
Requirement already satisfied: networkx in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (3.3)
Requirement already satisfied: jinja2 in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (3.1.4)
Requirement already satisfied: fsspec in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (2024.5.0)
Requirement already satisfied: mkl<=2021.4.0,>=2021.1.1 in f:\vapoursynth\lib\site-packages (from torch>=2.3.0->vsdpir) (2021.4.0)
Requirement already satisfied: charset-normalizer<4,>=2 in f:\vapoursynth\lib\site-packages (from requests->vsdpir) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in f:\vapoursynth\lib\site-packages (from requests->vsdpir) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in f:\vapoursynth\lib\site-packages (from requests->vsdpir) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in f:\vapoursynth\lib\site-packages (from requests->vsdpir) (2024.2.2)
Requirement already satisfied: colorama in f:\vapoursynth\lib\site-packages (from tqdm->vsdpir) (0.4.6)
Requirement already satisfied: stgpytools>=1.0.5 in f:\vapoursynth\lib\site-packages (from vstools->vsdpir) (1.0.5)
Requirement already satisfied: rich>=12.6.0 in f:\vapoursynth\lib\site-packages (from vstools->vsdpir) (13.7.1)
Requirement already satisfied: intel-openmp==2021.* in f:\vapoursynth\lib\site-packages (from mkl<=2021.4.0,>=2021.1.1->torch>=2.3.0->vsdpir) (2021.4.0)
Requirement already satisfied: tbb==2021.* in f:\vapoursynth\lib\site-packages (from mkl<=2021.4.0,>=2021.1.1->torch>=2.3.0->vsdpir) (2021.12.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in f:\vapoursynth\lib\site-packages (from rich>=12.6.0->vstools->vsdpir) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in f:\vapoursynth\lib\site-packages (from rich>=12.6.0->vstools->vsdpir) (2.18.0)
Requirement already satisfied: MarkupSafe>=2.0 in f:\vapoursynth\lib\site-packages (from jinja2->torch>=2.3.0->vsdpir) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in f:\vapoursynth\lib\site-packages (from sympy->torch>=2.3.0->vsdpir) (1.3.0)
Requirement already satisfied: mdurl~=0.1 in f:\vapoursynth\lib\site-packages (from markdown-it-py>=2.2.0->rich>=12.6.0->vstools->vsdpir) (0.1.2)

Even running python -m pip cache purge before python -m pip install -U vsdpir doesn't change it.

Okay, when I call python -m pip install -U vsdpir==4.1.0

Collecting vsdpir==4.1.0
  Using cached vsdpir-4.1.0-py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: numpy in f:\vapoursynth\lib\site-packages (from vsdpir==4.1.0) (1.26.4)
Requirement already satisfied: requests in f:\vapoursynth\lib\site-packages (from vsdpir==4.1.0) (2.31.0)
INFO: pip is looking at multiple versions of vsdpir to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement torch>=2.4.0.dev (from vsdpir) (from versions: 2.2.0, 2.2.1, 2.2.2, 2.3.0)
ERROR: No matching distribution found for torch>=2.4.0.dev

Seems like I missed something somewhere or something got changed, steps I did were:

HolyWu commented 3 months ago

The release note has mentioned PyTorch version bump. You need to install nightly build.

Selur commented 3 months ago

Ah, I still need python -m pip install torch_tensorrt-2.4.0.dev20240518+cu121-cp312-cp312-win_amd64.whl I wrongly assumed that: python -m pip install --pre torch-tensorrt --index-url https://download.pytorch.org/whl/nightly/cu121 would already do the job. => got 4.1.0 now thanks!

I assume that for BasicVSR++ the only thing I can do is wait till openmin suppots Python 3.12, right?

Selur commented 3 months ago

Argh,.. I still didn't get it properly. Tried it anew to be sure I got the right order, but with:

The conflict is caused by: vsfemasr 1.0.0 depends on torch>=1.13.0 torch-tensorrt-fx-only 1.4.0.post1 depends on torch<2.2 and >=2.0.1 vsfemasr 1.0.0 depends on torch>=1.13.0 torch-tensorrt-fx-only 1.3.0 depends on torch<1.14.0 and >=1.13.0

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict
    calling `python -m pip install -U vsfemasr==1.2.0`, I get:

    INFO: pip is looking at multiple versions of torch-tensorrt-fx-only to determine which version is compatible with other requirements. This could take a while. Collecting torch-tensorrt-fx-only>=1.3.0 (from vsfemasr==1.2.0) Using cached torch_tensorrt_fx_only-1.3.0-py3-none-any.whl.metadata (13 kB) ERROR: Cannot install vsfemasr and vsfemasr==1.2.0 because these package versions have conflicting dependencies.

The conflict is caused by: vsfemasr 1.2.0 depends on torch>=1.13.0 torch-tensorrt-fx-only 1.4.0.post1 depends on torch<2.2 and >=2.0.1 vsfemasr 1.2.0 depends on torch>=1.13.0 torch-tensorrt-fx-only 1.3.0 depends on torch<1.14.0 and >=1.13.0

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts



=> What is the right order to install the torch dependencies?
HolyWu commented 3 months ago

Either don't install projects which depend on outdated torch_tensorrt_fx_only at the moment, or install them with --no-deps flag and hope they don't break.

Selur commented 3 months ago

vsdpir and vsrife both install but do not work and complaint about torch_tensorrt.fx.converters.converter_utils missing. atm. I got:

Package                  Version
------------------------ ------------------------
annotated-types          0.7.0
antlr4-python3-runtime   4.9.3
beautifulsoup4           4.12.3
blis                     0.7.11
Bottleneck               1.3.8
catalogue                2.0.10
certifi                  2024.2.2
charset-normalizer       3.3.2
click                    8.1.7
cloudpathlib             0.16.0
colorama                 0.4.6
confection               0.1.4
contourpy                1.2.1
cupy-cuda11x             13.1.0
cycler                   0.12.1
cymem                    2.0.8
einops                   0.8.0
fastai                   1.0.60
fastprogress             1.0.3
fastrlock                0.8.2
filelock                 3.14.0
fonttools                4.52.1
fsspec                   2024.5.0
huggingface-hub          0.23.1
idna                     3.7
imageio                  2.34.1
intel-openmp             2021.4.0
Jinja2                   3.1.4
kiwisolver               1.4.5
kornia                   0.7.2
kornia_rs                0.1.3
langcodes                3.4.0
language_data            1.2.0
lazy_loader              0.4
llvmlite                 0.42.0
marisa-trie              1.1.1
markdown-it-py           3.0.0
MarkupSafe               2.1.5
matplotlib               3.9.0
mdurl                    0.1.2
mkl                      2021.4.0
mpmath                   1.3.0
murmurhash               1.0.10
networkx                 3.3
numba                    0.59.1
numexpr                  2.10.0
numpy                    1.26.4
nvidia-cuda-runtime-cu12 12.5.39
nvidia-ml-py3            7.352.0
omegaconf                2.3.0
opencv-contrib-python    4.9.0.80
opencv-python            4.9.0.80
packaging                24.0
pandas                   2.2.2
Pillow                   10.1.0
pip                      24.0
preshed                  3.0.9
pydantic                 2.7.1
pydantic_core            2.18.2
Pygments                 2.18.0
pyparsing                3.1.2
python-dateutil          2.9.0.post0
pytz                     2024.1
PyYAML                   6.0.1
requests                 2.32.2
rich                     13.7.1
safetensors              0.4.3
scikit-image             0.23.2
scipy                    1.13.1
setuptools               70.0.0
six                      1.16.0
smart-open               6.4.0
soupsieve                2.5
spacy                    3.7.4
spacy-legacy             3.0.12
spacy-loggers            1.0.5
srsly                    2.4.8
stgpytools               1.0.5
sympy                    1.12
tbb                      2021.12.0
tensorrt                 10.0.1
tensorrt-cu12            10.0.1
tensorrt-cu12_bindings   10.0.1
tensorrt-cu12_libs       10.0.1
thinc                    8.2.3
tifffile                 2024.5.22
timm                     1.0.3
torch                    2.4.0.dev20240523+cu121
torch_tensorrt           2.4.0.dev20240518+cu121
torch-tensorrt-fx-only   1.2.0
torchvision              0.19.0.dev20240525+cu121
tqdm                     4.66.4
typer                    0.9.4
typing_extensions        4.12.0
tzdata                   2024.1
urllib3                  2.2.1
VapourSynth              68
vsanimesr                1.0.0
vscodeformer             1.0.1
vsddcolor                1.0.1
vsdeoldify               3.5.3
vsdpir                   4.1.0
vsgan                    1.6.4
vsgmfss_fortuna          1.0.0
vsgrlir                  1.1.1
vshinet                  1.0.0
vsrealesrgan             4.0.0
vsrife                   5.0.0
vsscunet                 1.0.0
vsswinir                 2.0.0
vstools                  3.1.0
vsutil                   0.8.0
wasabi                   1.1.2
weasel                   0.3.4
wheel                    0.43.0
Selur commented 3 months ago

resetting everything up again, seems to work, will do some testing which tools installation breaks the rest.

Selur commented 3 months ago

Okay, not using:

Selur commented 3 months ago

Small update:

so atm. only: