search

tensorchord / envd

🏕️ Reproducible development environment
https://envd.tensorchord.ai/
Apache License 2.0
2.04k stars 160 forks source link

feat: support version fuzzy search in envd script #1195

Open popfido opened 2 years ago

popfido commented 2 years ago

Description

When I'm trying to use envd to maintain my dev environment, sometimes it takes me a little more time to ensure whether the version I assigned to one of the tools that I want to install has a correct version when I just want a fuzzy description on the version I need.

For example:

Let's say I have assigned I need CUDA version 11.7 and I wrote 

install.cuda(version="11.7", cudnn="8")

in the envd description file.

But when I follow up with envd up. There always comes with errors like

ERRO[2022-11-14T16:21:48+08:00] Buildkit error: failed to solve: failed to load cache key: failed to do request: Head "https://https//docker.mirrors.sjtug.sjtu.edu.cn/v2/nvidia/cuda/manifests/11.7-cudnn8-devel-ubuntu20.04?ns=docker.io": dial tcp: lookup https on 172.30.0.3:53: no such hostg cache manifest from docker.io/tensorchord/python-cache:envd-v0.2.4-cuda-11.7-cudnn-8

Until I make it clear that there only exists image of 11.7.1-cudnn8-devel-ubuntu20.04 on the docker hub. Which means I should explicitly assign it accurately in the envd script like 

install.cuda(version="11.7.1", cudnn="8")

But as an end user, when I type install.cuda(version="11.7", cudnn="8"), I usually mean that I can accept all CUDA version with suffix 11.7 and expect envd to find latest CUDA image version with suffix 11.7. It makes me quite weird that I need to double check on the docker hub to make sure that there exists an exact version of CUDA image that match my request.

This kind of logic also matches the situation when one wants to assign a fussy system version like ubuntu20, etc.

I think this kind of fuzzy search should be supported to make envd more user-friendly.

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

popfido commented 2 years ago

Ps: I'm tired in fighting with bot to decide which labels should be added. (It's not so Intelligent as I expected :) )

gaocegege commented 2 years ago

The bot will retry if you updated the title, or the body of the issue.

gaocegege commented 2 years ago

SGTM, I think we should support this feature.

popfido commented 2 years ago

Does the buildkit support fuzzy search? It seems that in order to implement this feature, we must be able to query images in fuzzy mode.

gaocegege commented 2 years ago

I do not think buildkit supports that. We need to implement it.

xieydd commented 2 years ago

I think image is specially, we can not figure out two images with almost same name but the layer is almost totally different.

gaocegege commented 2 years ago

We can support fuzz search for CUDA, I think.

cutecutecat commented 1 year ago

I think maybe an additional checker can be added to func BuildFunc() https://github.com/tensorchord/envd/blob/5efb0b12b98e3000d0ddc6ef015734e65411ac80/pkg/builder/build.go#L54

after the cache importer is generated.

The checker will fetch all available version of images, and do a fuzzy search. If match failed, it will still raise an error, but with the hint of best version.

Do a revision might be a bit radical, as 1.17 might be revised to 11.7.1, that is unacceptable, or other rules should be implement together with fuzzy search.

This lib can be used for fuzzy search https://github.com/lithammer/fuzzysearch