Module dependency version differences

[eero-t]

(This is continuation / expansion of #112 which was closed without any explanation.)

Looking at the latest main branch code...

For some components, there are different dependency versions listed in requirements.txt files:

$ find -name requirements.txt | xargs grep -h == | sort -u
frontend==0.0.3
haystack-ai==2.2.4
huggingface_hub==0.24.0
langchain #==0.1.12
langchain==0.1.16
langchain==0.2.12
langchain==0.2.9
langchain-community==0.2.1
langchain_community==0.2.5
langchain_community==0.2.7
langchain-core==0.2.21
langchainhub==0.1.20
llama-index-embeddings-huggingface==0.2.0
lm-eval==0.4.2
motor==3.4.0
openai==1.35.13
pgvector==0.2.5
prometheus-fastapi-instrumentator==7.0.0
pydantic #==1.10.13
pydantic==2.7.2
pydantic==2.7.3
pydantic==2.8.2
pymilvus==2.4.3
pymupdf==1.24.5
python-bidi==0.4.2
python-docx==0.8.11
sentence-transformers==3.0.1
torch==1.13.1 --index-url https://download.pytorch.org/whl/cpu
torchaudio==0.13.1 --index-url https://download.pytorch.org/whl/cpu
torchvision==0.14.1 --index-url https://download.pytorch.org/whl/cpu
unstructured[all-docs]==0.11.5

Both for langchain and langchain_community:

$ find -name requirements.txt | xargs grep langchain.*==
./comps/agent/langchain/requirements.txt:langchain==0.2.9
./comps/agent/langchain/requirements.txt:langchain_community==0.2.7
./comps/agent/langchain/requirements.txt:langchainhub==0.1.20
./comps/knowledgegraphs/requirements.txt:langchain_community==0.2.5
./comps/ragas/tgi/requirements.txt:langchain==0.1.16
./comps/dataprep/redis/langchain/requirements.txt:langchain==0.2.12
./comps/llms/faq-generation/tgi/requirements.txt:langchain==0.1.16
./comps/llms/text-generation/vllm/requirements.txt:langchain==0.1.16
./comps/llms/text-generation/ollama/requirements.txt:langchain==0.1.16
./comps/llms/text-generation/ray_serve/requirements.txt:langchain==0.1.16
./comps/llms/text-generation/vllm-ray/requirements.txt:langchain==0.1.16
./comps/llms/text-generation/vllm-xft/requirements.txt:langchain==0.1.16
./comps/llms/summarization/tgi/requirements.txt:langchain #==0.1.12
./comps/llms/requirements.txt:langchain==0.1.16
./comps/lvms/video-llama/server/requirements.txt:langchain==0.2.9
./comps/lvms/video-llama/server/requirements.txt:langchain-community==0.2.1
./comps/lvms/video-llama/server/requirements.txt:langchain-core==0.2.21

And pydantic:

$ find -name requirements.txt | xargs grep pydantic.*==
./comps/retrievers/langchain/milvus/requirements.txt:pydantic==2.7.3
./comps/agent/langchain/requirements.txt:pydantic #==1.10.13
./comps/asr/requirements.txt:pydantic==2.7.2
./comps/finetuning/requirements.txt:pydantic==2.8.2
./comps/llms/utils/lm-eval/requirements.txt:pydantic==2.7.2
./comps/lvms/requirements.txt:pydantic==2.7.2

Is this intentional, or just a bug?

[eero-t]

This is somewhat related to #265.

[wangkl2]

@eero-t Thanks for reporting this. Currently the new examples are enabled gradually and keep added to new releases, each of the examples might be based on different base library versions during developing and evolving due to different time frame. The Docker images are separate for all use cases. And we try to make sure each of the examples works.

@kevinintel Do we have plans to align all the versions of the common dependencies for all examples, such as langchain, langchain-community and pydantic?

[wangkl2]

@kevinintel Could you please comment on this issue? Thanks. The OP has some concerns that each example or each microservices components inside the example uses different versions of base library such as langchain, further putting obstacles into creating a common based docker images for those components to reduce the image size.

[kevinintel]

@eero-t Thanks for reporting this. Currently the new examples are enabled gradually and keep added to new releases, each of the examples might be based on different base library versions during developing and evolving due to different time frame. The Docker images are separate for all use cases. And we try to make sure each of the examples works.

@kevinintel Do we have plans to align all the versions of the common dependencies for all examples, such as langchain, langchain-community and pydantic?

if this is a community project, version would be different

[eero-t]

if this is a community project, version would be different

@kevinintel What do you mean?

Current Dockerfiles are awful. They should look something like this:

FROM opea/components-base
COPY some.py /home/user/
ENTRYPOINT ["python", "some.py"]

Which results in KB sized container (on top of the shared base image), but instead they are monsters each taking nearly GB of network bandwidth, disk space etc.

Making sure that same versions of dependencies work, is one step in fixing that i.e. getting to common base image(s).

---

@feng-intel Why did you close this?

[feng-intel]

@eero-t No reply for a long time about the before answer.