The web demo output is not as expected

[Arrkwen]

Describe Thanks for the amazing work [kosmos-2]. I deploy the web-demo in my local host following the Readme, and the running script like this:

#!/bin/bash

partition=
model_path=model/kosmos-2.pt
master_port=
master_ip=

srun -p $partition -N 1 -n 1 --cpus-per-task=1 -J kosmos-2 \
--nodelist=$master_ip --gres=gpu:1 \
python -m torch.distributed.launch --master_addr=$master_ip --master_port=$master_port --nproc_per_node=1 \
    demo/gradio_app.py None \
    --task generation_obj \
    --path $model_path \
    --model-overrides "{'visual_pretrained': '',
            'dict_path':'data/dict.txt'}" \
    --dict-path 'data/dict.txt' \
    --required-batch-size-multiple 1 \
    --remove-bpe=sentencepiece \
    --max-len-b 500 \
    --add-bos-token \
    --beam 1 \
    --buffer-size 1 \
    --image-feature-length 64 \
    --locate-special-token 1 \
    --batch-size 1 \
    --nbest 1 \
    --no-repeat-ngram-size 3 \
    --location-bin-size 32

then I run the example in the demo , but get the output as flow:

inputs [image]/tmp/user_input_test_image.jpg<tab><grounding>An image of
S-7     <image>. the, to and of as in I that' for is was- on’ it with The as at bet he have from by are " you his “ this said not has an ( but had we her they will my or were their): up about out who one all been she can more would It</image><grounding> An image of
W-7     0.431   seconds
H-7     -10.600826263427734     <image>. the, to and of as in I that' for is was- on’ it with The as at bet he have from by are " you his “ this said not has an ( but had we her they will my or were their): up about out who one all been she can more would It</image><grounding> An image of a a a
D-7     -10.600826263427734     <image>. the, to and of as in I that' for is was- on’ it with The as at bet he have from by are " you his “ this said not has an ( but had we her they will my or were their): up about out who one all been she can more would It</image><grounding> An image of a a a
P-7     -16.8181 -13.7804 -9.9349 -8.7125 -11.9695 -10.1048 -11.6673 -7.4806 -11.6694 -6.3118 -14.2826 -7.3461 -7.9916 -7.6706 -9.3138 -13.8442 -8.4752 -9.5478 -15.4756 -6.7853 -9.9017 -8.0266 -9.1790 -7.0517 -9.3948 -7.8633 -8.6803 -11.2337 -7.1120 -9.0534

can you help me?

[donglixp]

You could confirm that whether you followed the same steps as in https://github.com/microsoft/unilm/tree/master/kosmos-2#setup

[donglixp]

Especially the correct local-version fairseq as shown in https://github.com/microsoft/unilm/tree/master/kosmos-2#L1

[donglixp]

You could also refer to https://github.com/microsoft/unilm/issues/1167#issuecomment-1615564431

[Arrkwen]

Especially the correct local-version fairseq as shown in https://github.com/microsoft/unilm/tree/master/kosmos-2#L1

yep, I pip install -e fairseq $:pip show fairseq

Name: fairseq
Version: 1.0.0a0+874dfed
Summary: Facebook AI Research Sequence-to-Sequence Toolkit
Home-page: https://github.com/pytorch/fairseq
Author: 
Author-email: 
License: 
Location: /python/unilm/kosmos-2/fairseq

[Arrkwen]

You could confirm that whether you followed the same steps as in https://github.com/microsoft/unilm/tree/master/kosmos-2#setup

I cant run the docker container due to some terrible cluster configuration. I just start from step 2.

# virtual env
conda create -n kosmos python=3.8

# torch
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

# triton and ninja
pip install triton ninja

pip install -e fairseq
pip install -e  infinibatch
pip install ftfy
pip install -e torchscale
pip install -e open_clip
pip install tiktoken
pip install sentencepiece

# deepspeed
git clone -b jeffra/engine-xthru-v2 https://github.com/microsoft/DeepSpeed.git 
cd DeepSpeed
python setup.py install --user

# xformers
git clone https://github.com/facebookresearch/xformers.git 
git submodule update --init --recursive

python setup.py install

[Arrkwen]

so, can you show me the version of all dependencies?

[pengzhiliang]

Hi, @Arrkwen

so, can you show me the version of all dependencies?

I run pip freeze > requirements.txt in the docker after setup, and get requirements.txt. Please check it.

[BIGBALLON]

@Arrkwen Do you have any progress here? Which requirement is causing the issue?

[Arrkwen]

@Arrkwen Do you have any progress here? Which requirement is causing the issue?

I don't know what is causing the error in my cluster env，maybe xformers, which build in my local host, than upload to cluster node. lastly, I get the expect result by replaying operation in README on my local host.