search

hammoudhasan / SynthCLIP

Code base of SynthCLIP: CLIP training with purely synthetic text-image pairs from LLMs and TTIs.
88 stars 1 forks source link

TextGen missing setup steps? #8

Open escorciav opened 2 hours ago

escorciav commented 2 hours ago 
$ python captions_generator.py --save_path synthetic_captions                              --generation_idx 0                              --concept_bank_size -1                              --metadata metadata.json

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.3 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "/docker/perseverant_snail/projects/genai-research/synthclip/TextGen/captions_generator.py", line 18, in <module>
    from vllm import LLM, SamplingParams
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 6, in <module>
    from vllm.config import (CacheConfig, ModelConfig, ParallelConfig,
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/config.py", line 6, in <module>
    import torch
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/torch/__init__.py", line 1382, in <module>
    from .functional import *  # noqa: F403
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/torch/functional.py", line 7, in <module>
    import torch.nn.functional as F
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/torch/nn/__init__.py", line 1, in <module>
    from .modules import *  # noqa: F403
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/torch/nn/modules/__init__.py", line 35, in <module>
    from .transformer import TransformerEncoder, TransformerDecoder, \
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/torch/nn/modules/transformer.py", line 20, in <module>
    device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
Traceback (most recent call last):
  File "/docker/perseverant_snail/projects/genai-research/synthclip/TextGen/captions_generator.py", line 123, in <module>
    main(args)
  File "/docker/perseverant_snail/projects/genai-research/synthclip/TextGen/captions_generator.py", line 81, in main
    rewriter = Captions_Generator(args)
  File "/docker/perseverant_snail/projects/genai-research/synthclip/TextGen/captions_generator.py", line 48, in __init__
    self.llm = LLM(
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 109, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 351, in from_engine_args
    engine_configs = engine_args.create_engine_configs()
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 272, in create_engine_configs
    model_config = ModelConfig(self.model, self.tokenizer,
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/config.py", line 103, in __init__
    self.hf_config = get_config(self.model, trust_remote_code, revision)
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 35, in get_config
    raise e
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 23, in get_config
    config = AutoConfig.from_pretrained(
  File "/docker/perseverant_snail/envs/synthclip/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1049, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in ./LLMs/Mistral-7B-Instruct-v0.2. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: albert, align, altclip, audio-spectrogram-transformer, autoformer, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deformable_detr, deit, depth_anything, deta, detr, dinat, dinov2, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, git, glm, glpn, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, graphormer, grounding-dino, groupvit, hiera, hubert, ibert, idefics, idefics2, idefics3, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rwkv
escorciav commented 2 hours ago

I noticed the script created the folder .TextGen/LLMs/Mistral-7B-Instruct-v0.2

should I download the weights and place it there? :)