search

CERC-AAI / multimodal

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
Apache License 2.0
8 stars 3 forks source link

Text generation #43

Closed floatingbigcat closed 1 year ago

floatingbigcat commented 1 year ago

support text generation with image as input.

Use instruction:

  1. put the image_path and text prompt to jsonl file, set the config of text-generation.yml eg: sample_input.jsonl {'image':'image_path', 'text':'your_prompt'}
  2. set up the environment just like the one for training.
  3. run with python deepy.py generate.py -d your_train_configs(eg: 410M.yml summit-setup.yml) text-generation.yml

To do: support the use_cache=True, which shall speed up the inference.

daniel-z-kaplan commented 1 year ago

Example on how to run: python3 deepy.py generate.py -d configs magma_pythia_410M.yml magma_setup.yml text-generation.yml

Check text-generation for the input file name, make sure you have that file created, using the {image, text} dict setup described above.

You'll need to use your job.sh source, conda env if relevant, and export torch extensions also.