This project can be used on large visual language models?

[jiabao-wang]

how to use it on models like BLIP2, LLAVA, MINIGPT4?

[MrYxJ]

I see that your example models are open source on huggingface, but they do not support calculation by meta model. You can download it locally and use calculate_flops, I will download its full model weight to local for testing and tuning later.

[jiabao-wang]

Thank you for your reply. If you can apply calculate-flops to large visual language models such as BLIP2, LLAVA, MiniGPT4, etc., please provide sample code.

[jiabao-wang]

from calflops import calculate_flops from torchvision import models import requests from PIL import Image from transformers import Blip2Processor, Blip2ForConditionalGeneration

def Blip2(): processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b") model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b")

raw_image = Image.open('/data/wangjiabao/data/jiabao.jpg').convert('RGB')

question = "how many people are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt")
flops, macs, params = calculate_flops(model=model,
                                      kwargs=inputs,
                                      print_results=True)
print("Salesforce/blip2-opt-2.7b FLOPs:%s   MACs:%s   Params:%s \n" % (flops, macs, params))
# out = model.generate(**inputs)
# print(processor.decode(out[0], skip_special_tokens=True))

if name == 'main': Blip2()

================= this demo code can test FLOPs of BLIP2 model. The result is: Salesforce/blip2-opt-2.7b FLOPs:750.74 GFLOPS MACs:375.25 GMACs Params:3.74 B

[jiabao-wang]

but if the model inference with the code like: output_ids = model.generate( input_ids, images=image_tensor.half().cuda(), do_sample=True, num_beams=1, num_return_sequences=num, temperature=0.2, max_new_tokens=128, use_cache=True, )

I don't know how to use the calculate_flops to calculate.