System out of memory because of linear usage

[Yessin111]

System Info

Hi, hope this is the correct way to address my issue.

When running a model to identify objects in images, memory uses keeps rising until my system can't handle any more which usually happens after +- 15 images. I think I narrowed it down to the

outputs = model(**inputs)

variable declaration, as removing this for testing purposes gets rid of the memory increases. I'll paste relevant parts of my code in the Reproduction box.

Who can help?

@sgugger

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

texts = [['an illustration of ...']] # texts has about 500 prompts it looks for

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

for partition in range(1, max_partition): # my code detects 25 images at a time, then writes the results to file
    detect(25, partition)

# it then gets the urls from a json file and gets the images as a response, after which it does the following:

def get(response):
    inputs = processor(text=texts, images=response, return_tensors="pt")
    outputs = model(**inputs)
    target_sizes = torch.Tensor([response.size[::-1]])
    results = processor.post_process(outputs=outputs, target_sizes=target_sizes)

    i = 0
    text = texts[i]
    scores, labels = results[i]["scores"], results[i]["labels"]
    return zip(scores, labels)

I hope this explains it well. If need be I can also supply the actual Python file.

Expected behavior

Not crashing my system after 2 minutes of runtime.

[sgugger]

I don't know how you want us to help without a way to reproduce the error. It looks like you're not running inference within a torch.no_grad() context manager, which will consume memories of activations saved for the backward pass, which might be the issue.

Also cc @alaradirik

[Yessin111]

Thank you for your quick reply. This is the full script:

from transformers import OwlViTProcessor, OwlViTForObjectDetection
import json
import math
import requests
import torch
from PIL import Image

dictionary = []

def sub_question_two(part_size, part):
    print()
    print("-------------------------------------------------------")
    print("Processing partition " + str(part))
    print("-------------------------------------------------------")
    global dictionary
    dictionary = []

    with open("data.json", "r") as file_json:
        json_data = json.load(file_json)

    json_data = json_data[(part - 1) * part_size:part * part_size]

    for row in json_data:
        handler(row)

    write_to_file(part)

def write_to_file(part):
    if part == 2717:
        open("data_subquestion_two.json", "w")

        with open("data_subquestion_two.json", "r+") as file_json:
            file_json.seek(0)
            json.dump(dictionary, file_json, indent=4)
    else:
        with open("data_subquestion_two.json", "r+") as file_json:
            file_data = json.loads(file_json.read())
            file_data = file_data + dictionary
            file_json.seek(0)
            json.dump(file_data, file_json, indent=4)

def handler(json_row):
    print(json_row["isbn"])
    try:
        with Image.open(requests.get(json_row["library"]["cover"], stream=True).raw) as response:
            json_row["library"]["sq2"] = get_caption(response)
    except:
        pass

    try:
        with Image.open(requests.get(json_row["amazon"]["cover"], stream=True).raw) as response:
            json_row["amazon"]["sq2"] = get_caption(response)
    except:
        pass
    dictionary.append(json_row)

def get_caption(response):
    inputs = processor(text=texts, images=response, return_tensors="pt")
    outputs = model(**inputs)
    target_sizes = torch.Tensor([response.size[::-1]])
    results = processor.post_process(outputs=outputs, target_sizes=target_sizes)

    i = 0
    text = texts[i]
    scores, labels = results[i]["scores"], results[i]["labels"]

    zipped = zip(scores, labels)
    better_zip = {}
    found = []
    for el in zipped:
        if el[1] not in found:
            found.append(el[1])
            better_zip[text[el[1]]] = round(el[0].item(), 3)
        else:
            if better_zip[text[el[1]]] < round(el[0].item(), 3):
                better_zip[text[el[1]]] = round(el[0].item(), 3)

    return dict(sorted(better_zip.items(), key=lambda item: item[1], reverse=True))

partition_size = 25
max_partition_sq2 = math.ceil(93662 / partition_size) + 1

texts = [['an illustration of a tortoise', 'an illustration of a magpie', 'an illustration of a sea turtle', 'an illustration of a general football', 'an illustration of a ambulance', 'an illustration of a ladder', 'an illustration of a toothbrush', 'an illustration of a syringe', 'an illustration of a sink', 'an illustration of a toy', 'an illustration of a organ', 'an illustration of a apple', 'an illustration of a eye', 'an illustration of a cosmetics', 'an illustration of a paddle', 'an illustration of a snowman', 'an illustration of a beer', 'an illustration of a chopsticks', 'an illustration of a beard', 'an illustration of a bird', 'an illustration of a traffic light', 'an illustration of a croissant', 'an illustration of a cucumber', 'an illustration of a radish', 'an illustration of a towel', 'an illustration of a doll', 'an illustration of a skull', 'an illustration of a washing machine', 'an illustration of a glove', 'an illustration of a belt', 'an illustration of a sunglasses', 'an illustration of a banjo', 'an illustration of a cart', 'an illustration of a ball', 'an illustration of a backpack', 'an illustration of a bike', 'an illustration of a home appliance', 'an illustration of a centipede', 'an illustration of a boat', 'an illustration of a surfboard', 'an illustration of a boot', 'an illustration of a headphones', 'an illustration of a hot dog', 'an illustration of a shorts', 'an illustration of a fast food', 'an illustration of a bus', 'an illustration of a boy', 'an illustration of a bicycle wheel', 'an illustration of a barge', 'an illustration of a laptop', 'an illustration of a miniskirt', 'an illustration of a drill', 'an illustration of a dress', 'an illustration of a bear', 'an illustration of a waffle', 'an illustration of a pancake', 'an illustration of a brown bear', 'an illustration of a woodpecker', 'an illustration of a blue jay', 'an illustration of a pretzel', 'an illustration of a bagel', 'an illustration of a tower', 'an illustration of a teapot', 'an illustration of a person', 'an illustration of a bow and arrow', 'an illustration of a swimwear', 'an illustration of a beehive', 'an illustration of a brassiere', 'an illustration of a bee', 'an illustration of a bat', 'an illustration of a starfish', 'an illustration of a popcorn', 'an illustration of a burrito', 'an illustration of a chainsaw', 'an illustration of a balloon', 'an illustration of a tent', 'an illustration of a licence plate', 'an illustration of a lantern', 'an illustration of a flashlight', 'an illustration of a billboard', 'an illustration of a tiara', 'an illustration of a limousine', 'an illustration of a necklace', 'an illustration of a carnivore', 'an illustration of a scissors', 'an illustration of a stairs', 'an illustration of a computer keyboard', 'an illustration of a printer', 'an illustration of a traffic sign', 'an illustration of a chair', 'an illustration of a shirt', 'an illustration of a poster', 'an illustration of a cheese', 'an illustration of a sock', 'an illustration of a fire hydrant', 'an illustration of a land vehicle', 'an illustration of a earrings', 'an illustration of a tie', 'an illustration of a watercraft', 'an illustration of a cabinetry', 'an illustration of a suitcase', 'an illustration of a muffin', 'an illustration of a bidet', 'an illustration of a snack', 'an illustration of a snowmobile', 'an illustration of a clock', 'an illustration of a medical equipment', 'an illustration of a cattle', 'an illustration of a cello', 'an illustration of a jet ski', 'an illustration of a camel', 'an illustration of a coat', 'an illustration of a suit', 'an illustration of a desk', 'an illustration of a cat', 'an illustration of a bronze sculpture', 'an illustration of a juice', 'an illustration of a gondola', 'an illustration of a beetle', 'an illustration of a cannon', 'an illustration of a mouse', 'an illustration of a cookie', 'an illustration of a office', 'an illustration of a fountain', 'an illustration of a coin', 'an illustration of a calculator', 'an illustration of a cocktail', 'an illustration of a computer monitor', 'an illustration of a box', 'an illustration of a christmas tree', 'an illustration of a cowboy hat', 'an illustration of a hiking equipment', 'an illustration of a studio couch', 'an illustration of a drum', 'an illustration of a dessert', 'an illustration of a wine rack', 'an illustration of a drink', 'an illustration of a zucchini', 'an illustration of a ladle', 'an illustration of a mouth', 'an illustration of a dairy', 'an illustration of a dice', 'an illustration of a oven', 'an illustration of a dinosaur', 'an illustration of a couch', 'an illustration of a cricket ball', 'an illustration of a winter melon', 'an illustration of a whiteboard', 'an illustration of a door', 'an illustration of a hat', 'an illustration of a shower', 'an illustration of a fedora', 'an illustration of a guacamole', 'an illustration of a dagger', 'an illustration of a scarf', 'an illustration of a dolphin', 'an illustration of a sombrero', 'an illustration of a tin can', 'an illustration of a mug', 'an illustration of a tap', 'an illustration of a harbor seal', 'an illustration of a stretcher', 'an illustration of a goggles', 'an illustration of a human body', 'an illustration of a roller skates', 'an illustration of a coffee cup', 'an illustration of a cutting board', 'an illustration of a blender', 'an illustration of a plumbing fixture', 'an illustration of a stop sign', 'an illustration of a office supplies', 'an illustration of a volleyball', 'an illustration of a vase', 'an illustration of a slow cooker', 'an illustration of a wardrobe', 'an illustration of a coffee', 'an illustration of a paper towel', 'an illustration of a personal care', 'an illustration of a food', 'an illustration of a sun hat', 'an illustration of a tree house', 'an illustration of a skirt', 'an illustration of a gas stove', 'an illustration of a salt and pepper shakers', 'an illustration of a mechanical fan', 'an illustration of a fruit', 'an illustration of a french fries', 'an illustration of a nightstand', 'an illustration of a barrel', 'an illustration of a kite', 'an illustration of a tart', 'an illustration of a treadmill', 'an illustration of a fox', 'an illustration of a flag', 'an illustration of a horn', 'an illustration of a window blind', 'an illustration of a foot', 'an illustration of a golf cart', 'an illustration of a jacket', 'an illustration of a egg', 'an illustration of a street light', 'an illustration of a guitar', 'an illustration of a pillow', 'an illustration of a leg', 'an illustration of a isopod', 'an illustration of a grape', 'an illustration of a ear', 'an illustration of a power plugs and sockets', 'an illustration of a panda', 'an illustration of a giraffe', 'an illustration of a woman', 'an illustration of a door handle', 'an illustration of a rhinoceros', 'an illustration of a bathtub', 'an illustration of a goldfish', 'an illustration of a houseplant', 'an illustration of a goat', 'an illustration of a baseball bat', 'an illustration of a baseball glove', 'an illustration of a mixing bowl', 'an illustration of a marine invertebrates', 'an illustration of a kitchen utensil', 'an illustration of a light switch', 'an illustration of a house', 'an illustration of a horse', 'an illustration of a stationary bicycle', 'an illustration of a ceiling fan', 'an illustration of a sofa bed', 'an illustration of a harp', 'an illustration of a sandal', 'an illustration of a bicycle helmet', 'an illustration of a saucer', 'an illustration of a harpsichord', 'an illustration of a hair', 'an illustration of a hamster', 'an illustration of a curtain', 'an illustration of a bed', 'an illustration of a kettle', 'an illustration of a fireplace', 'an illustration of a scale', 'an illustration of a drinking straw', 'an illustration of a insect', 'an illustration of a invertebrate', 'an illustration of a food processor', 'an illustration of a bookcase', 'an illustration of a refrigerator', 'an illustration of a wood-burning stove', 'an illustration of a punching bag', 'an illustration of a common fig', 'an illustration of a jaguar', 'an illustration of a golf ball', 'an illustration of a fashion accessory', 'an illustration of a alarm clock', 'an illustration of a filing cabinet', 'an illustration of a artichoke', 'an illustration of a table', 'an illustration of a tableware', 'an illustration of a kangaroo', 'an illustration of a koala', 'an illustration of a knife', 'an illustration of a bottle', 'an illustration of a lynx', 'an illustration of a lavender', 'an illustration of a lighthouse', 'an illustration of a dumbbell', 'an illustration of a head', 'an illustration of a bowl', 'an illustration of a porch', 'an illustration of a lizard', 'an illustration of a billiard table', 'an illustration of a mammal', 'an illustration of a mouse', 'an illustration of a motorcycle', 'an illustration of a musical instrument', 'an illustration of a swim cap', 'an illustration of a frying pan', 'an illustration of a snowplow', 'an illustration of a bathroom cabinet', 'an illustration of a missile', 'an illustration of a bust', 'an illustration of a man', 'an illustration of a milk', 'an illustration of a plate', 'an illustration of a mobile phone', 'an illustration of a baked goods', 'an illustration of a mushroom', 'an illustration of a pitcher', 'an illustration of a mirror', 'an illustration of a lifejacket', 'an illustration of a table tennis racket', 'an illustration of a musical keyboard', 'an illustration of a scoreboard', 'an illustration of a briefcase', 'an illustration of a kitchen knife', 'an illustration of a tennis ball', 'an illustration of a plastic bag', 'an illustration of a oboe', 'an illustration of a chest of drawers', 'an illustration of a ostrich', 'an illustration of a piano', 'an illustration of a girl', 'an illustration of a plant', 'an illustration of a potato', 'an illustration of a sports equipment', 'an illustration of a pasta', 'an illustration of a penguin', 'an illustration of a pumpkin', 'an illustration of a pear', 'an illustration of a infant bed', 'an illustration of a polar bear', 'an illustration of a mixer', 'an illustration of a cupboard', 'an illustration of a jacuzzi', 'an illustration of a pizza', 'an illustration of a digital clock', 'an illustration of a pig', 'an illustration of a reptile', 'an illustration of a rifle', 'an illustration of a lipstick', 'an illustration of a skateboard', 'an illustration of a raven', 'an illustration of a high heels', 'an illustration of a red panda', 'an illustration of a rose', 'an illustration of a rabbit', 'an illustration of a sculpture', 'an illustration of a saxophone', 'an illustration of a shotgun', 'an illustration of a seafood', 'an illustration of a submarine sandwich', 'an illustration of a snowboard', 'an illustration of a sword', 'an illustration of a picture frame', 'an illustration of a sushi', 'an illustration of a loveseat', 'an illustration of a ski', 'an illustration of a squirrel', 'an illustration of a tripod', 'an illustration of a stethoscope', 'an illustration of a submarine', 'an illustration of a scorpion', 'an illustration of a segway', 'an illustration of a bench', 'an illustration of a snake', 'an illustration of a coffee table', 'an illustration of a skyscraper', 'an illustration of a sheep', 'an illustration of a television', 'an illustration of a trombone', 'an illustration of a tea', 'an illustration of a tank', 'an illustration of a taco', 'an illustration of a telephone', 'an illustration of a tiger', 'an illustration of a strawberry', 'an illustration of a trumpet', 'an illustration of a tree', 'an illustration of a tomato', 'an illustration of a train', 'an illustration of a tool', 'an illustration of a picnic basket', 'an illustration of a trousers', 'an illustration of a bowling equipment', 'an illustration of a football helmet', 'an illustration of a truck', 'an illustration of a coffeemaker', 'an illustration of a violin', 'an illustration of a vehicle', 'an illustration of a handbag', 'an illustration of a wine', 'an illustration of a weapon', 'an illustration of a wheel', 'an illustration of a worm', 'an illustration of a wok', 'an illustration of a whale', 'an illustration of a zebra', 'an illustration of a auto part', 'an illustration of a jug', 'an illustration of a cream', 'an illustration of a monkey', 'an illustration of a lion', 'an illustration of a bread', 'an illustration of a platter', 'an illustration of a chicken', 'an illustration of a eagle', 'an illustration of a helicopter', 'an illustration of a owl', 'an illustration of a duck', 'an illustration of a turtle', 'an illustration of a hippopotamus', 'an illustration of a crocodile', 'an illustration of a toilet', 'an illustration of a toilet paper', 'an illustration of a squid', 'an illustration of a clothing', 'an illustration of a footwear', 'an illustration of a lemon', 'an illustration of a spider', 'an illustration of a deer', 'an illustration of a frog', 'an illustration of a banana', 'an illustration of a rocket', 'an illustration of a wine glass', 'an illustration of a countertop', 'an illustration of a tablet computer', 'an illustration of a waste container', 'an illustration of a swimming pool', 'an illustration of a dog', 'an illustration of a book', 'an illustration of a elephant', 'an illustration of a shark', 'an illustration of a candle', 'an illustration of a leopard', 'an illustration of a porcupine', 'an illustration of a flower', 'an illustration of a canary', 'an illustration of a cheetah', 'an illustration of a palm tree', 'an illustration of a hamburger', 'an illustration of a maple', 'an illustration of a building', 'an illustration of a fish', 'an illustration of a lobster', 'an illustration of a asparagus', 'an illustration of a furniture', 'an illustration of a hedgehog', 'an illustration of a airplane', 'an illustration of a spoon', 'an illustration of a otter', 'an illustration of a bull', 'an illustration of a oyster', 'an illustration of a convenience store', 'an illustration of a bench', 'an illustration of a ice cream', 'an illustration of a caterpillar', 'an illustration of a butterfly', 'an illustration of a parachute', 'an illustration of a orange', 'an illustration of a antelope', 'an illustration of a moths and butterflies', 'an illustration of a window', 'an illustration of a closet', 'an illustration of a castle', 'an illustration of a jellyfish', 'an illustration of a goose', 'an illustration of a mule', 'an illustration of a swan', 'an illustration of a peach', 'an illustration of a seat belt', 'an illustration of a raccoon', 'an illustration of a fork', 'an illustration of a lamp', 'an illustration of a camera', 'an illustration of a squash', 'an illustration of a racket', 'an illustration of a face', 'an illustration of a arm', 'an illustration of a vegetable', 'an illustration of a unicycle', 'an illustration of a falcon', 'an illustration of a snail', 'an illustration of a shellfish', 'an illustration of a cabbage', 'an illustration of a carrot', 'an illustration of a mango', 'an illustration of a jeans', 'an illustration of a flowerpot', 'an illustration of a pineapple', 'an illustration of a drawer', 'an illustration of a stool', 'an illustration of a envelope', 'an illustration of a cake', 'an illustration of a dragonfly', 'an illustration of a sunflower', 'an illustration of a microwave oven', 'an illustration of a honeycomb', 'an illustration of a marine mammal', 'an illustration of a sea lion', 'an illustration of a ladybug', 'an illustration of a shelf', 'an illustration of a watch', 'an illustration of a candy', 'an illustration of a salad', 'an illustration of a parrot', 'an illustration of a handgun', 'an illustration of a sparrow', 'an illustration of a van', 'an illustration of a spice rack', 'an illustration of a light bulb', 'an illustration of a corded phone', 'an illustration of a sports uniform', 'an illustration of a tennis racket', 'an illustration of a wall clock', 'an illustration of a serving tray', 'an illustration of a kitchen & dining room table', 'an illustration of a dog bed', 'an illustration of a cake stand', 'an illustration of a bathroom accessory', 'an illustration of a kitchen appliance', 'an illustration of a tire', 'an illustration of a ruler', 'an illustration of a luggage and bags', 'an illustration of a microphone', 'an illustration of a broccoli', 'an illustration of a umbrella', 'an illustration of a pastry', 'an illustration of a grapefruit', 'an illustration of a animal', 'an illustration of a bell pepper', 'an illustration of a turkey', 'an illustration of a lily', 'an illustration of a pomegranate', 'an illustration of a doughnut', 'an illustration of a glasses', 'an illustration of a nose', 'an illustration of a pen', 'an illustration of a ant', 'an illustration of a car', 'an illustration of a aircraft', 'an illustration of a hand', 'an illustration of a teddy bear', 'an illustration of a watermelon', 'an illustration of a cantaloupe', 'an illustration of a dishwasher', 'an illustration of a flute', 'an illustration of a balance beam', 'an illustration of a sandwich', 'an illustration of a shrimp', 'an illustration of a sewing machine', 'an illustration of a binoculars', 'an illustration of a rays and skates', 'an illustration of a ipod', 'an illustration of a accordion', 'an illustration of a willow', 'an illustration of a crab', 'an illustration of a crown', 'an illustration of a seahorse', 'an illustration of a perfume', 'an illustration of a alpaca', 'an illustration of a taxi', 'an illustration of a canoe', 'an illustration of a remote control', 'an illustration of a wheelchair', 'an illustration of a rugby ball', 'an illustration of a helmet']]

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")

for partition in range(2717, max_partition_sq2):
    sub_question_two(partition_size, partition)

data.json contains the urls to the images. I will look into torch.no_grad().

[Yessin111]

Quick update. It looks like the torch.no_grad() function did the trick! Adding it after the get_caption(response) call made it so the memory usage remains stable at around 12G memory usage. I can't thank you enough, I've been up trying to fix this since I created this issue and will now sleep.