The GPU memory will keep increase

[sev777]

I noticed that when continuously analyzing multiple data streams with neurons, the GPU memory usage keeps increasing, eventually leading to an overflow. Is there any way to solve this problem?

[Lut-hub]

Hi, I came through the same question. And this may be caused by the KnowledgeNeurons class not releasing its memory in time. You can try del and torch.cuda.empty_cache(), like:

for dat in tqdm(new_data[:batch_size]):
    kn_ml = KnowledgeNeurons(ml_model, tokenizer, model_type=model_type(MODEL_NAME))
    TEXT = dat["question"]
    ANS = dat["answer"]
    test_neurons = kn_ml.get_scores(prompt=TEXT, ground_truth=ANS, pbar=False).cpu().detach().clone()
    del kn_ml
    torch.cuda.empty_cache()

Hope this might help! 😃