"Modify with LLM" may insert an entire test suite into test case code editor

[Vladislav0Art]

Describe the bug I used GPT-4o for test generation for the following class:

public class Calc {
    public int sum(int a, int b) {
        return a + b; // line 3
    }

    public int multiply(int a, int b) {
        return a * b;
    }

    public int sumWithOther(Calc other, int a, int b) {
        return other.sum(a, b);
    }

    public int sumWithOther(Calc1 other, int a, int b) {
        return other.plus(a, b);
    }
}

Then received test cases with the following test case among them:

import org.junit.Test;

import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.*;

public class GeneratedMultiply_LargeNumbers_Test {

    @Test
    public void multiply_LargeNumbers_Test() {
        Calc calc = new Calc();
        int result = calc.multiply(1000000, 2000000);
        assertEquals(2000000000000L, result);
    }
}

Then sent a prompt via the "Modify with LLM" button containing "Is not it an overflow of ints?"; the LLM responded with an entire test suite. Thus the test suite was inserted into the code editor dedicated to a single test case:

import org.junit.Test;

import static org.junit.Assert.assertEquals;

public class GeneratedMultiply_LargeNumbers_Test {

    @Test
    public void multiply_PositiveNumbers_Test() {
        Calc calc = new Calc();
        int result = calc.multiply(2, 3);
        assertEquals(6, result);

    }

    @Test
    public void multiply_NegativeNumbers_Test() {
        Calc calc = new Calc();
        int result = calc.multiply(-2, -3);
        assertEquals(6, result);

    }

    @Test
    public void multiply_PositiveAndNegativeNumbers_Test() {
        Calc calc = new Calc();
        int result = calc.multiply(2, -3);
        assertEquals(-6, result);

    }

    @Test
    public void multiply_Zero_Test() {
        Calc calc = new Calc();
        int result = calc.multiply(0, 3);
        assertEquals(0, result);

    }

    @Test
    public void multiply_LargeNumbers_Test() {
        Calc calc = new Calc();
        int result = calc.multiply(1000000, 2000000);
        int expectedOverflow = (int) ((long) 1000000 * 2000000);
        assertEquals(expectedOverflow, result);

    }

}

To Reproduce Steps to reproduce the behavior (it may not always be reproducible):

1. Select a class under test and request LLM test generation
2. Select any test case and use "Modify with LLM" asking a vague question (i.e., not giving an exact extraction)
3. LLM may respond with the entire test suite.

Expected behavior Only an updated test case should be populated into the mini-editor.

Screenshots

https://github.com/JetBrains-Research/TestSpark/assets/47076189/12351158-8a31-43ad-b233-8305106884da