Extension doesn't always compile code before running tests.

[TimKingtonFC]

If I edit code and don't save the file, running the tests sometimes executes the old code. When I click run, the files are saved, but there seems to be a race between the compiler and JUnit. Sometimes the compiler wins and things are okay, sometimes I get false test failures because JUnit is running the old code.

This is on MacOS 13.6, v0.40.0 of the runner, and vscode 1.83.1

[jdneo]

What kind of project are you working on? Maven/Gradle?

[TimKingtonFC]

The one with no build tools.

[jdneo]

Hmm, I haven't heard this kind of feedback before. The extension should complete the compilation before running the tests.

Could you share a snippet of your test cases? And the settings you are using?

[TimKingtonFC]

Here it is in action: https://github.com/microsoft/vscode-java-test/assets/13016916/43c64055-a55e-4c1a-a498-88f3e8e37008

It doesn't look like I've changed any settings in the extension. My java related settings:

  "java.format.settings.url": ".vscode/java-formatter.xml",
"java.jdt.ls.java.home": "/Library/Java/JavaVirtualMachines/jdk-18.0.2.1.jdk/Contents/Home",
  "[java]": {
    "editor.defaultFormatter": "redhat.java",
    "editor.pasteAs.enabled": false
  },

The tests aren't anything special. They're only doing things in-memory with POJOs. I can send you the whole folder if you like, but I've seen this with multiple projects so I don't think the actual code is important. The project only depends on junit-platform-console-standalone-1.10.0.jar.

Here's an example. It's from a Java course I teach. They're simulating a VM, but the test doesn't create any real threads or do anything like that.:

    /** Test execution. */
    @Test
    public void testPriSimpleExec() {
        hintContext = "(prio)";
        TestUtil.compileMethod("t1",
            "push 5\n" +
                "pop_local x\n" +
                "push 3\n" +
                "push_local x\n" +
                "add\n" +
                "pop_local y\n");
        VMThread t1 = new VMThread("t1");
        Map<String, Integer> t1Symbols = t1.getEntryPointLocals();

        List<VMThread> threadList = new ArrayList<VMThread>();
        threadList.add(t1);

        PriorityScheduler scheduler = new PriorityScheduler(threadList);

        assertSame(hint("getCurrentThread returns wrong object"), t1, scheduler.getCurrentThread());
        assertEquals(hint("symbol table should be empty initially"), 0, t1Symbols.size());

        scheduler.run(1);
        assertEquals(hint("symbol table should still be empty"), 0, t1Symbols.size());

        scheduler.run(1);
        assertEquals(hint("symbol table contains wrong number of entries"), 1, t1Symbols.size());
        assertEquals(hint("t1 symbol didn't have expected value"), 5, (int) t1Symbols.get("x"));

        scheduler.run(3);
        assertEquals(hint("symbol table contains wrong number of entries"), 1, t1Symbols.size());
        assertSame(hint("getCurrentThread returns wrong object"), t1, scheduler.getCurrentThread());

        scheduler.run(1);
        assertEquals(hint("symbol table contains wrong number of entries"), 2, t1Symbols.size());
        assertEquals(hint("t1 symbol didn't have expected value"), 5, (int) t1Symbols.get("x"));
        assertEquals(hint("symbol didn't have expected value"), 8, (int) t1Symbols.get("y"));
        assertNull(hint("getCurrentThread should return null"), scheduler.getCurrentThread());
    }

[jdneo]

Have you set java.debug.settings.forceBuildBeforeLaunch? Please check in both USER and WORKSPACE scope.

[TimKingtonFC]

No, it wasn't set. I set it just to be sure, no change.

[jdneo]

Hmm..

The video about looks very like java.debug.settings.forceBuildBeforeLaunch is disabled.

First time failed because it's running with the old class file. But the auto build will update the class file then, so the second time, the new class file has been generated.

Without this, so far, I cannot imagine other causes that makes this.

[jdneo]

Maybe share the whole folder to let me check if I can repro from my side

[TimKingtonFC]

Maybe that setting isn't working?

In User/settings.json: "java.debug.settings.forceBuildBeforeLaunch": true,

This is .vscode/settings.json: { "java.project.sourcePaths": ["src", "test"], "java.project.outputPath": "bin", "java.project.referencedLibraries": ["lib/*/.jar"], "java.format.settings.url": ".vscode/java-formatter.xml" }

Screenshot of the settings editor attached.

On Wed, Nov 1, 2023 at 7:49 AM Sheng Chen @.***> wrote:

Hmm..

The video about looks very like java.debug.settings.forceBuildBeforeLaunch is disabled.

First time failed because it's running with the old class file. But the auto build will update the class file then, so the second time, the new class file has been generated.

Without this, so far, I cannot imagine other causes that makes this.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-java-test/issues/1629#issuecomment-1788824222, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDJ6VDNOLOBSO26AUWX6DDYCIZMTAVCNFSM6AAAAAA6VBRBWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBYHAZDIMRSGI . You are receiving this because you authored the thread.Message ID: @.***>

[TimKingtonFC]

VM.zip

[jdneo]

It works on my side. What is the probability of this happening? https://github.com/microsoft/vscode-java-test/assets/6193897/8dbb8fb2-fc00-4231-a164-0175e0cdea18

[TimKingtonFC]

About 50%. I just tried it 20 times, 10 changing the test to fail and 10 changing it to pass. It worked correctly 5/10 changing it to fail, and 4/10 changing it to pass.

[jdneo]

I still suspect it an environment specific problem, because 50% is a very high ratio. If that applies to other users, we should receive quite a lot complains about this.

What java related extensions are you installed? Is this behavior also observable from your colleague/students?

[TimKingtonFC]

Here are all of my extensions: albert.TabOut DavidAnson.vscode-markdownlint dbaeumer.vscode-eslint eamodio.gitlens EditorConfig.EditorConfig emeraldwalk.RunOnSave esbenp.prettier-vscode golang.go mkhl.direnv ms-python.isort ms-python.python ms-python.vscode-pylance ms-vscode.hexeditor ms-vscode.test-adapter-converter redhat.fabric8-analytics redhat.java vadimcn.vscode-lldb VisualStudioExptTeam.intellicode-api-usage-examples VisualStudioExptTeam.vscodeintellicode vscjava.vscode-java-debug vscjava.vscode-java-dependency vscjava.vscode-java-pack vscjava.vscode-java-test vscjava.vscode-maven xaver.clang-format yaozheng.vscode-pde zxh404.vscode-proto3

I think the only Java related ones are the ones installed by the Java extension pack, and Eclipse PDE support.

On Wed, Nov 1, 2023 at 10:36 PM Sheng Chen @.***> wrote:

I still suspect it an environment specific problem, because 50% is a very high ratio. If that applies to other users, we should receive quite a lot complains about this.

What java related extensions are you installed? Is this behavior also observable from your colleague/students?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-java-test/issues/1629#issuecomment-1789969667, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDJ6VGJL6NEITZULPTDASDYCMBK5AVCNFSM6AAAAAA6VBRBWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBZHE3DSNRWG4 . You are receiving this because you authored the thread.Message ID: @.***>

[jdneo]

Sorry @TimKingtonFC, one more question:

Besides the test, is this behavior also observable if you run a main method?

[TimKingtonFC]

No, I'm not seeing it there. I created a launch.json, and I'm hitting F5 with the file edited but not saved, and it picked up the changes ten times in a row.

On Tue, Nov 7, 2023 at 1:43 AM Sheng Chen @.***> wrote:

Sorry @TimKingtonFC https://github.com/TimKingtonFC, one more question:

Besides the test, is this behavior also observable if you run a main method?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-java-test/issues/1629#issuecomment-1797907389, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDJ6VC7HDC2OKMTTRY3LG3YDHKCZAVCNFSM6AAAAAA6VBRBWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXHEYDOMZYHE . You are receiving this because you were mentioned.Message ID: @.***>

[jdneo]

I tried it again on a Windows machine today, but still cannot find a way to reproduce it.

Is this behavior also observable from your colleague/students?

Sorry that I could not give more help so far.

[TimKingtonFC]

I just set up vscode fresh on a Windows machine, and it has the same problem. I let vscode install Java 21, and have the same setup/extensions/project.

On Wed, Nov 8, 2023 at 3:19 AM Sheng Chen @.***> wrote:

I tried it again on a Windows machine today, but still cannot find a way to reproduce it.

Is this behavior also observable from your colleague/students?

Sorry that I could not give more help so far.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-java-test/issues/1629#issuecomment-1801297516, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDJ6VHFMD4HNZUARHIT65DYDM6CBAVCNFSM6AAAAAA6VBRBWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBRGI4TONJRGY . You are receiving this because you were mentioned.Message ID: @.***>

[TimKingtonFC]

I've noticed a couple of other things that might be related.

Fairly often - about once a day - I try to run my tests, and it gives me the popup that says the build failed, but the code is fine. It appears to be trying to build/run code from a few seconds earlier that didn't compile - I can tell from the error messages. Sometimes I can fix it by editing a file and saving, and sometimes I have to clean the Java workspace.

A little less often - maybe every other day - something weird happens when I'm editing. I'll be typing code, and then some of the code I've just typed disappears. It's usually about half a line of code. IIRC, hitting undo readds the code that disappears, but I'm not 100% sure on that point.

[callMeBigBen]

I had the same issue. Similar to this thread, I noticed that VSCode shows vscode java The project cannot be built *** in the Problems tab, which made me also think that somehow the java test was running against an old built version of code.

Running java: Reload Projects solved the issue for me.