Statix test does not open sometimes.

[AZWN]

Summary

Sometimes, nothing happens when executing an stxtest.

What you did

Create a new project, and create src/exam.stxtest, with the following content:

resolve {s1 s2 s3 ps}
  new s1, new s2, new s3,
  s1 -A-> s2,
  s1 -B-> s3,
  !foo["A"] in s2,
  !foo["B"] in s3,
  query foo filter (A|B)* and { x' :- true } min and true in s1 -> ps

signature
  relations foo : string
  name-resolution labels A B

Then execute Statix > Evaluate Test. Then correct the arrow (-> to |->). Then evaluate the test again.

What you expected to happen

I expected an editor with the test result to open.

What actually happened

Nothing happened.

Context

• Spoofax version: 0.15.3

Additional information

The log shows that the StatixAnalyzeFile task on the test file returns Err(MessagesException(KeyedMessages(0 messages))). To us (@Konpon96 and me), this looks suspicious.

stxtest-not-open.txt

[Gohla]

When I create that file and fix the arrow, I still get the following error in the editor: Unsolved: Duplicate imported constructor NAME(), and when I run Evaluate test I get the following errors in the details view:

Running command 'Evaluate test' on 'file 'eclipse-resource##/empty/src/exam.stxtest' failed.

Printing with context:
  currentDir: ./ = /empty/
Messages of ./src/exam.stxtest:
  ERROR 184@6: Syntax error, expected: 'imports'
  ERROR 184-185@6: Unresolved import module "."
  ERROR 184-185@6: Unsolved: ?./src/exam.stxtest-d-1 ?=S=> ?./src/exam.stxtest-s_syn'-1
  ERROR 184-185@6: Unsolved: Duplicate imported edge labels / relations NAME()
  ERROR 184-185@6: Unsolved: Duplicate imported constructor NAME()
  ERROR 184-185@6: Unsolved: Duplicate imported constraints NAME()
  ERROR 184-185@6: Unsolved: ?./src/exam.stxtest-importedRules-2 is set Reachables(?./src/exam.stxtest-s_syn'-1,Namespace("Rule"))
  ERROR 184-185@6: Unsolved: ?./src/exam.stxtest-otherImportedRules-1 is set Diff(Reachables(#./src/exam.stxtest-s_syn-1,Namespace("Rule")),NoProjection(),Union(Declarations(#./src/exam.stxtest-s_syn-1,Namespace("Rule")),Reachables(?./src/exam.stxtest-s_syn'-1,Namespace("Rule"))))
  ERROR 184-185@6: Unsolved: Duplicate imported namespaces NAME()
  ERROR 184-185@6: Unsolved: ?./src/exam.stxtest-d-1 ?=I=> ?./src/exam.stxtest-s_mod'-1
  ERROR 184-185@6: Unsolved: Duplicate imported resolution policies for NAME()

so there are seemingly still errors in the file?

[AZWN]

Oh, my bad, there was a . after ps, that should not be there. For me it works now. @Konpon96 can you confirm?

[Gohla]

A way to make it fail is to put it outside of the src directory, which the constraint analyzer for Statix ignores (it only analyzes files in the main source directory and include directories). This is what makes StatixAnalyzeFile return Err(MessagesException(KeyedMessages(0 messages))). There is an error message in that exception, but that is not printed.

[AZWN]

Ok, looking in the log, that might very well have been the case. Perhaps we should the either

• document that
• create some sort of warning in the editor when this happens
• support files outside src

[Gohla]

I will make sure to provide a better exception when this goes wrong. We could scan for all .stxtest files in the entire project, instead of only the source directory, do you think that is ok?

[AZWN]

Yes, I think that would be fine. stxtests never export anything to other files, so I don't see how that could mess up anything.

[AZWN]

But perhaps this problem could occur for other languages as well?

[kponichtera]

I have moved the stxtest file to the src directory and tried to run the test - nothing happened. After removing the . after ps the test still didn't run. However, after cleaning the project, the test worked and opened the new editor window with a result.

Here is the console output from the successful execution: stxtest_successful_execution.log. Unfortunately, I don't have the output from the previous executions, but I replicated the issue in the other Spoofax project:

1. Running the test with stxtest file having a typo and being in the project's root - failure: stxtest_2_typo_and_project_root.txt
2. Running the test with the file moved to src directory and the typo still present - failure: stxtest_2_typo.log
3. Running the test with the file moved to src directory and fixing the typo - failure: stxtest_2_fixed_typo.log
4. Running the test with the file moved, typo fixed AND project cleaned - success: stxtest_2_after_project_clean.log

[Gohla]

Thanks for these reproduction steps. I believe that there are several bugs related to deleting/moving files that cause the problems that you describe, which are fixed after a clean. There is already a bug report for this: https://github.com/metaborg/spoofax-pie/issues/42. In any case, a proper exception is now generated when a constraint analysis result is not found, and .stxtests can be anywhere in the project directory, fixed by: https://github.com/metaborg/spoofax-pie/commit/665cf9c10e54f7ce828188a7b68ae5c8c2aefa9d.