Open SimonCockx opened 1 year ago
Repro: https://github.com/SimonCockx/xtext-assertAllWhitespaceIsFormatted/tree/conditionalreplacer-bug
(on branch conditionalreplacer-bug
)
The language features expressions that are preceded by Expression:
, e.g.,
Expression:
if True then True exists
Expression:
True
Grammar:
Model:
('Expression' ':'
exprs+=Expression)*
;
Expression:
ExistsOperation
;
ExistsOperation returns Expression:
Primary =>({ExistsOperation.argument=current} 'exists')*
;
Primary returns Expression:
{BooleanLiteral} 'True'
| {ConditionalExpression} 'if' condition=Expression 'then' then=Expression
;
Formatting does the following:
indentDec
, this rule will be broken)if-then
expression is too long, put the then
branch on a new line.exists
operation is too long, put the operation on a new line and indent.Formatter:
class MyDslFormatter extends AbstractFormatter2 {
def dispatch void format(Model model, extension IFormattableDocument document) {
model.surround[indent]
model.regionFor.keywords('Expression').forEach[
prepend[newLine]
append[noSpace]
]
model.regionFor.keywords(':').forEach[
append[newLine]
]
model.exprs.forEach[
surround[indent]
format
]
}
def dispatch void format(ExistsOperation existsOperation, extension IFormattableDocument document) {
val existsRegion = existsOperation.regionForEObject
val existsRegionAndNext = existsRegion.merge(existsRegion.nextHiddenRegion)
formatConditionally(existsRegionAndNext.offset, existsRegionAndNext.length,
[doc | // case: long expression
val extension singleLineDoc = doc.requireFitsInLine
existsOperation.regionFor.keyword('exists')
.prepend[oneSpace]
existsOperation.argument
.format
],
[extension doc | // case: short expression
existsOperation.regionFor.keyword('exists')
.prepend[newLine]
set(
existsOperation.argument.nextHiddenRegion,
existsOperation.nextHiddenRegion,
[indent]
)
existsOperation.argument
.format
]
)
}
def dispatch void format(ConditionalExpression conditional, extension IFormattableDocument document) {
conditional.regionFor.keywords('if', 'then').forEach[
append[oneSpace]
]
val conditionalRegion = conditional.regionForEObject
val conditionalRegionAndNext = conditionalRegion.merge(conditionalRegion.nextHiddenRegion)
formatConditionally(conditionalRegionAndNext.offset, conditionalRegionAndNext.length,
[doc | // case: short conditional
val extension singleLineDoc = doc.requireFitsInLine
conditional.regionFor.keyword('then')
.prepend[oneSpace]
conditional.condition.format
conditional.then.format
],
[extension doc | // case: long conditional
conditional.regionFor.keyword('then')
.prepend[newLine]
conditional.condition.format
conditional.then.format
]
)
}
}
As a test, I try to format the following code.
Expression: if True then True exists
Expression : True
I would expect the result to be the following.
Expression:
if True
then True
exists
Expression:
True
The actual result is the following:
Expression:
if True
then True
exists
Expression:
True
Note that a region without indentation should never exist, because I've indented the whole model once.
Failing test:
class FormattingTest {
@Inject
FormatterTestHelper helper;
@Test
def void testWhiteSpaceIsFormattedBug() {
helper.assertFormatted[
preferences[
put(FormatterPreferenceKeys.maxLineWidth, 10);
]
toBeFormatted = '''
Expression: if True then True exists
Expression : True
'''
expectation = '''
«""»
Expression:
if True
then True
exists
Expression:
True
'''
]
}
}
Patching the HiddenRegionFormattingMerger
fixes this issue.
how does your patch look like? does it break any existing tests in xtext-core?
My patch simply removes the first couple of lines of HiddenRegionFormattingMerger::merge
:
public IHiddenRegionFormatting merge(List<? extends IHiddenRegionFormatting> conflicting) {
IHiddenRegionFormatting result = formatter.createHiddenRegionFormatting();
// Reversed so the merging order is consistent with the special case above
for (IHiddenRegionFormatting conflict : Lists.reverse(conflicting))
result.mergeValuesFrom(conflict);
return result;
}
I haven't tried running the tests, but I can try that today. I suspect there will be failures, given the warning in the comment on the original method, i.e.,
// If there are only 2 conflicts,
// usages of this method expect the second value to be updated to the merge result
Hi @cdietrich . It's later than I had hoped for, but I'm trying to run the tests right now.
I've checked out this repository and tried running ./gradlew build
, per instructions of the README. However, I'm getting 10 out of 4355 tests that fail, preventing the build to finish. I'll try out the patch anyway and see if that figure changes, but I thought it might be good to report this as well. I'm on Windows if that's relevant.
> Task :org.eclipse.xtext.tests:test
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.eclipse.xtext.util.MergeableManifest (file:/C:/Users/Werk/Desktop/related%20projects/xtext-core/org.eclipse.xtext.util/build/libs/org.eclipse.xtext.util-2.30.0-SNAPSHOT.jar) to field java.util.jar.Manifest.attr
WARNING: Please consider reporting this to the maintainers of org.eclipse.xtext.util.MergeableManifest
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[0: Maven|Maven/Gradle|HIERARCHICAL|None] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[1: Maven|Plain|HIERARCHICAL|None] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[2: Maven|Plain|HIERARCHICAL|None] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[6: Maven|Plain|HIERARCHICAL|None] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[7: Maven|Plain|HIERARCHICAL|None] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[8: Gradle|Plain|HIERARCHICAL|None] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[9: Maven|Plain|HIERARCHICAL|Fat Jar] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[10: Maven|Plain|HIERARCHICAL|Fat Jar] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[12: Maven|Plain|HIERARCHICAL|Regular] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest > testProjectCreation[13: Maven|Plain|HIERARCHICAL|Regular] FAILED
org.junit.ComparisonFailure at CliWizardIntegrationTest.java:386
4355 tests completed, 10 failed, 68 skipped
> Task :org.eclipse.xtext.tests:test FAILED
FAILURE: Build failed with an exception.
can you show an example fail from the html report? do you have e.g launch files touched locally?
do you have e.g launch files touched locally?
I haven't touched anything, although I did first open it in Eclipse before running the command. I'll do a hard reset and run the build again, see if that changes anything.
Here is a result of a failing test:
testProjectCreation[0: Maven|Maven/Gradle|HIERARCHICAL|None]
org.junit.ComparisonFailure: Missing file: \plainMaven.parent\.project expected:<[<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>plainMaven.parent</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.eclipse.m2e.core.maven2Builder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>org.eclipse.m2e.core.maven2Nature</nature>
</natures>
</projectDescription>
]> but was:<[]>
at org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest.lambda$compareFileTrees$19(CliWizardIntegrationTest.java:386)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest.compareFileTrees(CliWizardIntegrationTest.java:385)
at org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest.validateCreatedProjects(CliWizardIntegrationTest.java:335)
at org.eclipse.xtext.xtext.wizard.cli.CliWizardIntegrationTest.testProjectCreation(CliWizardIntegrationTest.java:325)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
at jdk.internal.reflect.GeneratedMethodAccessor115.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
i have no clue
Sorry, doing a clean build worked. I'm gonna try out the patch now.
Okay, an update on the tests:
Original code:
public IHiddenRegionFormatting merge(List<? extends IHiddenRegionFormatting> conflicting) {
// If there are only 2 conflicts,
// usages of this method expect the second value to be updated to the merge result
// TODO: Fix those usages so they no longer expect this method to edit its input
if (conflicting.size() == 2) {
conflicting.get(1).mergeValuesFrom(conflicting.get(0));
return conflicting.get(1);
}
IHiddenRegionFormatting result = formatter.createHiddenRegionFormatting();
// Reversed so the merging order is consistent with the special case above
for (IHiddenRegionFormatting conflict : Lists.reverse(conflicting))
result.mergeValuesFrom(conflict);
return result;
}
=> tests in org.eclipse.xtext.tests
run fine (just checking the baseline :))
Patch 1: remove the special case for two conflicts.
public IHiddenRegionFormatting merge(List<? extends IHiddenRegionFormatting> conflicting) {
IHiddenRegionFormatting result = formatter.createHiddenRegionFormatting();
// Reversed so the merging order is consistent with the special case above
for (IHiddenRegionFormatting conflict : Lists.reverse(conflicting))
result.mergeValuesFrom(conflict);
return result;
}
=> one failing test in org.eclipse.xtext.tests
: FormattableDocumentTest::autoWrapRewrite
.
Failure: ConflictingFormattingException: Conflicting values for 'space': '@' and '!'.
Patch 2: remove Lists.reverse
.
public IHiddenRegionFormatting merge(List<? extends IHiddenRegionFormatting> conflicting) {
IHiddenRegionFormatting result = formatter.createHiddenRegionFormatting();
for (IHiddenRegionFormatting conflict : conflicting)
result.mergeValuesFrom(conflict);
return result;
}
=> all tests in org.eclipse.xtext.tests
pass.
I think this patch should be good to go. :)
I'd like to see the tests for Xbase and Xtend succeed, too.
I would be willing to try them as well. I am able to setup xtend
, but for xbase
the build is failing.
Doing a clean ./gradlew build
on xbase
results in
> Task :org.eclipse.xtext.extras.tests:test
org.eclipse.xtext.util.MapVsMultimap > initializationError FAILED
java.lang.SecurityException at ClassLoader.java:1150
org.eclipse.xtext.xtext.ecoreInference.Xtext2EcoreTransformerTest > initializationError FAILED
java.lang.SecurityException at ClassLoader.java:1150
69 tests completed, 2 failed
> Task :org.eclipse.xtext.extras.tests:test FAILED
FAILURE: Build failed with an exception.
I'll try out the patch for xtend
.
For xtend
there are 10 failures having to do with indentation being broken...
I guess this will require more debugging to see where it is going wrong - something for which I unfortunately don't have the time.
Are these tests locally green for you when you run them against the baseline?
@szarnekow they are.
please also call builds with clean build -PuseJenkinsSnapshots=true
please also call builds with clean build -PuseJenkinsSnapshots=true
That works
For xbase
there are 22 failures, also having to do with faulty indentation.
I stumbled upon an issue where branches of
IFormattableDocument::formatConditionally
actually modify the text replacers of the parent document, leading to leftover formatting in the other branches.I managed to track down the code responsible for modifying the
IHiddenRegionFormatter
s to this method in the classHiddenRegionFormattingMerger
:See the scary
TODO
comment.Removing those first lines solves my issue. As a workaround I'm now using a custom
IMerger<IHiddenRegionFormatting>
.