eclipse-ocl / org.eclipse.ocl

Eclipse Public License 2.0
0 stars 0 forks source link

[parser] Modification to support IMP and exploit LPG 2 #307

Closed eclipse-ocl-bot closed 1 month ago

eclipse-ocl-bot commented 1 month ago

| --- | --- | | Bugzilla Link | 242153 | | Status | CLOSED FIXED | | Importance | P3 enhancement | | Reported | Jul 26, 2008 14:31 EDT | | Modified | May 27, 2011 02:55 EDT | | Version | 1.2.0 | | Depends on | 288666, 298542, 298562 | | Blocks | 297966, 298634 | | Reporter | Ed Willink |

Description

The Eclipse IMP project provides useful automation for editors,\ especially those using LPG. This could be very useful for OCL-based\ languiages such as QVT.

I therefore started modifying org.eclipse.ocl to use LPG 2.x rather\ than LPG 1.x so that IMP could be more naturally exploited.

Eventually I backed out nearly all my changes since they are not\ necessary and create more problems than they solve. A summary of\ these activities is provided below\ -----------------------------------------------------------------\ IMP provides a builder so *.g files are automatically built.\ Seems like a good idea, but only when there are no errors. If\ there are problems, the OCLParsersym.java is set to bad content\ and it's difficult to know which grammar is bad. I disabled the\ builder quite rapidly.

IMP provides an LPG editor, which works well for syntax colouring\ and outlines but has some strange ideas about where the errors are.

IMP (the Ganymede synchronised release) has a bad bug whereby any\ edit of a file with an error hogs CPU. A bit tedious when LPG\ files have spurious errors.

LPG 2 uses lpg.runtime rather than lpg.lpgjavaruntime so any\ change introduces a binary incompatibility. lpg.runtime is not\ in Orbit.

IMP exploits LPG 2 parsers and lexers through very small ILexer\ and IParser interfaces. It would be good for these to be directly\ supported by OCLParser and OCLLexer, however IMP is an incubation\ project, so introducing an MDT-OCL dependency on IMP is undesirable.

LPG 2 deprecates $ so e.g. $empty needs changing to %Empty.\ Unfortunately preventing '$' being converted to '' defeated\ me in the KWLexerMap, so I had to change the escape to # rather\ than $. This mostly worked, but for unexplained reasons the NewCase\ macro for EssentialOCL.gi got a BeginJava/EndJava wrtapped around it.\ Changing all BeginJava/EndJava to BeginCode/EndCode cured the problem.\ Similar macros not in an include did not have this problem.

LPG 2 seems to have an ordering problem with include paths. I had to change\ the %Import section to provide an explicit path. This may be because\ LPG 2 has the IMP paths built-in.

LPG2 appears to introduce bison-style production type suffixes\ such as unary$$OCLExpressionCS ::= ...\ This didn't work for me with either $$ or ##.\ ------------------------------------------------------------------\ IMP also supports non-LPG grammars, so I investigated use of the\ existing LPG 1 grammars as non-LPG IMP grammars. Very little problem,\ just need to clone IMP's SimpleLPGParseController.

Consequently, I recommend that MDT OCL delay migrating to LPG 2\ until LPG 2 and IMP are more mature.

The only significant changes I found worth making to OCL were:\ -------------\ Change CSTNode.startOffset:EInt to CSTNode.startToken:IToken\ Change CSTNode.endOffset:EInt to CSTNode.endToken:IToken\ Add CSTNode.getStartOffset() for compatibility\ Add CSTNode.getEndOffset() for compatibility

Modify all setOffsets() to use setXxxToken rather than setXxxOffset.

Only the dotArrowExpCS ::= NUMERIC_OPERATION '(' argumentsCSopt ')'\ causes trouble through use of setXxxOffset in a non-IToken fashion.\ Perhaps this partitioned IToken usage deserves revisiting anyway.\ -------------\ Add to AbstractLexer \ public abstract int [] getKeywordKinds();

so that IMP can access the list of tokens for colouring.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 03, 2009 06:55

(In reply to comment #50)

Preliminary comments:

Change the {lpg.exe} in the launches to {lpg2.exe} so that the old LPG cannot be used accidentally.

Ok.\

The test_hebrew_singleQuote_135321 fails in both Ecore and UML bindings.

That test has been "arbitrarily" failing to me, in some occasions since years ago. I guessed that comes from any character encoding reason which I don't know. I think that it's time to discover the reason ;)

EssentialOcl.gi has an "Invalid" token which EssentialOcl.g does not.

Ok, I'll take a look to this. I probably skipped something while merging.

Lexer.lexer() replaces lexToTokens(). lexToToken was not a verty good name, but lexer is differently bad; not a verb, and now requires caller to getIPrsStream. Suggest chanke to lex(AbstractParser) using a verb and hiding the indirection.

Similarly change parseTokensToCST to parse() rather than parser().

Well, that names comes from the original template. It can obviously be changed ;)

AbstractOCLParser has a @SuppressWarnings("nls"). I think this should only be used in auto-generated or test modules. Real code should have per-string NLS apologies, since at one point OCL was internationalised and might well be again.

Ok, that should be there since time ago. If it's not solved when this is applied, I'll silently add the change.

The revised architecture seems to solve a lot of the uncomfortable compatibilities I had to struggle with originally, but I think it deserves more careful thought to get it right in one go.

There really are two changes here. I do not like reviewing them together. The LPG change involves numerous trivial changes that are tedious and not that necessary to review. The architecture change involves just a few changes that are masked by the volume of trivial changes. Please prepare a first set of patches that is a no-functionality change migration to LPG2. We can commit these (perhaps to a branch to assist creation of a two stage patch) and then review the architecture revision patch.

Although I dislike the idea, I have no option since this is matter of two. I only hope that the patch doesn't die in the branch.

About including LPG into Orbit:

IMP's newsgroup message about including LPG v2.0.17 has not been replied. Collaboration with IMP's team is not being as I have liked. I'll try a final private email to Robert

I have discovered that an IPZilla CQ about LPG v2.0.16 has been approved, the bundle is not in Orbit though. Since LPG v2.0.16 is properly tagged in sourceforge version, I could try the patch on this lpg runtime to see if it properly works. However I would prefer including the last version.

I'll ask robert if he has any intention of including the last LPG version in Eclipse. If don't receive any response we could:

  1. Go straightforward on including LPG v2.0.16 bundle in Orbit.
  2. Including LPG v2.0.17, which would previously involve solving legal IP revision on LPG v2.0.17.

Suggestions are welcomed. Meanwhile I'll prepare the one step patch and try to see if it also works on LPG v2.0.16.

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Sep 03, 2009 15:35

(In reply to comment #51)

The test_hebrew_singleQuote_135321 fails in both Ecore and UML bindings.

That test has been "arbitrarily" failing to me, in some occasions since years ago. I guessed that comes from any character encoding reason which I don't know. I think that it's time to discover the reason ;)

Odd. This worked fione for me on the previous patch. I have never seen this test fail. (Spannish) character encoding may well provide an explanation.\

EssentialOcl.gi has an "Invalid" token which EssentialOcl.g does not.

Ok, I'll take a look to this. I probably skipped something while merging.

Merging in such a large list of trivial changes is very hard. Much easiuer to re-edit.\

Lexer.lexer() replaces lexToTokens(). lexToToken was not a verty good name, but lexer is differently bad; not a verb, and now requires caller to getIPrsStream. Suggest chanke to lex(AbstractParser) using a verb and hiding the indirection.

Similarly change parseTokensToCST to parse() rather than parser().

Well, that names comes from the original template. It can obviously be changed ;)

Yes. I wasn't expecting to change this API, but your restructuring is good and establishes a much smaller more relevant interface. If we're breaking this almost internal API, I thin k we might try to get it right.

Although I dislike the idea, I have no option since this is matter of two. I only hope that the patch doesn't die in the branch.

I hope not too. I want this resolved. The LPG delay pretty much forces the branch now. I was only suggesting the branch as an easy way of creating a part1 and part2 patch.

Please commit a re-edit of the original 'part1' patch to a branch so that we can concentrate on the 'part2' API improvement.

When LPG is available, it should be possible to re-edit part1. part2 might survive almost unaffected by intervening progress.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Sep 03, 2009 15:56

The test_hebrew_singleQuote_135321 fails in both Ecore and UML bindings.

That test has been "arbitrarily" failing to me, in some occasions since years ago. I guessed that comes from any character encoding reason which I don't know. I think that it's time to discover the reason ;)

Odd. This worked fione for me on the previous patch. I have never seen this test fail. (Spannish) character encoding may well provide an explanation.

The problem is the Acute definition between OCLLexer.g and OCLLexer.gi. You need to configure your Eclipse environment to avoid corrupting files. This might also explain why your fully decorated name caused so many problems.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 03, 2009 16:22

The problem is the Acute definition between OCLLexer.g and OCLLexer.gi.

I caught it this afternoon, while creating the simple patch.

You need to configure your Eclipse environment to avoid corrupting files. This might also explain why your fully decorated name caused so many problems.

Yes, I tried to avoid 'acuting' my surname. I can't figure out which is the problem. I always configure my workscape character encoding (Preferences -> General -> workspace )as UTF-8, any additional idea ?.

Yes. I wasn't expecting to change this API, but your restructuring is good and establishes a much smaller more relevant interface. If we're breaking this almost internal API, I thin k we might try to get it right.

Agree, I didn't want to change too much the LPG template, so that I show that essence of the new design comes from the LPG runtime API and its templates. You may find thatm for instance, the $getIToken, $getToken, etc macros will not generate the definitive code (Actually the proper as the new macro states, a method should be used instead). I think we will have a lot of oportunities to improve the grammars and parsing infraestructure with these new changes.

I hope not too. I want this resolved. The LPG delay pretty much forces the branch now. I was only suggesting the branch as an easy way of creating a part1 and part2 patch.

Please commit a re-edit of the original 'part1' patch to a branch so that we can concentrate on the 'part2' API improvement.

When LPG is available, it should be possible to re-edit part1. part2 might survive almost unaffected by intervening progress.

I was working on it this afternoon, when I encountered the problem with SimpleTypeEnum. It would also be good to incorporate finished patches, such us static feature (already revised) or that relating pre, pathNameCS and such (I'll revise it tomorrow in the morning).

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Sep 03, 2009 17:21

Yes, I tried to avoid 'acuting' my surname. I can't figure out which is the problem. I always configure my workscape character encoding (Preferences -> General -> workspace )as UTF-8, any additional idea ?.

Yes. Don't change the default presumably Unicode configuration.

UTF-8 requires that every charcter be encoded in 8 bits, so anything that isn't 7 bits gets a multi-byte recoding.

Java is Unicode.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 04, 2009 12:51

Created attachment 146527 (attachment deleted)\ OCL (Experimental) om LPG v2

This patch built on experimental branch, contains the related first step changes.

Once this patch is committed, I'll try to create the second one...

Comments are welcomed

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Sep 05, 2009 02:47

The new patch is very incomplete

trivially; no MANIFEST.mf, lpg2.exe launches, .api_filters

more seriously: no AbstractAnalyzer and everything else with a changed import,\ no ...lpg/*.gi templates.

[There is no need to pursue QVTr on this branch. I am happy that the LPG 2 change has an easy impact, that I can follow through in due course.]

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 05, 2009 15:07

(In reply to comment #57)

The new patch is very incomplete

trivially; no MANIFEST.mf, lpg2.exe launches, .api_filters

more seriously: no AbstractAnalyzer and everything else with a changed import, no ...lpg/*.gi templates.

Sorry again, I don't really know what I did when I createf the patch Yesterday>.<. Apologies.

Uploading again.\

[There is no need to pursue QVTr on this branch. I am happy that the LPG 2 change has an easy impact, that I can follow through in due course.]

Ok. I didn't worked on the QVTr changes needed to follow this OCL Branch changes.

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 05, 2009 15:11

Created attachment 146557 (attachment deleted)\ OCL (Experimental) on LPG v2

New try with patch.

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Sep 05, 2009 16:07

+1

minor observations:

Probably want an API filter rather than @since 3.0 on AbstractParser.parseTokensToCST, setOffsets etc etc\ The change of IToken package doen't make a method new.

reportError definitely needs sorting out. part2 will no doubt give a better structure.

Indentation for xxxsym.orderedTerminalSymbols is irregular.


Have you accounted for the delay in IP approval for LPG 2.0.18?

It may be possible to use parallel IP to put LPG 2.0.18 in CVS promptly, but I doubt that it can be exported in a build till IP approval is complete.

eclipse-ocl-bot commented 1 month ago

By Alexander Igdalov on Sep 07, 2009 08:23

(In reply to comment #59)

Created an attachment (id=146557) [details] OCL (Experimental) on LPG v2

New try with patch.

Cheers, Adolfo.

Sorry for slow response. My +1.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 09, 2009 14:41

(In reply to comment #60)

+1

minor observations:

Probably want an API filter rather than @since 3.0 on AbstractParser.parseTokensToCST, setOffsets etc etc The change of IToken package doen't make a method new.

I said my opinion about this. From the API point of view it's a new method (You could keep both methods if you preserved the old package). However, I'm not starting a discussion about this: having an extra API filter doesn't make me feel unhappy. So, I'll commit your suggestion.

reportError definitely needs sorting out. part2 will no doubt give a better structure.

Indentation for xxxsym.orderedTerminalSymbols is irregular.

I don't understand. They are generated files :?

Let me know what do you expect to incorporate it before comitting.


Have you accounted for the delay in IP approval for LPG 2.0.18?

It may be possible to use parallel IP to put LPG 2.0.18 in CVS promptly, but I doubt that it can be exported in a build till IP approval is complete.

Robert will start IP approval as soon as the release comes up. Since previous versions have already been approved, it shouldn't take too much time. Then I'll start to elaborate the Orbit bundle which we will be able to use in our project.

Meanwhile this patch, and further patches will be created using LPG v2.0.17. When 2.0.18 I'll revise that all it's ok with this version. When it is included in Orbit we will be able to do the merge.

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Sep 10, 2009 02:18

Probably want an API filter rather than @since 3.0 on AbstractParser.parseTokensToCST, setOffsets etc etc The change of IToken package doen't make a method new.

I said my opinion about this. From the API point of view it's a new method (You could keep both methods if you preserved the old package). However, I'm not starting a discussion about this: having an extra API filter doesn't make me feel unhappy. So, I'll commit your suggestion.

This is certainly debatable. Does a changed import change a use of the import?\ Maybe some API policy may help us out. Maybe the entire class is @since. I think I lean towards semantic compatibility; AbstractParser is an @since class change because the inherited PrsStream is very different; ProblemHandler is an API filter change because the IToken usage is very similar.

In auto-generated code there is a more practical consideration that it may be difficult to auto-generate @since's so an API filter is much more convenient. The changed manifest and imports make the change very clear, per-method @since's are clutter and may obscure other history.

Indentation for xxxsym.orderedTerminalSymbols is irregular.

I don't understand. They are generated files :?

Maybe there is a template tabbing error. Maybe it's a very minor LPG coding error. Not worth a great deal of effort.\

Let me know what do you expect to incorporate it before comitting.

All the above where minor observations not inhibiting the +1. Go for it. If the observations can be conveniently accommodated, great, if not, who really cares?

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Sep 12, 2009 14:18

Patch comitted to BRANCH (OCL_2_1_on_LPGv2_Experimental), with the following additions:

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Dec 22, 2009 12:44

Created attachment 154939 (attachment deleted)\ OCL (HEAD) on LPG v2.0.17

Comments in the next thread

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Dec 22, 2009 12:46

The previous attachment contains the patch to make MDT-OCL be aligned to LPGv2.0.17. The patch is created against the current HEAD. Apart from the aforementioned details, there some new comments to do:

When committing the patch, I'll remove the old .g grammars.

P.S: LPgv2.0.17 will be soon in the Orbit repository. Meanwhile you can get the LPG runtime from the sourceforge CVS.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 22, 2009 16:41

Created attachment 154947 Adapt to Bug 298201 commit

Attached updates CSTPackage.java and .api_filters updated by Bug 298201.

Attached also adds the patch for ocl.psf.

:notepad_spiral: Bug242153.patch

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Dec 24, 2009 05:36

Ed,

Thanks for reconciling the API filters file. I think that all is (finally :) ) ready for comittment.

I guess there isn't any inconvenience with the patch, is it ?

I prefer to commit and close this long bug, and open a new one to deal with the refactoring to exploit LPGv2 runtime.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 26, 2009 08:54

Created attachment 155049 Better launch configs

Attached launches exploit the naming conventions used by the official 'lpg' launch to make the two OCL launches work directly from the imported LPG projects on arbitrary platforms. Spurious builds and refreshes are also fixed.

A further generic 'LPG2 selected-file' works on either OCL .g file or QVTc or QVTr .g files.

:notepad_spiral: Bug242153-launches.patch

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Dec 27, 2009 10:38

As discussed in Bug 297966, there are some test cases related to backtracking parser which are currently failing. It seems that error codes associated to parsing error is different, This provokes a change in the error message expected by the parser.

On the one hand, it seems that a suspicious warning in the backtracking parser may be the cause of the change. On the other hand, QVTo and QVTd grammars which extend OCL grammars are not experimenting any problem about this, so it may be considered as a simple change about the error generated by the parser.

Since I'm having holidays this week, I'll commit the patch so nobody get blocked by this and all downstream project may react this change during this week. When I'm back the 4th in January, I'll face on this again so that I can either detect why the suspicious warning is occurring and maybe the test are recovered again or change the test cases to align to the new error message.

We have a month before M5 to take a decision .

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Dec 27, 2009 10:52

Patch finally committed to HEAD (including removal of the old .g grammars).

Ed, could you commit your contribution of the launching configurations?

Alex, coud you update any building related files to make the builds use the new LPGv2.0.17 runtime ? (I would appreciate a patch to see which files are involved).

When this is done I'll resolve the bug (Ed/Alex could also do it in my absence).

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 27, 2009 16:11

(In reply to comment #71)

Ed, could you commit your contribution of the launching configurations?

Done

Alex, coud you update any building related files to make the builds use the new LPGv2.0.17 runtime ? (I would appreciate a patch to see which files are involved).

This is difficult without an Orbit drop to build against. Any idea when an Orbit Stable build will be published?

Otherwise we have to resurrect the old technology to redistribute LPG with OCL.\

When this is done I'll resolve the bug (Ed/Alex could also do it in my absence).

Is there a test case demonstrating the problem?

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 27, 2009 17:42

N-Build now passes, but only by using explicit HEAD tags in the map since HEAD cannot be used for LPG 2. Hopefully an Orbit S build will appear soon.

eclipse-ocl-bot commented 1 month ago

By Sergey Boyko on Dec 29, 2009 16:15

(In reply to comment #72)

When this is done I'll resolve the bug (Ed/Alex could also do it in my absence).

Is there a test case demonstrating the problem?

Launch org.eclipse.ocl.ecore.tests/launches/org.eclipse.ocl.ecore.tests (Standalone Backtracking).launch\ All tests in "Parser Backtracking Tests" fail.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 30, 2009 02:47

I'm not worried by the backtracking parser failures, which are incidentally almost the only tests that test message severity and content rather than just existence.

The change of message is a nuisance but indicates that perhaps the parser is better, so I've raised Bug 298630 to review the error grammar.

Testing message content is awkard so I've raised Bug 298629, which can be tackled at the same time as Bug 296991. Someone please +1 the principle of 298629.

I'll submit an initial patch on Bug 298630 to make the tests pass, hopefully as both Ecore and UML tests.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 30, 2009 03:12

(In reply to comment #75)

I'm not worried by the backtracking parser failures, which are incidentally almost the only tests that test message severity and content rather than just existence.

But when I change the tests to match the new message, the tests then fail with a null CST. This is a serious problem. The backtracking parser is not backtracking to produce a best parse. The wanted error message arise when an ERROR_TOKEN is parsed where an SimpleNameCS parse failed allowing the parse to recover. This is not happening. Perhaps it's just some trivial initialisation that needs to change to activate the recovery.

As it stands, the backtracking parser is fine if there are no errors, but is unable to diagnose multiple errors in a parsed document. This renders the backtracking support useless.

Adolfo; I think this is your problem unless anyone else has time.

eclipse-ocl-bot commented 1 month ago

By Ed Merks on Dec 30, 2009 16:34

Guys,

Did you file a CQ for this different Orbit dependency?

Note that you've hard coded a specific version number rather than a range; that seems like a bad idea.

Note too that you reexport this dependency, just like the previous dependency, and that's resulting in compile errors in org.eclipse.m2m.qvt.oml and org.eclipse.m2m.qvt.oml.cst.parser. Are you coordinating this type of change with the affected projects?

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Dec 30, 2009 20:16

Ed(willink),

I would have been surprised, if that suspicious LPG related warning only produced an error messaege change. Your comments have demonstrated the real panic of the warning. As you may know, i'm away in holidays, so i'll face on this when i'm back next monday. If we (i have required some support to LPG team), can't fix this the Following weeks, I'm afraid we will have to stay in LPG v1.1

Ed(merks),

Sorry for not providing links (i'm using a not userfriendly mobile phone), but CQ has been granted, even patch for QVTo has been uploaded. I guess u may find info in one of the dependent bugs.

Sergey,\ dont commit qvto patch until this is fixed.we have time before m5. Hopefully we can complete the migration in time :)

have a happy new year,\ Adolfo

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Dec 31, 2009 01:03

(In reply to comment #78)

Ed(merks),

Sorry for not providing links (i'm using a not userfriendly mobile phone), but CQ has been granted

https://dev.eclipse.org/ipzilla/show_bug.cgi?id=3656

even patch for QVTo has been uploaded.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=297966\

Sergey, dont commit qvto patch until this is fixed.we have time before m5. Hopefully we can complete the migration in time :)

If OCL is requesting QVTo to hold back, OCL should roll back. CVS HEAD is broken for anyone using both OCL and QVTo (or QVTd); ie. GMF ...

Ed(willink),

I would have been surprised, if that suspicious LPG related warning only produced an error messaege change. Your comments have demonstrated the real panic of the warning. As you may know, i'm away in holidays, so i'll face on this when i'm back next monday. If we (i have required some support to LPG team), can't fix this the Following weeks, I'm afraid we will have to stay in LPG v1.1

If this is not a quick fix, we must roll back. I suspect it is a simple fix which is why I have not already rolled back, but if other projects are affected we cannot allow this inconsistency and uncertainty to persist for another four and probably more days.

Any other opinions?

eclipse-ocl-bot commented 1 month ago

By Ed Merks on Dec 31, 2009 06:30

Thanks for confirming a CQ was filed. :-)

Given most folks are not working the next few days, resolving the problem on Monday is probably okay. In general though, committing changes that break the stack is of course a practice to be avoided. Someone is bound to notice. :-P

The PDE seems to have developed a nasty habit of inserting specific hard coded version numbers by default; please be sure to either remove that number or change it to be a range.

eclipse-ocl-bot commented 1 month ago

By Sergey Boyko on Dec 31, 2009 13:44

(In reply to comment #78)\

Sergey, dont commit qvto patch until this is fixed.we have time before m5. Hopefully we can complete the migration in time :)

Ok, QVTo is waiting till that CR be resolved (either rolled back or fixed for backtracking).

eclipse-ocl-bot commented 1 month ago

By Sergey Boyko on Jan 04, 2010 03:53

Some exploring on backtracking parser

As for QVTo parser both solutions ('-1' as value of 'error_repair_count' argument and 'btParser.fuzzyParse()' in place of 'btParser.parse()') solve the problems with tests on error recovery behavior.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Jan 04, 2010 05:43

Thanks Sergey. The error productions are certainly written so that the next ; } or ) resynchronises.

I think we could propagate the LPG 2 repairCount API.

If repairCount == 0 use DeterministicParser.parse (current behaviour)\ If repairCount < 0 use BacktrackingParser.parse (unexpected current behaviour)\ If repairCount > 0 use BacktrackingParser.fuzzyParse (changed behaviour)

or do we need to give users more control?

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 08:06

Created attachment 155227 (attachment deleted)\ Fixing BacktrackingParser patch

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 08:10

Sergey,

Thank you very much for the insight. You have saved me some debugging tasks. The previous attatchment fixes MDT-OCL counterpart. It also fixes the lpg.runtime dependency range.

Ed,\ migration's second phase , re-introduced the use of templates (dtParseTemplateF.gi). I'll work into this to also reintroduce the btParseTemplateF.gi, so that all the customized macros to make both parser coexist are removed from the grammar files.

After this, using a backtracking or deterministic parser should only imply using a different template in the options section.

In principle, I have coded in the OCL facade to do the following:\ errorRepair == 0 => deterministic (.parse)\ errorRepair != 0 => backtracking (.fuzzyParse)

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Alexander Igdalov on Jan 04, 2010 08:21

(In reply to comment #84)

Created an attachment (id=155227) [details] Fixing BacktrackingParser patch

Looks good. My +1

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Jan 04, 2010 08:51

Clearly the LPG 2 parse() API has changed for the backtracking parser.

Our existing parse(10) recovery is only partially emulated by parse(-1); it recovers on one token but doesn't limit to 10 recoveries.

I cannot see the documentation on fuzzyParse() so cannot tell whether fuzzyParse() replaces parse() or not. Otherwise we perhaps need the three way test from comment 83.

Either way, +1 for now since this is a big improvement. However our previous API was N>0 is N backtracking recoveries, with -ve an error. It would be nice not to change this if we are not actually changing anything, or is -1 for unlimited a new facility?

Any comments on the direction for Bug 298634?

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 10:30

Ed, Alex

The attatched patch is wrong. Actually the tests only pass when error_count_repair = -1 and dtParser.parse(error_repair_count) is called. Test cases fail when dtParser.fuzzyParse(error_repair_count) is used regardless the value of the error_repair_count. So even doing what you say in comment #83 it seems that the backtracking parser won't recover if repair_count > 0.

A tricky workaround to make this more consistent (make the backtracking parser always work) may be stablishing -1 as error_repair_count regardless the provided error_repair_count. I hope to find better solution during the afternoon >.<.

Regards,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Alexander Igdalov on Jan 04, 2010 10:42

(In reply to comment #88)

Ed, Alex

The attatched patch is wrong. Actually the tests only pass when error_count_repair = -1 and dtParser.parse(error_repair_count) is called. Test cases fail when dtParser.fuzzyParse(error_repair_count) is used regardless the value of the error_repair_count. So even doing what you say in comment #83 it seems that the backtracking parser won't recover if repair_count > 0.

Sad news.

A tricky workaround to make this more consistent (make the backtracking parser always work) may be stablishing -1 as error_repair_count regardless the provided error_repair_count. I hope to find better solution during the afternoon >.<.

It would be nice to have a better workaround - the QVTO code completion parser sets error_repair_count to a relatively small number since in case of numerous errors and infinite error_repair_count the parser's response may be very slow.

Regards, Adolfo.

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Jan 04, 2010 11:07

If necessary we can presumably count the number of error messages we have generated and throw a sufficiently brutal exception to terminate the parse.

For QVTd I use a derived ProblemHandler with error and warning count limits set by default at 100.

eclipse-ocl-bot commented 1 month ago

By Sergey Boyko on Jan 04, 2010 12:35

(In reply to comment #88)\ Hi Adolfo,

Analysis of BacktrackingParser.parse(..) method reveals the following:

public Object parse(int max_error_count) throws BadParseException\ {\ return parseEntry(0, max_error_count);\ }\ public Object parseEntry(int marker_kind, int max_error_count) throws BadParseException\ { ...

if (max_error_count > 0 && tokStream instanceof IPrsStream)\
    max_error_count = 0;

...\

}

And AbstractOCLParser is an instance of IPrsStream. So calling parse(..) with any 'max_error_count > 0' results in 'max_error_count = 0' which means that "no Error token recoveries occur".

I believe that this is guard on using parse(..) method with limits on the number of Error token recoveries. And is indirect reference that fuzzyParse(..) should be used for such purpose.

So I'm agree with comment 83.

The attatched patch is wrong. Actually the tests only pass when error_count_repair = -1 and dtParser.parse(error_repair_count) is called. Test cases fail when dtParser.fuzzyParse(error_repair_count) is used regardless the value of the error_repair_count. So even doing what you say in comment #83 it seems that the backtracking parser won't recover if repair_count > 0.

A tricky workaround to make this more consistent (make the backtracking parser always work) may be stablishing -1 as error_repair_count regardless the provided error_repair_count. I hope to find better solution during the afternoon >.<.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 12:59

Hi All,

I have got some more conclusions:

The point here, is why the fuzzyParser is failing when trying to recover. Probably the supression of the suspicious warning may help. I hope it's not a LPG Runtime error itself, somebody should previously have noticed it.I'll add phillippe to the bugzilla to see if he can give us a hand. I

I think that we could temporally make max error_repair_count be -1 (the painless solution) so unlimited count is used when using the backtracking version. I prefer having a "slow" backtracking parser than a non-backtracking one. This should make downstream projects to adopt the change while we solve the specific issue (probably in a separate bugzilla).

What do you think ?.

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 13:10

Sergey,

Mid-air collision. Some comments:

Cheers,\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 13:29

The point here, is why the fuzzyParser is failing when trying to recover. Probably the supression of the suspicious warning may help.

Sorry, I forgot to paste the warning:

OCLParserErrors.gi:35:60:35:70:971:981: Warning: The terminal symbol "ERROR_TOKEN" was not imported from OCLBacktrackingLexer.gi

eclipse-ocl-bot commented 1 month ago

By Sergey Boyko on Jan 04, 2010 14:00

(In reply to comment #93)\ Hi Adolfo,

  • I believe that parse is not related with fuzzyParse. They are independent implementations.

Yes, I think the same. And fuzzyParse() is the new implementation for the backtracking with limited number of Error token recoveries.

  • Are you sure QVTo counterpart works with fuzzyParse method ?. I had to use parse method + error_repair_count = -1 to make the QVTo tests work.

Yes, I'm sure :) \ For the QVTo fuzzyPars() method diagnoses/recovers any number of errors. From the unit tests point of view it works just the same as parse(-1).

About BadParseException. Looks like it throws when top level rule for the error input is not correctly defined. I mean that it's grammar problem not the parser.

For the QVTo quite complicated script with arbitrary number of errors is parsed by the fuzzyParse() method without BadParseException.

Regards,\ Sergey

eclipse-ocl-bot commented 1 month ago

By Sergey Boyko on Jan 04, 2010 14:12

(In reply to comment #95)\ Hi Adolfo,

  • Are you sure QVTo counterpart works with fuzzyParse method ?. I had to use parse method + error_repair_count = -1 to make the QVTo tests work.

Yes, I'm sure :)
For the QVTo fuzzyPars() method diagnoses/recovers any number of errors. From the unit tests point of view it works just the same as parse(-1).

Looks like I understand why you're not successful with QVTo unit tests. We have our own template for the parser that is located in miscellaneous.gi (suppresses template from EssentialOCL.gi).

Btw, I've done some modifications to the grammar comparing to the patch you submitted. I'll attach new patch to the bug 297966 tomorrow.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 14:27

Looks like I understand why you're not successful with QVTo unit tests. We have our own template for the parser that is located in miscellaneous.gi (suppresses template from EssentialOCL.gi).

Yes, but I simple changed the call from in the QVToParser to invoke fuzzyParse and test failed, they didn't using parse one (with the -1 as error_repair_count). I should have done something wrong...\

Btw, I've done some modifications to the grammar comparing to the patch you submitted. I'll attach new patch to the bug 297966 tomorrow.

You mean QVTo patch ?. I'm just creating OCL counterpart to suport Comment 85 (filing in a little bit).\ BTW, how are we dealing with the contribution?. I'm not sure if it needs a new CQ for it >.<. In any case I guess we can discuss that in the other bug.

Cheers.\ Adolfo.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 14:55

Created attachment 155269 Backtracking fixing patch

The attachment provides problem's fix (including modified generated files), using the policy suggested by Comment 85.

It contains some nasty macro's use, which may be cured when reinserting the LPG templates.

Cheers, Adolfo.

:notepad_spiral: ocl242153_OCLBacktrackingFix_try2.patch

eclipse-ocl-bot commented 1 month ago

By Ed Willink on Jan 04, 2010 15:15

+1 for the interim fix. This should allow Bug 298634 to go in too.

eclipse-ocl-bot commented 1 month ago

By Adolfo Sanchez-Barbudo Herrera on Jan 04, 2010 18:23

Patch committed to HEAD.

I guess that QVTo may go ahead with its patch.

Cheers,\ Adolfo.