Take a look at OCL and QVTd pack200 issues

[eclipse-ocl-bot]

| --- | --- | | Bugzilla Link | 463131 | | Status | RESOLVED FIXED | | Importance | P3 normal | | Reported | Mar 25, 2015 15:51 EDT | | Modified | Mar 25, 2021 16:55 EDT | | See also | 572316 | | Reporter | David Williams |

Description

Based on Mars M6 experience, and \ this message to cross-project list:

https://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg11691.html

I am curious why there would be so many problems with "packed jars". 10% is way more than I have ever seen before.

With so many ... I'm wondering if there is a pattern?

So, I'm opening this bug, just to "look at more closely with you" if you are willing? If not, fine, close as "won't fix".

And, I know neither of us has time for a deep diagnostic excursion, but, maybe 30 or 60 minutes?

An easy starting point would be to point me to jars that are "plain", neither packed, signed, and not even "pre-conditioned" for packing?

And, or a pointer to your build site?

And, beyond that, there must be an easy way to tell buckminster to "not pack" individual bundles. Does it respect the eclipse.inf file and it's directives?

[eclipse-ocl-bot]

By Ed Willink on Mar 26, 2015 03:14

When you look at the dates there is a much stronger pattern.

The 8 failures are all recent

org.eclipse.ocl.pivot,1.0.0.v20150323-1850\ org.eclipse.ocl.xtext.completeocl,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.completeocl.ui,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.essentialocl,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.oclinecore,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.oclinecore.ui,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.oclstdlib,1.0.0.v20150323-1849\ org.eclipse.ocl,3.5.0.v20150321-0925

4 other recent builds were ok

org.eclipse.ocl.examples.codegen,2.0.0.v20150323-1850\ org.eclipse.ocl.xtext.base.ui,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.essentialocl.ui,1.0.0.v20150323-1849\ org.eclipse.ocl.xtext.oclstdlib.ui,1.0.0.v20150323-1849

The next most recent and all earlier ones are ok

org.eclipse.ocl.xtext.base,1.0.0.v20150320-1757\ org.eclipse.ocl.xtext.markup,1.0.0.v20150320-1757

So it appears to be a 66% failing caused by a change after 20 March.

Or: jdt.core caused trouble on 5-March, so we moved to Java 7 builds on Hudson long before 20 March. But there are few builds whose most recent date is between 5-Natch and 20-March, so if could regard it as a 50% failure rate for simrel Java 6 unpacking of Java 7 packing.

Or: The new certificates caused trouble from 7 March, so it is difficult to distinguish which might be an influence.

[eclipse-ocl-bot]

By Ed Willink on Mar 26, 2015 04:34

On the cross-project thread I commented that I had done a trial with java 7 expecting pack200 problems and had encountered none. This was probably because nothing had yet been packed by Java 7.

Bug 376306 seems to indicate that using the Java 7 pack200 is an absolute no no, and in Bug 376306 comment #20 you recommend Java 6 packing. So when I asked by email whether:

"David: I notice that the aggregator job options include

-Dorg.eclipse.update.jarprocessor.pack200=/shared/common/jdk1.6.0-latest/jre/bin

Should we be using a similar option in our now-1.7 build?"

shouldn't the answer have been:

"Of course you should. Why are you wasting everyone's time using Java7 packing?"

rather than

"I would find that academically interesting, but the real question is if the aggregator should be any longer, or just using with Java 7."

[eclipse-ocl-bot]

By David Williams on Mar 26, 2015 09:30

(In reply to Ed Willink from comment #2)

"Of course you should. Why are you wasting everyone's time using Java7 packing?"

That bug 376306, and bug 361628, is about issues with Java 7 and packed nested jars.

I don't believe you use packed nested jars. Right?

The best alternative is not to use nested jars, or, if you do use them, do not pack the nested jars. As I mention in bug 463010, window builder is only case I see of using packed nested jars any longer, and it appears to make no difference, at least to the aggregator.

So, that's why I'm concluding 'vm' is not the issue. (Of course, it could still be, if for example, you "pre-condition" with one vm, and then do the final pack, with another vm -- or, perhaps other cases?

But, nested packed jars was the original reason for that advise, so long ago.

[eclipse-ocl-bot]

By Ed Willink on Mar 26, 2015 10:42

(In reply to David Williams from comment #3)

I don't believe you use packed nested jars. Right?

Not as far as I know, and not according to the repo reports.

I was searching through the bugzillas to try and find the command lines since it should be possible to test interactively.

Take one of the problem jar.pack.gz's and use its jar to practice the command with Java 6 and Java 7. This should be able to replicate the problem and then allow alternative commands to be assessed bypassing the slow round-trip of an OCL build and an aggregator run..

But without studying the simrel build scripts I don't know what commands to practice.

[eclipse-ocl-bot]

By Ed Willink on Mar 30, 2015 04:17

Are we hoping that with Java 7 all round the problems will just go away?

Let me know when you want a Java 7 packed contribution.

[eclipse-ocl-bot]

By David Williams on Mar 30, 2015 14:52

(In reply to Ed Willink from comment #5)

Are we hoping that with Java 7 all round the problems will just go away?

Let me know when you want a Java 7 packed contribution.

Let's see what people say about bug 463510. That will give us some insight, if some experts chime in.

[eclipse-ocl-bot]

By David Williams on Mar 31, 2015 14:09

(In reply to David Williams from comment #6)

(In reply to Ed Willink from comment #5)

Are we hoping that with Java 7 all round the problems will just go away?

Let me know when you want a Java 7 packed contribution.

Let's see what people say about bug 463510. That will give us some insight, if some experts chime in.

I think given Thomas reply on buckminster dev list, you can "move to Java 7" (with packing them) at any time.

https://dev.eclipse.org/mhonarc/lists/buckminster-dev/msg01377.html

FWIW, even if you have trouble with some of them, from Thomas' reply, you can put an "eclipse.inf" file in the META-INF directory, that contains\ jarprocessor.exclude.pack=true

And then that one bundle won't be packed, but still signed. That's better than "not packing anything".

It'd be interesting to know "savings" (of size) if you feel so inclined.

Thanks,

[eclipse-ocl-bot]

By Ed Willink on Mar 31, 2015 18:27

The normal Java 7 Buckminster contribution now seems acceptable for OCL.

QVTd to follow.

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 06:32

(In reply to Ed Willink from comment #8)

The normal Java 7 Buckminster contribution now seems acceptable for OCL.

Only to the BUILD and VALIDATE jobs. The CLEAN-BUILD failed just as before.

Suspicion: maybe the lastSuccessful build repo that inhibits gratuitous reversioning is caching bad things. Let's go for a brand new date version all round.

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 11:01

(In reply to Ed Willink from comment #4)

But without studying the simrel build scripts I don't know what commands to practice.

What are the commands used by simrel?

[eclipse-ocl-bot]

By David Williams on Apr 01, 2015 13:45

(In reply to Ed Willink from comment #10)

(In reply to Ed Willink from comment #4)

But without studying the simrel build scripts I don't know what commands to practice.

What are the commands used by simrel?

At the point we are seeing the error, it is b3 aggregator code that is running (and probably calling out to p2 function, for this). \ http://git.eclipse.org/c/b3/b3.git/

But there is a simple way to test isolated cases (which, should be perfectly equivalent to what ever b3 editor and Java are doing). \ [This is off the top of my head, so ... don't fear to sanity check.]

For this, the jar file is not needed.

First, get a copy of the "*jar.pack.gz" file.

Then call gzip --decompress .jar.pack.gz file. \ (feel free to add any "verbose" options, if any, if that might print an error just with unzipping). \ That creates a .jar.pack file

Then call (with same VM that created it) \

/unpack200 .jar.pack .jar AND THEN, if no errors so far, \ check if new jar has a valid signature: /jarsigner --verify .jar\ (again, might find some "verbose" options handy?) Hope that helps,

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 15:20

Taking one of the failures, I used Winzip to unzip the pack.gz, and then used all four permutations of {jdk1.6.0_45/jdk.7.0_72}{unpack200/jarsigner} and they all work fine.

As far as I am concerned my contributions are perfect with no changes from the Java 6 based M5 contributions.

But perhaps an expert can analyze the content of

osgi.bundle,org.eclipse.ocl.xtext.oclinecore.ui,1.0.0.v20150323-1849

in

http://download.eclipse.org/modeling/mdt/ocl/updates/milestones/6.0.0/S201503231726

and explain what is wrong with it.

[eclipse-ocl-bot]

By David Williams on Apr 01, 2015 15:33

(In reply to Ed Willink from comment #12)

Taking one of the failures, I used Winzip to unzip the pack.gz, and then used all four permutations of {jdk1.6.0_45/jdk.7.0_72}{unpack200/jarsigner} and they all work fine.

As far as I am concerned my contributions are perfect with no changes from the Java 6 based M5 contributions.

But perhaps an expert can analyze the content of

osgi.bundle,org.eclipse.ocl.xtext.oclinecore.ui,1.0.0.v20150323-1849

in

http://download.eclipse.org/modeling/mdt/ocl/updates/milestones/6.0.0/ S201503231726

and explain what is wrong with it.

Did you remember to do the jarsigner -verify step? \ I got the jar.pack.gz file from that location, \ used gzip -d\ and unpacked that result, \ and then \ jarsigner -verify (resulting jar) \ gives: \ jarsigner: java.lang.SecurityException: SHA1 digest error for org/eclipse/ocl/xtext/oclinecore/ui/contentassist/antlr/internal/InternalOCLinEcoreParser$FollowSets001.class

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 16:00

(In reply to David Williams from comment #13)

Did you remember to do the jarsigner -verify step?

Yes.

I got the jar.pack.gz file from that location,

But I used the wrong source *.pack.gz.

With org.eclipse.ocl.xtext.oclinecore.ui,1.0.0.v20150323-1849, all 4 permutations of 6/7 unpack200/jarsigner now give me the same error.

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 17:28

(In reply to Ed Willink from comment #14)

With org.eclipse.ocl.xtext.oclinecore.ui,1.0.0.v20150323-1849, all 4 permutations of 6/7 unpack200/jarsigner now give me the same error.

Clearly the problem is in the producer.

We could force Java 6 for Buckminster and its reconditioning, but I suspect that is not the problem.

We have stayed at EMF's BREE of 5 and so have a Buckminster

setpref complianceLevel=1.5

but the platform has moved to BREE 7, without even changing service release levels, so the platform is now providing Java 7 classes. These may not play well with a 1.5 complianceLevel build.

Checking around, I note that EMFt M6 is still Java 5 classes, but is not packed. Acceleo has moved to Java 6 and packs successfully. EMF is still Java 5 and packed, but EMF was built a couple of hours before the problem started.

I suspect that EMF has a surprise in store at M7, and that overall we have a choice between surrendering Java 5 compliance or packing.

I'll try a complianceLevel=1.7.

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 18:26

(In reply to Ed Willink from comment #15)

I'll try a complianceLevel=1.7.

Makes difference. Same error. Still Jave 6 classes files content.

Maybe needs to be touched.

[eclipse-ocl-bot]

By Ed Willink on Apr 01, 2015 19:11

(In reply to Ed Willink from comment #16)

(In reply to Ed Willink from comment #15)

I'll try a complianceLevel=1.7.

Makes difference. Same error. Still Jave 6 classes files content.

Correcion: Makes no difference.\

Maybe needs to be touched.

Changing plugin to V7 forces a new version date, and the problem is fixed.

Change back to V6 to see if touching or V7 is the important change.

[eclipse-ocl-bot]

By David Williams on Apr 01, 2015 20:56

(In reply to Ed Willink from comment #17)

(In reply to Ed Willink from comment #16)

(In reply to Ed Willink from comment #15)

I'll try a complianceLevel=1.7.

Makes difference. Same error. Still Jave 6 classes files content. Correcion: Makes no difference.

Maybe needs to be touched.

Changing plugin to V7 forces a new version date, and the problem is fixed.

Change back to V6 to see if touching or V7 is the important change.

Do you use "JDT" to do the compilation? Or, "javac" from what ever jdk you have?

I ask, since if "JDT", they might be able to comment if "java 7 source to java 5 runtime compatibility" is a stable, or wise thing to do. (Correct me if that's not what you are saying.)

I know I was surprised that in some cases when compiling with Java 8 (experiments only, so far) there was actually a warning in the log, something like "compiling to Java 5 level is deprecated and will be removed next release" (or, something like that ... don't quote me :)\ IMHO, going "all the way down" to Java 5 is bound to be harder that compiling down to Java 6 (i.e. either outright errors, or "funny" byte codes). Honestly, I've not followed any of those compatibility modes.

In any case, my point is, seems like a "happy medium" if you can reliable "compile down" to Java 6, if that's important for your community.

[eclipse-ocl-bot]

By Ed Willink on Apr 02, 2015 02:08

(In reply to Ed Willink from comment #17)

Change back to V6 to see if touching or V7 is the important change.

It is broken again.

That rules out magic caches and confusion over new/old certificates.

It just seems to be that Buckminster's tooling cannot reliably support preconditioning of Java 6 plugins that have references to the Java 7 plugins that the platform now provides. I suspect that Buckminster is relevant since if the many more Tycho users were suffering too there would have been many more victims.

(In reply to Ed Willink from comment #15)

I'll try a complianceLevel=1.7.

I saw no evidence that this made any difference to Buckminster. I think that the BREE/jdt.prefs in each plugin override the Buckminster setting.

(In reply to David Williams from comment #18)

Do you use "JDT" to do the compilation? Or, "javac" from what ever jdk you have?

javac unfortunately, which means that I get hundreds of @NonNull warnings. Although Buckminster is based on JDT it doesn't seem to be used.

In any case, my point is, seems like a "happy medium" if you can reliable "compile down" to Java 6, if that's important for your community.

I have no evidence of what the community wants, it just seems reckless to ignore the EMF BREE for EMF-based projects.

[eclipse-ocl-bot]

By Ed Willink on Apr 02, 2015 02:47

(In reply to Ed Willink from comment #19)

It just seems to be that Buckminster's tooling cannot reliably support preconditioning of Java 6 plugins that have references to the Java 7 plugins

It doesn't help that the HIPPs only provide Buckminster 4.2 (Juno), and that even Buckminster 4.4 (Luna) is still Java 6-based. It is now 5 months since the last Buckminster commit, so a Buckminster fix does not seem like the way forward.

Choices: Java 7 classes throughout - increasing the ripple to consumers\ Choices: Java 7 classes selectively - an inconsistent ripple to consumers\ Choices: no pack200 selectively - unpredictable aggregator contributions\ Choices: no pack200 throughout - significant P2 download degradations

Moving to Tycho, after Mars, may eliminate the problem, so selective no pack200 may be sensible until then if we really want to retain Java 5 / Java 6 classes.

I'll check with the most obvious consumers.

[eclipse-ocl-bot]

By David Williams on Apr 02, 2015 09:21

(In reply to Ed Willink from comment #20)

(In reply to Ed Willink from comment #19)

It just seems to be that Buckminster's tooling cannot reliably support preconditioning of Java 6 plugins that have references to the Java 7 plugins

It doesn't help that the HIPPs only provide Buckminster 4.2 (Juno), and that even Buckminster 4.4 (Luna) is still Java 6-based. It is now 5 months since the last Buckminster commit, so a Buckminster fix does not seem like the way forward.

Choices: Java 7 classes throughout - increasing the ripple to consumers Choices: Java 7 classes selectively - an inconsistent ripple to consumers Choices: no pack200 selectively - unpredictable aggregator contributions Choices: no pack200 throughout - significant P2 download degradations

Moving to Tycho, after Mars, may eliminate the problem, so selective no pack200 may be sensible until then if we really want to retain Java 5 / Java 6 classes.

I'll check with the most obvious consumers.

Thanks for your work here, Ed. \ It's good know more about it, and which ever decision you make will be fine, for Mars.

[eclipse-ocl-bot]

By David Williams on Apr 06, 2015 21:43

(In reply to Ed Willink from comment #19)\

javac unfortunately, which means that I get hundreds of @NonNull warnings. Although Buckminster is based on JDT it doesn't seem to be used.

I think Buckminster does use JDT ... but ... if there is no way for you to "specify exact version" of JDT to use, then it would still be using the "old" Juno version ... from other things you've said.

There is, however, a more recent version of buckminster, apparently, at \ /shared/common/buckminster-4.4

Not sure how you get that into "Hudson", exactly, but your HIPP instance should have access to \ /shared/common/buckminster-4.4\ if there is some way to configure you version of Hudson to use that one.

And, yes, pretty sure is always uses BREE, perhaps some way to override, but a) I doubt it, and b) if you were, you'd know about it :)

If you ever want to check, you can look at the .class files, and their "magic number" (well, technically what comes right after the magic number). See

http://en.wikipedia.org/wiki/Java_class_file#General_layout

Which isn't as hard to do as it sounds, if you use Linux, you can use 'file' and it will tell you class version (easier than hex viewer).

Example:

$ file After.class\ After.class: compiled Java class data, version 49.0 (Java 1.5

HTH

[eclipse-ocl-bot]

By David Williams on Apr 06, 2015 21:46

(In reply to David Williams from comment #22)\

Not sure how you get that into "Hudson", exactly, but your HIPP instance should have access to /shared/common/buckminster-4.4 if there is some way to configure you version of Hudson to use that one.

BTW, a note to cbi-dev or bug for webmasters might provide some help on "how to configure to a more recent Buckminster".

I'm sure someone has done it :)

[eclipse-ocl-bot]

By Ed Willink on Apr 11, 2015 05:33

(In reply to David Williams from comment #0)

With so many ... I'm wondering if there is a pattern?

Certainly not just one. Glancing at the source code of some affected areas points a finger slightly at:

a) very repetitive long names perhaps involving double underscores

• perhaps a name mangling conflict
• perhaps a stronger dependency analysis allowing different re-orderings

b) 'redundant' template parameters

• Java 7 certainly has a different name hiding rule to Java 6 (Bug 464193)
• perhaps the additional templating of IAdaptable.getAdapter is provocative

[eclipse-ocl-bot]

By Ed Willink on Apr 15, 2015 07:53

Most OCL plugins have been converted to Java 7; no pack200 problems.

Classic OCL plugins could not be converted uniformly, so not at all, (Bug 464193).

org.eclipse.ocl fails to pack so packing omitted for one plugin. (Bug 464434).

[eclipse-ocl-bot]

By Ed Willink on Mar 25, 2021 16:55

Bug 572316 moves on to a post Java 14 world where pack200 is a no-op.