Repo Merge & Switch to tycho.

[cdietrich]

following the discussions in https://github.com/eclipse/xtext-core/issues/2052 and https://github.com/eclipse/xtext/issues/2133 we consider Repo Merge & Switch to tycho again

the following things we need to clarify

• [x] when to do what (M1,..,RC,GA)
• [x] monorepo including xtend ?
• [x] how to deal with (xtext-maven/, xtext-xtend/maven-plugin, xtext-web, gwt in xtext-lib), maybe time to drop some things?
• [x] how to "sign and deploy" and test it, maybe we can just sign and publish some lib and core artifacts, deploy them, let sonatype nexus validate it, but do not release them. or we build and mx. aölso how to validate the eclipse signing (how does ed in his report do that)
• [x] how to test if produced p2 and maven artifacts are proper (sample, client projects)
• [x] ....

cc @LorenzoBettini @szarnekow @kthoms

Thinks not to forget

• [x] website + website jobs
• [x] adapt github links on website(s) and source code refs
• [x] api-diff job
• [x] nightly builds of master with tests and different java and tycho versions (deploy only on changes ?!?)
• [x] oomph, finalize setup and point oomph catalogue to new loc
• [x] committer-activity-count

[cdietrich]

ps: most repos contain a years old ade-newbuild branch with the experiments @ArneDeutsch did back then

[LorenzoBettini]

the maven plugins can be part of the same repo without problems; I think they should. I'm going to try to port xtext-lib, but I have no clue about gwt (I never used it).

[mmosconi1]

Hi. We did some bigger repository merge (4 into one) some time ago (actually containing a set of Xtext DSLs). Two things you should consider first:

1. Are there branches other than 'master' that should be preserved or even merged? Each branch has to be merged separately. In our case there were several maintenance and some feature branches that also needed to be merged while retaining history.
2. Most likely you have to deal with same-named tags in the different repositories, e.g. release tags. In the end, each tag name must be unique in the targert repo. That's why we pre-processed them (in the original repositories, prior to migration) to add a prefix like "repo1/v1.2.3", repo2/v1.2.3".

To make things reproducible (and we did need some test iterations...) we wrote scripts to automate the whole process. I can provide them if you like.

[cdietrich]

@LorenzoBettini i added some npm script to package json https://github.com/eclipse/xtext-web/compare/cd_npm_experiments so that test and optimize should be callable from maven via https://github.com/aseovic/npm-maven-plugin#usage (or another node maven plugin)

[LorenzoBettini]

@LorenzoBettini i added some npm script to package json https://github.com/eclipse/xtext-web/compare/cd_npm_experiments so that test and optimize should be callable from maven via https://github.com/aseovic/npm-maven-plugin#usage (or another node maven plugin)

@cdietrich I cherry picked your commit on my branch and added the frontend-maven-plugin (I think it's a mainstream one) https://github.com/LorenzoBettini/xtext-web/commit/a8c964e2d03008c1ad40624836794c2c5fea8e0f

I don't know much about nodejs but I think I specified the goals correctly in the POM

By running mvn clean verify I can see this output (I only show a part of it). I'm not an expert but I seem to understand that everything runs fine, doesn't it :)

[INFO] --------------< org.eclipse.xtext:org.eclipse.xtext.web >---------------
[INFO] Building Xtext Web API 2.30.0-SNAPSHOT                             [4/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ org.eclipse.xtext.web ---
[INFO] Deleting /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web/target
[INFO] 
[INFO] --- build-helper-maven-plugin:3.2.0:add-test-source (add-test-sources) @ org.eclipse.xtext.web ---
[INFO] Test Source directory: /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web/src/test/xtext-gen added.
[INFO] 
[INFO] --- frontend-maven-plugin:1.12.1:install-node-and-npm (install node and npm) @ org.eclipse.xtext.web ---
[INFO] Installing node version v16.17.1
[INFO] Unpacking /Users/bettini/.m2/repository/com/github/eirslett/node/16.17.1/node-16.17.1-darwin-arm64.tar.gz into /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web/node/tmp
[INFO] Copying node binary from /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web/node/tmp/node-v16.17.1-darwin-arm64/bin/node to /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web/node/node
[INFO] Installed node locally.
[INFO] Installing npm version 8.15.0
[INFO] Unpacking /Users/bettini/.m2/repository/com/github/eirslett/npm/8.15.0/npm-8.15.0.tar.gz into /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web/node/node_modules
[INFO] Installed npm locally.
[INFO] 
[INFO] --- frontend-maven-plugin:1.12.1:npm (npm ci) @ org.eclipse.xtext.web ---
[INFO] Running 'npm ci' in /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web
[INFO] 
[INFO] added 80 packages, and audited 81 packages in 457ms
[INFO] 
[INFO] 20 packages are looking for funding
[INFO]   run `npm fund` for details
[INFO] 
[INFO] found 0 vulnerabilities
[INFO] 
[INFO] --- frontend-maven-plugin:1.12.1:npm (javascript tests) @ org.eclipse.xtext.web ---
[INFO] Running 'npm run test' in /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT test
[INFO] > mocha -u tdd --reporter dot src/test/js/*.js
[INFO] 
[INFO] 
[INFO] 
[INFO]   .....................................................
[INFO] 
[INFO]   53 passing (278ms)
[INFO] 
[INFO] 
[INFO] --- frontend-maven-plugin:1.12.1:npm (optimize) @ org.eclipse.xtext.web ---
[INFO] Running 'npm run optimize' in /Users/bettini/work/xtext/xtext-sources/xtext-web/org.eclipse.xtext.web
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimize
[INFO] > npm run optimizeAce && npm run optimizeOrion && npm run optimizeCodeMirror
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeAce
[INFO] > npm run optimizeAceJs && npm run optimizeAceJsMin && npm run optimizeAceCss
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeAceJs
[INFO] > r.js -o src/main/js/requirejs-ace-config.js logLevel=2 out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-ace.js optimize=none
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeAceJsMin
[INFO] > r.js -o src/main/js/requirejs-ace-config.js logLevel=2 out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-ace.min.js optimize=uglify preserveLicenseComments=false
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeAceCss
[INFO] > r.js -o logLevel=2 cssIn=src/main/css/xtext/xtext-ace.css out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-ace.css
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeOrion
[INFO] > npm run optimizeOrionJs && npm run optimizeOrionJsMin && npm run optimizeOrionCss
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeOrionJs
[INFO] > r.js -o src/main/js/requirejs-orion-config.js logLevel=2 out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-orion.js optimize=none
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeOrionJsMin
[INFO] > r.js -o src/main/js/requirejs-orion-config.js logLevel=2 out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-orion.min.js optimize=uglify preserveLicenseComments=false
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeOrionCss
[INFO] > r.js -o logLevel=2 cssIn=src/main/css/xtext/xtext-orion.css out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-orion.css
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeCodeMirror
[INFO] > npm run optimizeCodeMirrorJs && npm run optimizeCodeMirrorJsMin && npm run optimizeCodeMirrorCss
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeCodeMirrorJs
[INFO] > r.js -o src/main/js/requirejs-codemirror-config.js logLevel=2 out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-codemirror.js optimize=none
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeCodeMirrorJsMin
[INFO] > r.js -o src/main/js/requirejs-codemirror-config.js logLevel=2 out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-codemirror.min.js optimize=uglify preserveLicenseComments=false
[INFO] 
[INFO] 
[INFO] > xtext@2.30.0-SNAPSHOT optimizeCodeMirrorCss
[INFO] > r.js -o logLevel=2 cssIn=src/main/css/xtext/xtext-codemirror.css out=build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-codemirror.css
[INFO] 

[cdietrich]

@LorenzoBettini this looks good. the question is: here is how the jar looks like with gradle

jar tf org.eclipse.xtext.web-2.27.0-SNAPSHOT.jar | grep js
META-INF/resources/xtext/2.27.0-SNAPSHOT/xtext-ace.js
META-INF/resources/xtext/2.27.0-SNAPSHOT/xtext-ace.min.js
META-INF/resources/xtext/2.27.0-SNAPSHOT/xtext-codemirror.js
META-INF/resources/xtext/2.27.0-SNAPSHOT/xtext-codemirror.min.js
META-INF/resources/xtext/2.27.0-SNAPSHOT/xtext-orion.js
META-INF/resources/xtext/2.27.0-SNAPSHOT/xtext-orion.min.js

in your jar they are not packaged maybe the build/opt-resources/META-INF/resources/xtext/${npm_package_version}/xtext-codemirror.css expressions also need adaptation

[LorenzoBettini]

I've been working on that, and now they are:

[LorenzoBettini]

I've basically updated all the projects (with tests as well).

The one that remains to be done is org.eclipse.xtext.web.example.jetty.

That one is not meant to be deployed to Maven Central, is it? I don't see it on Maven Central. That is a WAR project I guess.

[cdietrich]

yes and no. its not deployed. but it does two things. it produces a war to test there is a gradlew jettyRun command that status jetty locally to test (basically running the org.eclipse.xtext.web.example.jetty.ServerLauncher, same as if you would run it from eclipse)

[LorenzoBettini]

yes and no. its not deployed. but it does two things. it produces a war to test there is a gradlew jettyRun command that status jetty locally to test (basically running the org.eclipse.xtext.web.example.jetty.ServerLauncher, same as if you would run it from eclipse)

Let's see what I can do with that project. I see that also Orion has to be downloaded...

[cdietrich]

yes there is unfortunately no webjar for it

[LorenzoBettini]

So, I converted also that project and configured the POM to download and unzip Orion.

If I run the example I get this, which looks fine, right?

[cdietrich]

yes. and the same should work for ace and codemirror

[LorenzoBettini]

ace and codemirror were already working because they are part of our sources; Orion instead had to be downloaded and unzipped

[szarnekow]

Great work, @LorenzoBettini ! I'm amazed how you powered through this involving change of the build infra. I'm really looking forward to merging this after the release and then tackle the repo merge.

[cdietrich]

again my question: can we already downport parts like manifest/build.properties changes and renamings?

[szarnekow]

Starting with the class renames would make sense. Having these consistent prior to the subsequent changes, allows for easier diffing the results.

[LorenzoBettini]

@cdietrich, I forgot to answer your past question: I'm not sure what you meant. If you mean merging the PRs, I'd first need a few small adjustments and, most of all, harmonize xtext-maven (and I'm not yet done with xtext-web).

@szarnekow thank you for the kind words :) For sure, concerning xtext-web, I wouldn't be able to port the node part without the help of Christian.

I'd also need some additional information concerning the org.eclipse.xtext.web.example.jetty. The last part that I still have to do is packaging /org.eclipse.xtext.web.example.jetty/test-files. I'm currently running the example server from Eclipse, so those files are found. But they need to be packaged into the WAR file.

By looking at the build.gradle

task zipTestFiles(type: Zip) {
    from('test-files') {
        include '**/*'
        into 'test-files'
    }
    destinationDirectory = new File( project.buildDir,'libs')
    archiveFileName = 'test-files.zip'
}

I seem to understand that the contents of test-files directory must be zipped and put into the output folder inside "libs"?

If you already have a WAR file generated using the current configuration, could you please show me the expected layout of the WAR?

This is the current layout (without test-libs):

org.eclipse.xtext.web.example.jetty-2.30.0-SNAPSHOT
├── ace-statemachine-resource.html
├── ace-statemachine-resource-multi.html
├── ace-statemachine-stateless.html
├── ace-statemachine-static.html
├── ace-statemachine-static-multi.html
├── codemirror-statemachine-resource.html
├── codemirror-statemachine-resource-multi.html
├── codemirror-statemachine-stateless.html
├── codemirror-statemachine-static.html
├── codemirror-statemachine-static-multi.html
├── favicon.ico
├── i18n.js
├── images
│   └── output.gif
├── index.html
├── META-INF
├── orion
│   ├── code_edit
│   │   ├── built-codeEdit-amd.js
│   │   ├── built-codeEdit-amd.min.js
│   │   ├── built-codeEdit.css
│   │   ├── built-codeEdit.js
│   │   └── built-codeEdit.min.js
│   ├── javascript
│   │   └── plugins
│   │       ├── javascriptPlugin.html
│   │       ├── javascriptPlugin.js
│   │       ├── javascriptPlugin-non-minified.js
│   │       ├── ternWorkerCore.js
│   │       ├── ternWorkerCore-non-minified.js
│   │       └── ternWorker.js
│   ├── orion
│   │   ├── require-config.js
│   │   └── requireErrorHandler.js
│   ├── others
│   │   └── plugins
│   │       ├── jslintPlugin.html
│   │       ├── jslintPlugin.js
│   │       └── languages
│   │           └── json
│   │               ├── jsonPlugin.html
│   │               └── jsonPlugin.js
│   ├── plugins
│   │   ├── embeddedToolingPlugin.html
│   │   └── embeddedToolingPlugin.js
│   ├── requirejs
│   │   └── require.min.js
│   └── webtools
│       └── plugins
│           ├── webToolsPlugin.html
│           └── webToolsPlugin.js
├── orion-entities-resource.html
├── orion-entities-static.html
├── orion-statemachine-autogenerate.html
├── orion-statemachine-resource.html
├── orion-statemachine-resource-multi.html
├── orion-statemachine-stateless.html
├── orion-statemachine-static.html
├── orion-statemachine-static-multi.html
├── style.css
├── WEB-INF
│   ├── classes
│   │   └── org
│   │       └── eclipse
│   │           └── xtext
│   │               └── web
│   │                   └── example
│   │                       └── jetty
│   │                           ├── AbstractEntitiesWebModule.class
│   │                           ├── AbstractStatemachineWebModule.class
│   │                           ├── EntitiesWebModule.class
│   │                           ├── EntitiesWebSetup.class
│   │                           ├── MyXtextServlet.class
│   │                           ├── resource
│   │                           │   ├── StatemachineContentTypeProvider.class
│   │                           │   └── StatemachineResourceSetProvider.class
│   │                           ├── ServerLauncher$1.class
│   │                           ├── ServerLauncher.class
│   │                           ├── StatemachineWebModule.class
│   │                           └── StatemachineWebSetup.class
│   └── lib
│       ├── ace-1.3.3.jar
│       ├── antlr-runtime-3.2.jar
│       ├── aopalliance-1.0.jar
│       ├── asm-9.4.jar
│       ├── checker-qual-3.5.0.jar
│       ├── classgraph-4.8.149.jar
│       ├── codemirror-5.41.0.jar
│       ├── error_prone_annotations-2.3.4.jar
│       ├── failureaccess-1.0.1.jar
│       ├── gson-2.9.1.jar
│       ├── guava-30.1-jre.jar
│       ├── guice-5.0.1.jar
│       ├── j2objc-annotations-1.3.jar
│       ├── javax.inject-1.jar
│       ├── jquery-3.6.0.jar
│       ├── jsr305-3.0.2.jar
│       ├── listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
│       ├── log4j-1.2.17.jar
│       ├── org.eclipse.emf.common-2.24.0.jar
│       ├── org.eclipse.emf.ecore-2.26.0.jar
│       ├── org.eclipse.emf.ecore.change-2.14.0.jar
│       ├── org.eclipse.emf.ecore.xmi-2.16.0.jar
│       ├── org.eclipse.equinox.common-3.16.200.jar
│       ├── org.eclipse.lsp4j-0.19.0.jar
│       ├── org.eclipse.lsp4j.generator-0.19.0.jar
│       ├── org.eclipse.lsp4j.jsonrpc-0.19.0.jar
│       ├── org.eclipse.osgi-3.18.100.jar
│       ├── org.eclipse.xtend.lib-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtend.lib.macro-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.common.types-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.ide-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.util-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.web-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.web.example.entities-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.web.example.entities.ide-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.web.example.statemachine-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.web.example.statemachine.ide-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.web.servlet-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.xbase-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.xbase.ide-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.xbase.lib-2.30.0-SNAPSHOT.jar
│       ├── org.eclipse.xtext.xbase.web-2.30.0-SNAPSHOT.jar
│       ├── reload4j-1.2.24.jar
│       ├── requirejs-2.3.6.jar
│       ├── requirejs-text-2.0.15.jar
│       ├── slf4j-api-1.7.33.jar
│       └── slf4j-log4j12-1.7.33.jar
└── xtext
    ├── ace-mode-entities.js
    ├── ace-mode-statemachine.js
    ├── cm-mode-entities.js
    ├── cm-mode-statemachine.js
    ├── entities-syntax.js
    └── statemachine-syntax.js

[cdietrich]

Great work, @LorenzoBettini ! I'm amazed how you powered through this involving change of the build infra. I'm really looking forward to merging this after the release and then tackle the repo merge.

+1 on that

@LorenzoBettini no i dont mean merge the prs. i mean merge parts of the changes that can be safely done on master.

[cdietrich]

about the jetty war. i have no idea about the testlibs. this also might be a bug. here is from master:

META-INF/
META-INF/MANIFEST.MF
WEB-INF/
WEB-INF/classes/
WEB-INF/classes/org/
WEB-INF/classes/org/eclipse/
WEB-INF/classes/org/eclipse/xtext/
WEB-INF/classes/org/eclipse/xtext/web/
WEB-INF/classes/org/eclipse/xtext/web/example/
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/ServerLauncher.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/StatemachineWebModule.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/AbstractStatemachineWebModule.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/EntitiesWebSetup.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/resource/
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/resource/StatemachineContentTypeProvider.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/resource/StatemachineResourceSetProvider.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/ServerLauncher$1.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/StatemachineWebSetup.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/AbstractEntitiesWebModule.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/MyXtextServlet.class
WEB-INF/classes/org/eclipse/xtext/web/example/jetty/EntitiesWebModule.class
WEB-INF/lib/
WEB-INF/lib/org.eclipse.xtext.web.servlet-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.web.example.statemachine.ide-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.xbase.web-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.web.example.entities.ide-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.web-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.web.example.statemachine-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.xbase.ide-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.ide-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.web.example.entities-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.xbase-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.common.types-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.util-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.lsp4j-0.19.0.jar
WEB-INF/lib/org.eclipse.lsp4j.generator-0.19.0.jar
WEB-INF/lib/org.eclipse.xtend.lib-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtend.lib.macro-2.30.0-SNAPSHOT.jar
WEB-INF/lib/org.eclipse.xtext.xbase.lib-2.30.0-SNAPSHOT.jar
WEB-INF/lib/requirejs-text-2.0.15.jar
WEB-INF/lib/requirejs-2.3.6.jar
WEB-INF/lib/jquery-3.6.0.jar
WEB-INF/lib/ace-1.3.3.jar
WEB-INF/lib/codemirror-5.41.0.jar
WEB-INF/lib/slf4j-reload4j-1.7.36.jar
WEB-INF/lib/reload4j-1.2.24.jar
WEB-INF/lib/guice-5.0.1.jar
WEB-INF/lib/aopalliance-1.0.jar
WEB-INF/lib/guava-30.1-jre.jar
WEB-INF/lib/jsr305-3.0.2.jar
WEB-INF/lib/error_prone_annotations-2.3.4.jar
WEB-INF/lib/classgraph-4.8.149.jar
WEB-INF/lib/antlr-runtime-3.2.jar
WEB-INF/lib/org.eclipse.emf.ecore.change-2.14.0.jar
WEB-INF/lib/org.eclipse.emf.ecore.xmi-2.16.0.jar
WEB-INF/lib/org.eclipse.emf.ecore-2.26.0.jar
WEB-INF/lib/org.eclipse.emf.common-2.24.0.jar
WEB-INF/lib/org.eclipse.lsp4j.jsonrpc-0.19.0.jar
WEB-INF/lib/org.eclipse.equinox.common-3.16.200.jar
WEB-INF/lib/org.eclipse.osgi-3.18.100.jar
WEB-INF/lib/gson-2.9.1.jar
WEB-INF/lib/slf4j-api-1.7.36.jar
WEB-INF/lib/javax.inject-1.jar
WEB-INF/lib/failureaccess-1.0.1.jar
WEB-INF/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
WEB-INF/lib/checker-qual-3.5.0.jar
WEB-INF/lib/j2objc-annotations-1.3.jar
ace-statemachine-static-multi.html
codemirror-statemachine-static.html
codemirror-statemachine-stateless.html
ace-statemachine-resource.html
orion-entities-static.html
orion/
orion/webtools/
orion/webtools/plugins/
orion/webtools/plugins/webToolsPlugin.html
orion/webtools/plugins/webToolsPlugin.js
orion/requirejs/
orion/requirejs/require.min.js
orion/javascript/
orion/javascript/plugins/
orion/javascript/plugins/ternWorker.js
orion/javascript/plugins/javascriptPlugin.js
orion/javascript/plugins/ternWorkerCore-non-minified.js
orion/javascript/plugins/ternWorkerCore.js
orion/javascript/plugins/javascriptPlugin.html
orion/javascript/plugins/javascriptPlugin-non-minified.js
orion/orion/
orion/orion/requireErrorHandler.js
orion/orion/require-config.js
orion/others/
orion/others/plugins/
orion/others/plugins/jslintPlugin.js
orion/others/plugins/languages/
orion/others/plugins/languages/json/
orion/others/plugins/languages/json/jsonPlugin.html
orion/others/plugins/languages/json/jsonPlugin.js
orion/others/plugins/jslintPlugin.html
orion/code_edit/
orion/code_edit/built-codeEdit-amd.js
orion/code_edit/built-codeEdit.js
orion/code_edit/built-codeEdit.css
orion/code_edit/built-codeEdit.min.js
orion/code_edit/built-codeEdit-amd.min.js
orion/plugins/
orion/plugins/embeddedToolingPlugin.html
orion/plugins/embeddedToolingPlugin.js
ace-statemachine-static.html
index.html
orion-statemachine-autogenerate.html
codemirror-statemachine-static-multi.html
style.css
codemirror-statemachine-resource-multi.html
orion-statemachine-static.html
images/
images/output.gif
ace-statemachine-resource-multi.html
favicon.ico
orion-entities-resource.html
ace-statemachine-stateless.html
xtext/
xtext/cm-mode-statemachine.js
xtext/entities-syntax.js
xtext/cm-mode-entities.js
xtext/ace-mode-statemachine.js
xtext/ace-mode-entities.js
xtext/statemachine-syntax.js
codemirror-statemachine-resource.html
orion-statemachine-resource.html
orion-statemachine-stateless.html
i18n.js
orion-statemachine-resource-multi.html
orion-statemachine-static-multi.html

[LorenzoBettini]

mh... there's no trace of test-libs. If you run the example with jetty, you should get an error when trying to select one of the resources:

[cdietrich]

this trys to load file from disk. see https://github.com/eclipse/xtext-umbrella/blob/0c7d12aa264b140279658c0d416ca590ca667d2c/releng/docker/xtext-web-deploy/Jenkinsfile#L24

i dont know to which path this points to in your example so the question would be what is the "." dir in gradle if jetty is run

[LorenzoBettini]

@cdietrich can you please confirm that if you run jetty (with gradle) from that project it finds the test input files or it doesn't?

The Jenkinsfile doesn't tell me much, because it only seems to confirm that the test files are not in the WAR.

[cdietrich]

it is. and they are picked separately from filesystem. so that they also can be written/saved

[cdietrich]

jps | grep ServerL
852698 ServerLauncher
pwdx 852698
852698: /home/dietrich/xtext-master/git/xtext-web/org.eclipse.xtext.web.example.jetty

[LorenzoBettini]

so it's using the actual source files? If you modify a file and save it it becomes dirty in Eclipse? (just like it happens if you run the Java main file from Eclipse, instead of jetty)

[cdietrich]

so it's using the actual source files?

yes. but you can also copy then relative to the cwd of the process you start

[LorenzoBettini]

so it's using the actual source files?

yes. but you can also copy then relative to the cwd of the process you start

I don't understand your last answer.

However, I seem to understand that the zip of test files has nothing to do with jetty: you need them to then create a Docker image, right?

So, the jetty maven plugin is not really needed in this project, or am I missing something? I mean, it starts correctly from Eclipse.

[LorenzoBettini]

By running your command, this is what I get:

jps | grep ServerL      
5738 XMLServerLauncher

As I said, I'll avoid digging this further, because it works from Eclipse.

I'm also packaging the zip file into target/libs/test-files.zip so that, once adapted, the Jenkinsfile will find the .zip file with test resources.

OK?

[LorenzoBettini]

@cdietrich by looking at my output and yours, I can confirm that I see ServerLauncher only if I run the Java main class from Eclipse.

Moreover, by looking at the gradle file:

task jettyRun(type:JavaExec) {
    dependsOn(sourceSets.main.runtimeClasspath, unpackOrion)
    classpath = sourceSets.main.runtimeClasspath.filter{it.exists()}
    mainClass = "org.eclipse.xtext.web.example.jetty.ServerLauncher"
    standardInput = System.in
}

"jetty" is just the name of the task: it does not use any specific jetty plugin (unless I miss something): this corresponds to run the Java main class (as we do from Eclipse). After all, our ServerLauncher manually starts jetty.

I thought it was using the gradle version of the jetty maven plugin, but it doesn't.

So, it's useless to fight with jetty maven plugin.

[cdietrich]

the task is called jettyRun and it runs the ServerLauncher which itself runs embedded jetty. => is there an exec maven plugin that starts and waits for input so that script can stop. and then be called with mvn doSomething?

=> question would be how you start jetty and what is the working dir. (cwd)

[LorenzoBettini]

Probably the exec maven plugin that would correspond to start the main file from eclipse, and the cwd would probably be the current project's basedir. Why would you want to do that?

Jetty is still started from the main Java file.

[cdietrich]

cause this is what jettyRun does? we want to start ServerLauncher in basedir. then the test files can be found

[LorenzoBettini]

You do that from eclipse, by running the java class with main: run as java application.

[cdietrich]

Yes and there is an option to do it from cmd line. So you propose to remove this option

[LorenzoBettini]

One reason why having a maven command (based on exec plugin) could be useful is that to enjoy the real example, orion must be already downloaded and unpacked, and that is done via Maven.

Once orion has been downloaded and unpacked, it can be run from Eclipse. Without Orion, the orion version of the example would lack syntax highlighting (and code completion, I guess).

Let's see if I can make it easy with a maven launch configuration and exec plugin.

[LorenzoBettini]

I implemented a possible solution to run the example with exec plugin.

However, at least once, one has to run the build with install phase from the aggregator project, and then the exec plugin can be run on the example project. In fact, for running the example with exec plugin, we must be in the directory of the web example project (we knew that).

I also created two launch configurations (one for installing the dependencies and one for running).

That's the only thing that's working if one wants to run it with exec plugin (at least, the only solution I could come up with)

[cdietrich]

That’s ok. We should not spend too much time on corner cases

[LorenzoBettini]

Great work, @LorenzoBettini ! I'm amazed how you powered through this involving change of the build infra. I'm really looking forward to merging this after the release and then tackle the repo merge.

+1 on that

@LorenzoBettini no i dont mean merge the prs. i mean merge parts of the changes that can be safely done on master.

So you would cherry-pick the renamed test cases, right?

Concerning build.properties and MANIFEST, would would be the benefit? They are not used in the current build by Gradle.

In general, this partial merge might take some time, so are we sure we could benefit?

[cdietrich]

we will have to do version bump and merges after the current release. => from my pov the less changes that are on your branches the better. (am not that experienced with gigantic merges)

[LorenzoBettini]

Should I take care of cherry picking? IIRC renamings are single commits, but manifest and build property changes are not. I guess it'll take some time..

[cdietrich]

Yes this would be nice. Maybe just copy files over also does work

[LorenzoBettini]

OK, I'll see how easy and feasible it can be. I'll create another branch, something like lb_2052_cherry_pick. Of course, in the meantime, JIRO started to be slow and low on resources...

[LorenzoBettini]

So the cherry-picking is done since xtext-lib did not introduce any such changes, xtext-eclipse was already tycho-based, and xtext-web has nothing that can be cherry-picked.

[cdietrich]

thx @LorenzoBettini

i wonder if we can create a branch of xtext-reference projects that uses the n branch repos from ci.eclipse.org/xtext/repo/yourbranch to consume xtext artifacts as well as the p2 repo from your umbrellabranch in the tycho based builds

[LorenzoBettini]

thx @LorenzoBettini

i wonder if we can create a branch of xtext-reference projects that uses the n branch repos from ci.eclipse.org/xtext/repo/yourbranch to consume xtext artifacts as well as the p2 repo from your umbrellabranch in the tycho based builds

Yes, that'd be a good idea!

Will you take care of that or should I?

Concerning the next steps, what would you propose? (I'm also working on a temporary Oomph setup for the Git repositories I'm working on)

[cdietrich]

if you can find time before monday you can do otherwise i will do on monday

[LorenzoBettini]

if you can find time before monday you can do otherwise i will do on monday

I can try that now; could you please provide any hint on where I should modify things appropriately?

[cdietrich]

https://github.com/itemis/xtext-reference-projects/tree/master/domainmodel/2.30.0

https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-gradle/2.30.0/org.xtext.example.mydsl.parent/build.gradle#L16

https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-gradle/2.30.0-J11/org.xtext.example.mydsl.parent/build.gradle#L16

https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-maven/2.30.0/org.xtext.example.mydsl.parent/pom.xml#L130 https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-maven/2.30.0/org.xtext.example.mydsl.parent/pom.xml#L153

https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-maven/2.30.0-J11/org.xtext.example.mydsl.parent/pom.xml#L130 https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-maven/2.30.0-J11/org.xtext.example.mydsl.parent/pom.xml#L153

https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-tycho/2.30.0/org.xtext.example.mydsl.parent/pom.xml#L272 https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-tycho/2.30.0/org.xtext.example.mydsl.parent/pom.xml#L317 https://github.com/itemis/xtext-reference-projects/blob/master/greetings-tycho/2.30.0/org.xtext.example.mydsl.target/org.xtext.example.mydsl.target.target

https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-tycho/2.30.0-J11/org.xtext.example.mydsl.parent/pom.xml#L272 https://github.com/itemis/xtext-reference-projects/blob/9479fd3b902a40d5a65a1a6976539203d39ee8c3/greetings-tycho/2.30.0-J11/org.xtext.example.mydsl.parent/pom.xml#L317 https://github.com/itemis/xtext-reference-projects/blob/master/greetings-tycho/2.30.0-J11/org.xtext.example.mydsl.target/org.xtext.example.mydsl.target.target

[LorenzoBettini]

mh... so there's no automatic mechanism, is there?

I can manually adjust the target platforms because we have a unique p2 repository (even with a find/replace), but for the Maven artifacts we have several of them...