3.0 Roadmap

[scphantm]

I have made extensive changes to this code base. I have basically completely scrapped and rewritten large swaths of code to make this work correctly in my enterprise. To put it simply, my enterprise build system expected this to work like a gradle plugin, not an ant plugin, so everything blew up on me. Ive rewritten the source directory structure, the configuration file management system, the deploy task was basically deleted and rewritten from scratch, cross and subproject functionality was added, and the kick in the gut liberty gave me today means im most likely going to have to gut and rewrite the InstallApps task as well.

I am currently in my corporate process to contribute my changes back to the project, i hope to have final permission by January.

My changes are so extensive, a 3.0 release is without question, and a conversation really needs to happen if they are drastic enough to warrant a new project. Especially since i had to redefine the source folders to accommodate IDE's like IntelliJ that don't allow overlapping sourcesets.

While i am waiting on my VP and Legal to give me permission to push this stuff back to you guys, how do we start a conversation about what should be in a 3.0 release. especially since there's a high likelyhood im going to have to scrap this InstallApps task.

[hrstoyanov]

Thank you! I think this forum is fine, but in order to prevent further development on the current branch, you should publish the code asap.

[scphantm]

trust me, i would love to. I have a meeting with my tech committee and a meeting with my VP to get permission. Nothing happens fast in my shop, trust me. Im plowing thru the red tape as fast as i can.

[hughesj]

I have made extensive changes to this code base... Especially since i had to redefine the source folders to accommodate IDE's like IntelliJ that don't allow overlapping sourcesets.

This is exactly the kind of feedback that's useful to us so thank you! We want this to work from command line, Eclipse IDE, IntelliJ IDEA, VSCode etc etc. So what would be great is to see a list of problems/challenges you had with the existing codebase and see a PR for each. If/when you get approval to donate your changes, I imagine a phase of sifting through that, to determine the what and the why. It will be so much easier if that is split out into specific problems you had with the existing codebase.

[scphantm]

Ok, i got thru my first hurdle today. one last one to go and i should be able to commit.

In the mean time... I am not able to run the unit test suite in my network because all the download sites used by the unit test system (repos and such) are blocked by my firewall. What you can do while im getting approval is start adding some CI features to this project. First is this

https://travis-ci.org/getting_started https://www.appveyor.com/

These are CI servers. Create an account and configure this project on them to run the unit and integration suite on every commit. Its important to use both because its quite common in the gradle world to have something run on Linux but not Windows. Travis is a linux based ci/cd and appveyor is windows.

This way, when I commit, the unit test suite will run and I can work on getting those working.

[scphantm]

BTW, you can check out this project

https://github.com/linkedin/pygradle

for an example of both travis and appveyor configured. there are config yaml files you need to put in the root.

[hughesj]

@scphantm we do have a .travis.yml and do run our builds on travis-ci.org and there is a build status badge embedded in the README.md. However, I just checked the log for jdk8 and it looks like the integration tests aren't being run - although they are on my laptop. Will look into it.

[scphantm]

yea, i missed the travis file. you should hook it up to appveyor too. just so its tested on windows as well. I tried hacking the unit suite so it will run on my network but it would take WAY more work than i care to put into it. ill just use travis to run the suite.

[scphantm]

I finished merging in the latest code last night. I'm quite sure that i screwed up some of the new features, but i am in the midst of testing now. When my VP gives me the thumbs up, i am going to submit this as several pull requests, one stacked on the other. First is everything done to date. Thats rebuilding the sourceset system, reengineering the config files compilers and deployers, the rebuilt deploy task with the configuration based deployer, and the first stage of the overhauled Liberty.groovy. The code for this is done, the documentation and unit tests need updated before its ready to go.

the second PR will be stacked on top of the initial foundation PR and include the reengineered installApps task. I am going to rebuild that in the same style as deploy. Making use of Gradles API instead of Java's api, simplifying, and making better use of Gradle's incremental builds and workflow engine.

Third PR will be rebuilding the liberty-plugin-config task. Im going to pull that into the same style as the rest of the plugin

Last PR will be CodeNarc. When i opened this code in intellij, intellij almost had a stroke with all the groovy errors in it. I had to disable a lot of the groovy checks for the project to get any speed out of it. Adding CodeNarc will help clear all that up.

When all that is tested and documented, that should be a release 3.0. Vastly simplified, much more modular, and acting like an actual gradle plugin instead of an Ant plugin, which was my original problem. our enterprise gradle system didn't know what to make of this, especially sourcesets and problems with multi-project builds, thats why i set out in rebuilding it. The new stuff works with gradle archetypes plugins with seed tasks (due to the sourceset stuff) and it very easily supports multi-project builds with multiple ears/wars being deployed to a single server (which 100% of my apps are). It even supports pulling an ear file out of a maven repo and deploying it pretty easily. (least it should, havn't tested that one yet)

[hughesj]

When you commit to GitHub could you use git -s to sign off with the DCO - thanks.

[scphantm]

Oh man, thats a rough one. That will take months to setup. The problem is my company still thinks its the 90's. the only way we are allowed to sign something is if we push it thru our CI server. our CI server is the only thing authorized to sign things. now, there are several problems. first, i would need about 3 more approvals before i could do that. First being us update our NDA with IBM to include this work. Then my next problem is the build.gradle file is completely incompatible with my CI server, as our CI server is setup to only build things that are inventoried on our network (thank SOC2, ISO20000-1, ISO9001, ITIL, and HIGHTRUST for that one) so i would have to rebuild the build.gradle to be compatible with our build system. That would in turn remove my authority to publish to you guys because the new build file would violate our competitively sensitive information clauses.

So, to make that happen, I would have to start a series of meetings with our security and audit team in order to change the policy to allow single developers to digitally sign documents outside of the CI server. Just scheduling the meeting won't happen till January at least. Then it typically takes 6 month to change a policy like that, if I can convince them to do it.

I know the process because i had a team want to publish an IPhone app this year. When we figured out what it would take to sign the executable (cuz our CI system doesn't support iOS builds, yet), the team decided that was easier to scrap the app and just rebuild the website to be mobile friendly.

So is it really that big of a deal?

[scphantm]

Whats the purpose of the ApplicationXmlDocument class? i see it being called from InstallAppsTask, but i don't see the point of it. Is this to accommodate things being dropped in the apps folder that are not listed in the server.xml?

[mattbsox]

Yup, we're using it to create the new xml file that gets added to configDropins when an application is getting installed to apps but isn't configured in the server.xml.

[scphantm]

yea, I finally figured it out, one of my liberty admins told me about that configDropins. thats what screwed me up.

in my rebuilt system, im adding a parameter to the server extension to turn that off. That will wreak havoc in my company especially. Primarily because we have spent thousands of man hours this past year to make developer machines look like the servers. So the liberty config files thats on the developer machine will also be the config files that are deployed to the server by the CI system. So if the thing isn't configured in server.xml correctly, i don't want it to just take care of it, i want it to blow up until the developer fixes the config file. otherwise it will blow up when they deploy the ear to the server. But I do see some people wanting that so a switch to turn it on and off is the best.

[Azquelt]

The git -s signoff is not a digital signature, it just adds a line to the end of your commit message like this: Signed-off-by: Super Developer <super.dev@gmail.com>

Including this line means that you certify your contribution under the Developer Certificate of Origin - which basically says that either you own the code, or you have permission to contribute it from whoever does own the code.

In this case, it's basically a formal and well-recognised way of saying that you have permission from your employer to contribute the code. Have a read through the link above (it's pretty short) and if you're happy, do your commits with git commit --signoff

[scphantm]

just a heads up, I did get this approved. My code reviewer will be back next week and i can open up the first MR. At the moment, we finished rebuilding the core, and im working on the unit tests.

We were a little surprised when we got the unit suite to run and very few tests failed, we expected pretty much all of them to fail. We dug a bit deeper and realized that you aren't using TestKit and are trying to inject the test classpath manually. The tests didn't fail because we didn't break anything, the tests didn't fail because we werent running it against our code, we were running it against the previous release code. Thats this weeks problem to solve.