Jar patcher

[plavjanik]

Resolves #27

This includes a new subproject called jarpatcher. This project is a simple Java application with the main class jarpatcher.JarPatcher without any dependencies at runtime. It can compare two JAR files and create a patch JAR file that contains new and updated entries and special empty entries for deleted entries (prefixed by _deleted_-). The JarPatcher classes are added to the patch and ignored during patching.

The zowe-api-dev deploy and zosbuild commands do not upload files when they have not changed. It remembers the last file in .zowe-api-dev/uploadedFiles/${zoweProfile} directory. If the changed file is a JAR file then it creates the patch and uploads it to z/OS. It executes the patcher that it is included in the patch and that updates the target JAR file. Patch files are only few kilobytes large while full JARs have more than 50 megabytes.

[codecov[bot]]

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@cf8c6e8). Click here to learn what that means. The diff coverage is 87.41%.

@@            Coverage Diff            @@
##             master      #59   +/-   ##
=========================================
  Coverage          ?   87.41%           
  Complexity        ?       39           
=========================================
  Files             ?        1           
  Lines             ?      143           
  Branches          ?       24           
=========================================
  Hits              ?      125           
  Misses            ?       10           
  Partials          ?        8

Impacted Files	Coverage Δ	Complexity Δ
...arpatcher/src/main/java/jarpatcher/JarPatcher.java	87.41% <87.41%> (ø)	39 <39> (?)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cf8c6e8...23ea3ee. Read the comment docs.

[dkelosky]

I'll test this this morning (hopefully :-) )

[dkelosky]

On Windows I'm getting:

$ ./gradlew build

> Task :jarpatcher:test

jarpatcher.JarPatcherTests > directoriesCanBeAddedToEmptyArchive FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > filesCanBeDeletedFromDirectoryInArchive FAIL
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > sameArchivesRemainSame FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > filesCanBeChangedInArchive FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > emptyFilesCanBeAddedToEmptyArchive FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > filesCanBeAddedToEmptyArchive FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > directoryCanBeDeletedInArchive FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > filesCanBeAddedToDirectoryInArchive FAILED
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

jarpatcher.JarPatcherTests > lastFileCanBeDeletedFromDirectoryInArchive F
    java.lang.RuntimeException at JarPatcherTests.java:183
        Caused by: java.lang.IllegalArgumentException at JarPatcherTests.
            Caused by: java.net.URISyntaxException at JarPatcherTests.jav

9 tests completed, 9 failed

> Task :jarpatcher:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jarpatcher:test'.
> There were failing tests. See the report at: file:///C:/dev/java/sample

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or -

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatibld, making it incompatible with Gradle 6.0.       eprecation warnings.
Use '--warning-mode all' to show the individual dnd_line_interface.html#seprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

[dkelosky]

Last commit fixed my tests - trying this out now

[dkelosky]

Just recording problems as they occur. On gradlew build from in zowe-rest-api-sample-spring I get:

$ ./gradlew build
> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.zowe:zowe-rest-api-commons-spring:0.0.0-SNAPSHOT.
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 11s
1 actionable task: 1 executed

[dkelosky]

Built via cd to sample-spring-boot-api-service then:

• ./gradlew zowe-rest-api-commons-spring:build
• ./gradlew zowe-rest-api-sample-spring:build

zowe-api-dev deploy took 1min 35sec wall clock time on first deploy.

I changed GreetingController.java field: private static final String DEFAULT_NAME = "dan";

Rebuild via cd .. && ./gradlew zowe-rest-api-sample-spring:build.

I also had to upgrade @zowedev/zowe-api-dev to 0.4.2.

I'm sure I'm missing something obvious but upload time hasn't yet changed.

[dkelosky]

Final question - would https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.izua700/IZUHPINFO_API_PutWriteDataSet.htm with X-IBM-Data-Type be any help here? To use z/OSMF to upload a diff? We could add this into the zowe CLI if so.

[plavjanik]

Just recording problems as they occur. On gradlew build from in zowe-rest-api-sample-spring I get:

$ ./gradlew build
> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.zowe:zowe-rest-api-commons-spring:0.0.0-SNAPSHOT.
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 11s
1 actionable task: 1 executed

I understand why it is confusing. Typically, in a Gradle project with multiple subprojects, you cannot go to a subproject like zowe-rest-api-sample-spring and do ./gradlew build. You can do ./gradlew :zowe-rest-api-sample-spring:build to build a specific project at the root level.

In our case, the zowe-rest-api-sample-spring is a subproject of this repository and a standalone sample at the same time. If you do cd zowe-rest-api-sample-spring and then ./gradlew build, then you do a standalone build. This standalone build requires all the dependencies to be in a Maven repository. When zowe-rest-api-sample-spring is published then it will contain the version of the zowe-rest-api-commons-spring that is being published (e.g. 0.4.2). During a local build, it tries to find 0.0.0-SNAPSHOT that is not published to a public repository. It can be located in a local Maven repository if you publish it using ./gradlew :zowe-rest-api-commons-spring:publishToMavenLocal.

During SDK development, you usually want to issue just ./gradlew at the repository root level and let Gradle update all projects.

[plavjanik]

Built via cd to sample-spring-boot-api-service then:

• ./gradlew zowe-rest-api-commons-spring:build
• ./gradlew zowe-rest-api-sample-spring:build

zowe-api-dev deploy took 1min 35sec wall clock time on first deploy.

I changed GreetingController.java field: private static final String DEFAULT_NAME = "dan";

Rebuild via cd .. && ./gradlew zowe-rest-api-sample-spring:build.

I also had to upgrade @zowedev/zowe-api-dev to 0.4.2.

I'm sure I'm missing something obvious but upload time hasn't yet changed.

I should have written better instructions for testing.

The changes that make use of JarPatcher are not in a published @zowedev/zowe-api-dev. You need to do:

cd zowe-api-dev
npm pack
npm link

This will install the zowe-api-dev from the source code.

[plavjanik]

Final question - would https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.izua700/IZUHPINFO_API_PutWriteDataSet.htm with X-IBM-Data-Type be any help here? To use z/OSMF to upload a diff? We could add this into the zowe CLI if so.

This looks like a very powerful feature in z/OSMF. It is usable for text files so it will not help with binary JAR files but it can be useful in editing large files, for example in VS Code extension (@jellypuno, @Alexandru-Dumitru, @crawr).

When set to 'text' and "Content-Type" is "application/x-ibm-diff-e", the input consists of commands in the same format as produced by the z/OS UNIX 'diff -e' command. These commands are used to add, replace and delete lines in the target data set. The following commands are supported:

• a
• c
• d
• s/.// opt : g|, g means global n means search and replace times

Each command may be optionally preceded by a line or line range, as allowed by the z/OS UNIX 'ed' command. If an error is detected while processing a command,status code 500 is returned with an exception.

[plavjanik]

You should be seeing something like this in case of a single class change:

$ zowe-api-dev deploy                                                                                  
Patching /a/plape03/zowe-rest-api-sample-spring/bin/zowe-rest-api-sample-spring.jar to be same as build/libs/zowe-rest-api-sample-spring-0.0.1-SNAPSHOT.jar
success: true
from:    build/libs/zowe-rest-api-sample-spring-0.0.1-SNAPSHOT.jar-patch
to:      /a/plape03/zowe-rest-api-sample-spring/bin/zowe-rest-api-sample-spring.jar-patch

USS file uploaded successfully.
Executing z/OS UNIX command '/sys/java64bt/v8r0m0/usr/lpp/java/J8.0_64/bin/java  -cp /a/plape03/zowe-rest-api-sample-spring/bin/zowe-rest-api-sample-spring.jar-patch jarpatcher.JarPatcher patch /a/plape03/zowe-rest-api-sample-spring/bin/zowe-rest-api-sample-spring.jar /a/plape03/zowe-rest-api-sample-spring/bin/zowe-rest-api-sample-spring.jar-patch jarpatcher' in directory /a/plape03/zowe-rest-api-sample-spring
$ Patching: BOOT-INF/classes/org/zowe/sample/apiservice/greeting/GreetingController.class 
Ignoring: jarpatcher/ 
Ignoring: jarpatcher/JarPatcher$CompareResult.class 
Ignoring: jarpatcher/JarPatcher.class
transferFile: 8628.031ms
✔ Deployment to z/OS completed
ℹ Use 'zowe-api-dev config' to configure your application to z/OS or 'zowe-api-dev start' to start already configured application

This took 11.3 seconds while the full update took 5.6 minutes via VPN!

When nothing has changed then you should see:

$ zowe-api-dev deploy                                                                   
build/libs/zowe-rest-api-sample-spring-0.0.1-SNAPSHOT.jar has not changed
✔ Deployment to z/OS completed
ℹ Use 'zowe-api-dev config' to configure your application to z/OS or 'zowe-api-dev start' to start already configured application