As a system admin, I want reproducible zip archives (at least on the same machine with same JDK)

[epag]

---

Author Name: Jesse (Jesse) Original Redmine Issue: 89009, https://vlab.noaa.gov/redmine/issues/89009 Original Date: 2021-03-05

---

The jars seem to be reproducible (or re-used from cache) on the build server, but not the zips.

---

Redmine related issue(s): 93272

---

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-03-05T19:11:36Z

---

Estimate of 8h is to apply https://docs.gradle.org/current/userguide/common_caching_problems.html#achieving_stable_task_inputs solution and see if it mostly works:

tasks.register('createZip', Zip) {
    preserveFileTimestamps = false
    reproducibleFileOrder = true
    // ...
}
</code>

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T16:41:45Z

---

It looks like we apply these properties to jar, but not to zip:

    jar
    {
        // toward reproducible builds:
        preserveFileTimestamps = false
        reproducibleFileOrder = true

Could it be as simple as adding that to a zip block in the application?

https://docs.gradle.org/current/dsl/org.gradle.api.tasks.bundling.Zip.html#org.gradle.api.tasks.bundling.Zip:preserveFileTimestamps

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T16:53:37Z

---

Better than just the zip, from https://github.com/palantir/gradle-baseline#compalantirbaseline-reproducibility is a 4-line block to add. It works for the outer build, setting datetimes to 1980-02-01T00:00:00Z (not sure if Z or no timezone in zips). But it apparently needs to be set for each subproject? Maybe adding to the subprojects it will work for all those as well, trying it.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T16:57:47Z

---

Seems to have worked as far as setting datetimes.

The test to see if it truly works is to push the change, let the jenkins build complete, manually start another jenkins build (which deletes the zip artifacts and rebuilds them), compare the artifacts sha256sums. Doing that.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T17:08:32Z

---

Oh, that little block affects the jars as well. Removed the redundant reproducible-jar declarations. Also added the same block to systests.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T17:10:03Z

---

I expect builds 3921 and -3921- 3922 to produce zips with exactly the same sha256sums with different names if all is working (exception: systests, another build would be needed after -3921- 3922 to see if that one's working).

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T17:32:43Z

---

not identical.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T18:18:14Z

---

Ah. The version number difference is probably the only difference.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-18T18:49:16Z

---

d1ac56939cf8cf0a959a24469128cfe30a5da8b0a5081745cc02c04c84f4f7d6 *wres-20210618-99644d4.zip
d1ac56939cf8cf0a959a24469128cfe30a5da8b0a5081745cc02c04c84f4f7d6 *wres-20210618-99644d4_3923.zip

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2021-06-18T18:59:36Z

---

Nice!

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T16:48:27Z

---

Thanks. It would be nice to confirm that I can build a fresh clone from the 5.11 tag and get the same sha256sum zip, and that you on a different machine can do the same.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T16:53:09Z

---

It were not the same : (

[wres5-11]$ sha256sum build/distributions/wres-20210621-c2523ed.zip
32cdf850cda132e1336ee78cf1193edf74bfa4ef4c2cdd00521ddc33c6e76bcf  build/distributions/wres-20210621-c2523ed.zip

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T16:57:55Z

---

Where were the differences? In the jars:

$ diff 5-11_built_by_jenkins_official_build_artifacts_sha256sums.txt 5-11_built_from_tag_fresh_artifacts_sha256sums.txt
1c1
< [~]$ find wres-20210621-c2523ed -type f -exec sha256sum {} \+
---
> [distributions]$ find wres-20210621-c2523ed -type f -exec sha256sum {} \+
79,84c79,84
< e0931dc3452677321dba88184b0b03d897ea47d303db664df440312cf1421e20  wres-20210621-c2523ed/lib/wres-20210621-c2523ed.jar
< cf835191f919267cc294ca3a3455e965c2056820e092bfb5a5bc0df5775bf042  wres-20210621-c2523ed/lib/wres-io-20210621-c2523ed.jar
< 9292fbabf937080fde7968f00868ecca666b891436b26fe60db212239bc5467e  wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
< c183114585f935d98a23d1d07d15388ed335085f115a0c91b9a3aa3a244f25d2  wres-20210621-c2523ed/lib/wres-system-20210621-3b55093.jar
< 7386d4ab031d0035812ee53dc43faa6fe231b83f8f5605d1b479b13e1a0e01ac  wres-20210621-c2523ed/lib/wres-metrics-20210621-3b55093.jar
< 7640b81c7b9e884c4a23e320227b87ff99644141de2d88aa2f8f41086bc48d6d  wres-20210621-c2523ed/lib/wres-vis-20210621-3b55093.jar
---
> 28b668a0110db9c2790a028da7b8a8f25c35afdad6e775f806c89f1e29720c86  wres-20210621-c2523ed/lib/wres-20210621-c2523ed.jar
> 72f5aa6615476bd2e2510fd3c88d75c5f5c96e59d94789ef81d0f7110e5822b4  wres-20210621-c2523ed/lib/wres-io-20210621-c2523ed.jar
> 81cd60dd0cecf4e3e1bebd9063889dae7bcb8527d2a5d713431aaa76f6614c34  wres-20210621-c2523ed/lib/wres-grid-20210621-34d2c86.jar
> c137adfcc07e5fecfbfa9d1de78bb5c73490ecf466672efca8bea1e285b5d76e  wres-20210621-c2523ed/lib/wres-system-20210621-34d2c86.jar
> e28c66a5c3c73ed7222b9a44c67fd85921248b1f3b4bea2fb3dfbf2f041d5d7c  wres-20210621-c2523ed/lib/wres-metrics-20210621-34d2c86.jar
> 24b86d49db2a2b08280f620246a168c265f37f8ff492e225a95020f5e7173206  wres-20210621-c2523ed/lib/wres-vis-20210621-34d2c86.jar
88,93c88,93
< 2af1454bf4d50614159f1ce94ae1c81a6ca37439218005050e626d6982e8a0bd  wres-20210621-c2523ed/lib/wres-util-20210621-3b55093.jar
< 07cd6d71997673b930b0b26e387ae4a4cf2fbb82c8ef0aaa31667d80cdb70155  wres-20210621-c2523ed/lib/wres-datamodel-20210621-3b55093.jar
< 624b218413cc57d69206440ebeae29edd00f7f2b6a3b19717ba0a970c06e2ff5  wres-20210621-c2523ed/lib/wres-events-20210621-3b55093.jar
< 2c60fe829ef65f4b28e2c0aeb4f53f1f48ca9c62af4c4c9b7df4ece771ee406c  wres-20210621-c2523ed/lib/wres-statistics-20210621-3b55093.jar
< 2b7dce5b6f4ff708f062e1db84f71b3a2e6543b28451f1be92c5e03d7691f189  wres-20210621-c2523ed/lib/wres-config-20210621-3b55093.jar
< 81ebfa427a8185e18f6fb0ca1666a29b167f6d84cb6daa8f16f6d20d6763350a  wres-20210621-c2523ed/lib/wres-eventsbroker-20210621-3b55093.jar
---
> 10fe0db6c38d89df80013af972ce7373cfe696d8697ee6ce73fb5e2d5c3b4d67  wres-20210621-c2523ed/lib/wres-util-20210621-34d2c86.jar
> 77abfbebda776014b8a97db3602d8b02882df354301deacee00491409b9a983d  wres-20210621-c2523ed/lib/wres-datamodel-20210621-34d2c86.jar
> f4f4c0a356f7712ab8fadc8521fb23f1740150903649b57b5ee7540566fc70b7  wres-20210621-c2523ed/lib/wres-events-20210621-34d2c86.jar
> 00534ddefaf58217da17ac5e6c45da138462ecb48d33320591e1fe4931713cb9  wres-20210621-c2523ed/lib/wres-statistics-20210621-34d2c86.jar
> eec20691aecc317ab5e5eb027129765767b91ed3abb2a39cc80d005822b796da  wres-20210621-c2523ed/lib/wres-config-20210621-34d2c86.jar
> 521704ee3b02e1335b564e1cbc90766f21b4504cd3e908814019cf713a71924b  wres-20210621-c2523ed/lib/wres-eventsbroker-20210621-34d2c86.jar
237,248c237,248
< 9ca55995da5548072aaf5227f387175b1fd01cdbfafba1509c7eee8f4175143c  wres-20210621-c2523ed/bin/wres
< 03b84b4ed3ad6560bc947b54279bfa17263cc9055666e0bfca1f466bb8c273f1  wres-20210621-c2523ed/bin/wres.bat
< bb41447d591d87149301812d29f76a4a849507946fd780d24704b3a66cae7028  wres-20210621-c2523ed/src/wres-util-20210621-3b55093-sources.jar
< 2da9c8314d514c93ab4a5104e055caf44a2d2640e20842fbab1a95e3c34360ea  wres-20210621-c2523ed/src/wres-system-20210621-3b55093-sources.jar
< c7cc93e90e40c13c52c4f830006e75662d4fc537900e0a81b4c6c91cc2d2d4be  wres-20210621-c2523ed/src/wres-datamodel-20210621-3b55093-sources.jar
< eb4c184ed2087656677e51ee4932f0a2e437aade1041b51985399cb5283c1dee  wres-20210621-c2523ed/src/wres-io-20210621-c2523ed-sources.jar
< 5fd49d2a6c56ba3e712cfa6f7e1b7a57d12a9d8eaba81b740dbe465bd629673b  wres-20210621-c2523ed/src/wres-metrics-20210621-3b55093-sources.jar
< 767d96777abd79843f15b3a627d447e16812f932dd72c63419e50ca66159c89e  wres-20210621-c2523ed/src/wres-config-20210621-3b55093-sources.jar
< ac994b7c3b46505ebc1f3ac6da405838901fdd7417586a77194966fabc282884  wres-20210621-c2523ed/src/wres-vis-20210621-3b55093-sources.jar
< 5154b159c6b0db73bbc35929dd859f2832e7b2431a8170e7a93fad6a08e320b5  wres-20210621-c2523ed/src/wres-events-20210621-3b55093-sources.jar
< 988ed95a9f4b4fc8e67b42645def60cf84746418a782033fd4e2611308512520  wres-20210621-c2523ed/src/wres-eventsbroker-20210621-3b55093-sources.jar
< 1e93737a2178c76ecf662a6828315fa807df0c5b78ce4c655b1a62deeba9b0ed  wres-20210621-c2523ed/src/wres-statistics-20210621-3b55093-sources.jar
---
> 5bd0ac12002e1c4dfb656f65f01a4f687309a13efec86d1733c94362ec8d434f  wres-20210621-c2523ed/bin/wres
> 1ef416968944b6756c3b38bfee01676017dcf04331deae7dcf0eb0d2170152bd  wres-20210621-c2523ed/bin/wres.bat
> a6aa8a290a9282223456428075fa964f9588c92631e2ce14f80ad01646cb452f  wres-20210621-c2523ed/src/wres-util-20210621-34d2c86-sources.jar
> aca0c6c41804c08c3a1b0505aec76cb97f5bc7cfcfeb43eaa174e2ab675c0b2f  wres-20210621-c2523ed/src/wres-system-20210621-34d2c86-sources.jar
> 8991a540c09579c72e798e7f7fee5ff0cb2baa133181fc3201a30eba0dc356a4  wres-20210621-c2523ed/src/wres-datamodel-20210621-34d2c86-sources.jar
> cfd3080fb9d9e8b7c999e5934b0a053d538838349d054f5f9da11459b491918c  wres-20210621-c2523ed/src/wres-io-20210621-c2523ed-sources.jar
> 06f8230e362fad10fad31246762cab989b92fafc6691b97da301af06c944f7aa  wres-20210621-c2523ed/src/wres-metrics-20210621-34d2c86-sources.jar
> 6d9292feed78aff6592d193abf66094bb3945448ef3776e45e25cfe79e26a52e  wres-20210621-c2523ed/src/wres-config-20210621-34d2c86-sources.jar
> 7c17d8ad84daea4d49ecd0a36658b680b9f6ff4369f0ef9427b5a7ba63214925  wres-20210621-c2523ed/src/wres-vis-20210621-34d2c86-sources.jar
> 00b280b38dac6ffabbeb3eb04841fa89ef8f6080d403feda434f96251161d864  wres-20210621-c2523ed/src/wres-events-20210621-34d2c86-sources.jar
> 69d635ffd03eb6834c33f82ceb21dd948dba173c686570060ca04bd17c3f2e25  wres-20210621-c2523ed/src/wres-eventsbroker-20210621-34d2c86-sources.jar
> d084007b2bad2cc120638f992ae438dbb61a42e37561832ee02ee457754e444b  wres-20210621-c2523ed/src/wres-statistics-20210621-34d2c86-sources.jar

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:00:55Z

---

What's the smallest jar? Grid. Looking closer at that one.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:05:32Z

---

[distributions]$ diff wres-20210621-c2523ed/grid/ ../../../../wres-20210621-c2523ed/grid/
Common subdirectories: wres-20210621-c2523ed/grid/META-INF and ../../../../wres-20210621-c2523ed/grid/META-INF
Common subdirectories: wres-20210621-c2523ed/grid/wres and ../../../../wres-20210621-c2523ed/grid/wres

When unzipping, though, I see that the names were different.

When I cloned, I only cloned depth=5, so that changes the versions of older-than-5-deep-commits jars.

Trying again with a clone of depth 500.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:22:23Z

---

Attempt 2, with clone depth of 500:

[wres5-11-again]$ sha256sum build/distributions/wres-20210621-c2523ed.zip
0d5d2e2134a3451358bf9779407152d3b93c5fad05fd010b9fdddbbe51ac5f06  build/distributions/wres-20210621-c2523ed.zip

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:25:51Z

---

This looks more promising:

[~]$ diff wres-20210621-c2523ed code/wres5-11-again/build/distributions/wres-20210621-c2523ed
Common subdirectories: wres-20210621-c2523ed/bin and code/wres5-11-again/build/distributions/wres-20210621-c2523ed/bin
Common subdirectories: wres-20210621-c2523ed/doc and code/wres5-11-again/build/distributions/wres-20210621-c2523ed/doc
Common subdirectories: wres-20210621-c2523ed/lib and code/wres5-11-again/build/distributions/wres-20210621-c2523ed/lib
Common subdirectories: wres-20210621-c2523ed/SQL and code/wres5-11-again/build/distributions/wres-20210621-c2523ed/SQL
Common subdirectories: wres-20210621-c2523ed/src and code/wres5-11-again/build/distributions/wres-20210621-c2523ed/src
Common subdirectories: wres-20210621-c2523ed/wresproto and code/wres5-11-again/build/distributions/wres-20210621-c2523ed/wresproto

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:29:06Z

---

Hmm, not quite:

$ diff 5-11_built_by_jenkins_official_build_artifacts_sha256sums.txt 5-11_built_from_tag_with_clone_depth_500_fresh_artifacts_sha256sums.txt
1c1
< [~]$ find wres-20210621-c2523ed -type f -exec sha256sum {} \+
---
> [distributions]$ find wres-20210621-c2523ed -type f -exec sha256sum {} \+
79,84c79,84
< e0931dc3452677321dba88184b0b03d897ea47d303db664df440312cf1421e20  wres-20210621-c2523ed/lib/wres-20210621-c2523ed.jar
< cf835191f919267cc294ca3a3455e965c2056820e092bfb5a5bc0df5775bf042  wres-20210621-c2523ed/lib/wres-io-20210621-c2523ed.jar
< 9292fbabf937080fde7968f00868ecca666b891436b26fe60db212239bc5467e  wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
< c183114585f935d98a23d1d07d15388ed335085f115a0c91b9a3aa3a244f25d2  wres-20210621-c2523ed/lib/wres-system-20210621-3b55093.jar
< 7386d4ab031d0035812ee53dc43faa6fe231b83f8f5605d1b479b13e1a0e01ac  wres-20210621-c2523ed/lib/wres-metrics-20210621-3b55093.jar
< 7640b81c7b9e884c4a23e320227b87ff99644141de2d88aa2f8f41086bc48d6d  wres-20210621-c2523ed/lib/wres-vis-20210621-3b55093.jar
---
> 28b668a0110db9c2790a028da7b8a8f25c35afdad6e775f806c89f1e29720c86  wres-20210621-c2523ed/lib/wres-20210621-c2523ed.jar
> 72f5aa6615476bd2e2510fd3c88d75c5f5c96e59d94789ef81d0f7110e5822b4  wres-20210621-c2523ed/lib/wres-io-20210621-c2523ed.jar
> 89f289ea4d5aaae71a151061682ac554ce14f2630ebc1efe7c7337d3da75e42b  wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
> 4c29c0e299620d80e0f4681454cbf396d354f24c582d1978701c4832b94faedc  wres-20210621-c2523ed/lib/wres-system-20210621-3b55093.jar
> 62ba5074bb6f2ff736414eff3452f0fe1e48530dcf3c67d28d85ddafb77ce2fe  wres-20210621-c2523ed/lib/wres-metrics-20210621-3b55093.jar
> fce728474836f644848fbc60cbaed2485c225bab4635f9ae97698e16abe73eac  wres-20210621-c2523ed/lib/wres-vis-20210621-3b55093.jar
88,93c88,93
< 2af1454bf4d50614159f1ce94ae1c81a6ca37439218005050e626d6982e8a0bd  wres-20210621-c2523ed/lib/wres-util-20210621-3b55093.jar
< 07cd6d71997673b930b0b26e387ae4a4cf2fbb82c8ef0aaa31667d80cdb70155  wres-20210621-c2523ed/lib/wres-datamodel-20210621-3b55093.jar
< 624b218413cc57d69206440ebeae29edd00f7f2b6a3b19717ba0a970c06e2ff5  wres-20210621-c2523ed/lib/wres-events-20210621-3b55093.jar
< 2c60fe829ef65f4b28e2c0aeb4f53f1f48ca9c62af4c4c9b7df4ece771ee406c  wres-20210621-c2523ed/lib/wres-statistics-20210621-3b55093.jar
< 2b7dce5b6f4ff708f062e1db84f71b3a2e6543b28451f1be92c5e03d7691f189  wres-20210621-c2523ed/lib/wres-config-20210621-3b55093.jar
< 81ebfa427a8185e18f6fb0ca1666a29b167f6d84cb6daa8f16f6d20d6763350a  wres-20210621-c2523ed/lib/wres-eventsbroker-20210621-3b55093.jar
---
> bb31798ca445762bf472d5940c1a96353a7fa56e6618bd28194eb3546c4543c0  wres-20210621-c2523ed/lib/wres-util-20210621-3b55093.jar
> d078903a0867a405cc6c2021c79f5179de1ba2f6cc3478e81d1a9cf4262ceb1b  wres-20210621-c2523ed/lib/wres-datamodel-20210621-3b55093.jar
> cf4136f8a8a994b77b9be9f165905f69614e4e90a788f7be41dae12e3d04c8db  wres-20210621-c2523ed/lib/wres-events-20210621-3b55093.jar
> 7143d0e933771508b54f4e4843884f0032e1d678b11150beaad55af50d135f56  wres-20210621-c2523ed/lib/wres-statistics-20210621-3b55093.jar
> 3b0056dfbc66a2ed57f21cdcc8a4a8f8358b0251936c8506835155e91a8ec431  wres-20210621-c2523ed/lib/wres-config-20210621-3b55093.jar
> 830635b294d5740e886293351fd73eefea4b8979390a406952489cd4038a697e  wres-20210621-c2523ed/lib/wres-eventsbroker-20210621-3b55093.jar
208c208
< 2fa4c2142d460e6b1e61eab64eeff89daca218b0de8e435f9ddf2d135ae2e5ba  wres-20210621-c2523ed/wresproto/doc/README.html
---
> e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  wres-20210621-c2523ed/wresproto/doc/README.html
239,248c239,248
< bb41447d591d87149301812d29f76a4a849507946fd780d24704b3a66cae7028  wres-20210621-c2523ed/src/wres-util-20210621-3b55093-sources.jar
< 2da9c8314d514c93ab4a5104e055caf44a2d2640e20842fbab1a95e3c34360ea  wres-20210621-c2523ed/src/wres-system-20210621-3b55093-sources.jar
< c7cc93e90e40c13c52c4f830006e75662d4fc537900e0a81b4c6c91cc2d2d4be  wres-20210621-c2523ed/src/wres-datamodel-20210621-3b55093-sources.jar
< eb4c184ed2087656677e51ee4932f0a2e437aade1041b51985399cb5283c1dee  wres-20210621-c2523ed/src/wres-io-20210621-c2523ed-sources.jar
< 5fd49d2a6c56ba3e712cfa6f7e1b7a57d12a9d8eaba81b740dbe465bd629673b  wres-20210621-c2523ed/src/wres-metrics-20210621-3b55093-sources.jar
< 767d96777abd79843f15b3a627d447e16812f932dd72c63419e50ca66159c89e  wres-20210621-c2523ed/src/wres-config-20210621-3b55093-sources.jar
< ac994b7c3b46505ebc1f3ac6da405838901fdd7417586a77194966fabc282884  wres-20210621-c2523ed/src/wres-vis-20210621-3b55093-sources.jar
< 5154b159c6b0db73bbc35929dd859f2832e7b2431a8170e7a93fad6a08e320b5  wres-20210621-c2523ed/src/wres-events-20210621-3b55093-sources.jar
< 988ed95a9f4b4fc8e67b42645def60cf84746418a782033fd4e2611308512520  wres-20210621-c2523ed/src/wres-eventsbroker-20210621-3b55093-sources.jar
< 1e93737a2178c76ecf662a6828315fa807df0c5b78ce4c655b1a62deeba9b0ed  wres-20210621-c2523ed/src/wres-statistics-20210621-3b55093-sources.jar
---
> a6aa8a290a9282223456428075fa964f9588c92631e2ce14f80ad01646cb452f  wres-20210621-c2523ed/src/wres-util-20210621-3b55093-sources.jar
> aca0c6c41804c08c3a1b0505aec76cb97f5bc7cfcfeb43eaa174e2ab675c0b2f  wres-20210621-c2523ed/src/wres-system-20210621-3b55093-sources.jar
> 8991a540c09579c72e798e7f7fee5ff0cb2baa133181fc3201a30eba0dc356a4  wres-20210621-c2523ed/src/wres-datamodel-20210621-3b55093-sources.jar
> cfd3080fb9d9e8b7c999e5934b0a053d538838349d054f5f9da11459b491918c  wres-20210621-c2523ed/src/wres-io-20210621-c2523ed-sources.jar
> 06f8230e362fad10fad31246762cab989b92fafc6691b97da301af06c944f7aa  wres-20210621-c2523ed/src/wres-metrics-20210621-3b55093-sources.jar
> f9c72801a821052d1831d9c09836521acf7de7280427971444b4956eab3947a5  wres-20210621-c2523ed/src/wres-config-20210621-3b55093-sources.jar
> 7c17d8ad84daea4d49ecd0a36658b680b9f6ff4369f0ef9427b5a7ba63214925  wres-20210621-c2523ed/src/wres-vis-20210621-3b55093-sources.jar
> 00b280b38dac6ffabbeb3eb04841fa89ef8f6080d403feda434f96251161d864  wres-20210621-c2523ed/src/wres-events-20210621-3b55093-sources.jar
> 69d635ffd03eb6834c33f82ceb21dd948dba173c686570060ca04bd17c3f2e25  wres-20210621-c2523ed/src/wres-eventsbroker-20210621-3b55093-sources.jar
> d084007b2bad2cc120638f992ae438dbb61a42e37561832ee02ee457754e444b  wres-20210621-c2523ed/src/wres-statistics-20210621-3b55093-sources.jar

The version numbers aren't the difference here, though.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:32:56Z

---

Looking at the contents of the grid jar, however, it looks like it's exactly the same:

[~]$ mkdir 5-11_grid_official
[~]$ cd 5-11_grid_official/
[5-11_grid_official]$ unzip ../wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
Archive:  ../wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
   creating: META-INF/
  inflating: META-INF/MANIFEST.MF
   creating: wres/
   creating: wres/grid/
   creating: wres/grid/client/
  inflating: wres/grid/client/Fetcher.class
  inflating: wres/grid/client/GridDataRequest.class
  inflating: wres/grid/client/Request.class
  inflating: wres/grid/client/Response.class
  inflating: wres/grid/client/SingleValuedTimeSeriesResponse.class
   creating: wres/grid/reading/
  inflating: wres/grid/reading/GriddedCoordinate.class
  inflating: wres/grid/reading/GriddedPath.class
  inflating: wres/grid/reading/GriddedReader$GridFileReader.class
  inflating: wres/grid/reading/GriddedReader$GridValue.class
  inflating: wres/grid/reading/GriddedReader.class
[5-11_grid_official]$ cd ..
[~]$ mkdir 5-11_grid_built_separately
[~]$ cd 5-11_grid_built_separately/
[5-11_grid_built_separately]$ unzip ../code/wres5-11-again/build/distributions/wres-20210621-c2523ed/
bin/       doc/       lib/       SQL/       src/       wresproto/
[5-11_grid_built_separately]$ unzip ../code/wres5-11-again/build/distributions/wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
Archive:  ../code/wres5-11-again/build/distributions/wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar
   creating: META-INF/
  inflating: META-INF/MANIFEST.MF
   creating: wres/
   creating: wres/grid/
   creating: wres/grid/client/
  inflating: wres/grid/client/Fetcher.class
  inflating: wres/grid/client/GridDataRequest.class
  inflating: wres/grid/client/Request.class
  inflating: wres/grid/client/Response.class
  inflating: wres/grid/client/SingleValuedTimeSeriesResponse.class
   creating: wres/grid/reading/
  inflating: wres/grid/reading/GriddedCoordinate.class
  inflating: wres/grid/reading/GriddedPath.class
  inflating: wres/grid/reading/GriddedReader$GridFileReader.class
  inflating: wres/grid/reading/GriddedReader$GridValue.class
  inflating: wres/grid/reading/GriddedReader.class
[5-11_grid_built_separately]$ cd ..
[~]$ find 5-11_grid* -type f -exec sha256sum {} \+
0e966937da8ab03fda58a69c74ad88df96b036befeab8ac7e0b4a6dd62e9b16e  5-11_grid_built_separately/META-INF/MANIFEST.MF
3a55a5633baa840dd4a4c33bbdd65f00c1852019922bd9b290016a3a7e4fb1b2  5-11_grid_built_separately/wres/grid/client/Fetcher.class
1a64c2927cecf945d186d930f1fe53fada51dd755b33e8269e1848eef16e59ad  5-11_grid_built_separately/wres/grid/client/GridDataRequest.class
853c290df87eaa19f52f740c6ff88424e01332806b9929dd4676d21349037da2  5-11_grid_built_separately/wres/grid/client/Request.class
ef4ff2c3982a0871190cbe859d0e1e0ab37c3a3b93e5c78d0dfa89a93823fe25  5-11_grid_built_separately/wres/grid/client/Response.class
b7838d94c7075489f23933cc4e4341177f68400ef56a4f86dcb0876364f262a1  5-11_grid_built_separately/wres/grid/client/SingleValuedTimeSeriesResponse.class
24987a750eb9dbcea1b809093a743cb4e678e4d85c03321df6cca98dc6e81dc9  5-11_grid_built_separately/wres/grid/reading/GriddedCoordinate.class
3f2564f8854346f3b2b18ed2952ca4579dd758076878d6249b38474adb8cfe49  5-11_grid_built_separately/wres/grid/reading/GriddedPath.class
6ebae956f285406cdd5eb329c3f9e97dc4a3f2533c473dea43cae1f68c5917f0  5-11_grid_built_separately/wres/grid/reading/GriddedReader$GridFileReader.class
1b1cbd2086556ec000a786fd1fe7aa4ec138741348de59cfdbf57a99402bcfc3  5-11_grid_built_separately/wres/grid/reading/GriddedReader$GridValue.class
63ba22af8e9133b91c38971f3d44f1a751cf96b303b34e2329a0f272f9e84574  5-11_grid_built_separately/wres/grid/reading/GriddedReader.class
0e966937da8ab03fda58a69c74ad88df96b036befeab8ac7e0b4a6dd62e9b16e  5-11_grid_official/META-INF/MANIFEST.MF
3a55a5633baa840dd4a4c33bbdd65f00c1852019922bd9b290016a3a7e4fb1b2  5-11_grid_official/wres/grid/client/Fetcher.class
1a64c2927cecf945d186d930f1fe53fada51dd755b33e8269e1848eef16e59ad  5-11_grid_official/wres/grid/client/GridDataRequest.class
853c290df87eaa19f52f740c6ff88424e01332806b9929dd4676d21349037da2  5-11_grid_official/wres/grid/client/Request.class
ef4ff2c3982a0871190cbe859d0e1e0ab37c3a3b93e5c78d0dfa89a93823fe25  5-11_grid_official/wres/grid/client/Response.class
b7838d94c7075489f23933cc4e4341177f68400ef56a4f86dcb0876364f262a1  5-11_grid_official/wres/grid/client/SingleValuedTimeSeriesResponse.class
24987a750eb9dbcea1b809093a743cb4e678e4d85c03321df6cca98dc6e81dc9  5-11_grid_official/wres/grid/reading/GriddedCoordinate.class
3f2564f8854346f3b2b18ed2952ca4579dd758076878d6249b38474adb8cfe49  5-11_grid_official/wres/grid/reading/GriddedPath.class
6ebae956f285406cdd5eb329c3f9e97dc4a3f2533c473dea43cae1f68c5917f0  5-11_grid_official/wres/grid/reading/GriddedReader$GridFileReader.class
1b1cbd2086556ec000a786fd1fe7aa4ec138741348de59cfdbf57a99402bcfc3  5-11_grid_official/wres/grid/reading/GriddedReader$GridValue.class
63ba22af8e9133b91c38971f3d44f1a751cf96b303b34e2329a0f272f9e84574  5-11_grid_official/wres/grid/reading/GriddedReader.class

I wonder if the vendor of the jdk ends up in a jar header or something.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:35:27Z

---

Not in the first 10 lines of xxd output, but a diff in the last 10 lines of xxd output:

[~]$ diff <( xxd wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar | head ) <( xxd code/wres5-11-again/build/distributions/wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar | head )
[~]$ diff <( xxd wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar | tail ) <( xxd code/wres5-11-again/build/distributions/wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar | tail )
6c6
< 0003780: 0000 0000 a481 4422 0000 7772 6573 2f67  ......D"..wres/g
---
> 0003780: 0000 0000 b481 4422 0000 7772 6573 2f67  ......D"..wres/g

a4 vs b4 there in byte 0003784

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:37:26Z

---

Similar one-bit differences in several places in the jar:

[~]$ diff <( xxd wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar ) <( xxd code/wres5-11-again/build/distributions/wres-20210621-c2523ed/lib/wres-grid-20210621-3b55093.jar )
826c826
< 0003390: 0000 0000 0010 00ed 41c0 0000 0077 7265  ........A....wre
---
> 0003390: 0000 0000 0010 00fd 41c0 0000 0077 7265  ........A....wre
829c829
< 00033c0: 0000 0000 0000 0000 1000 ed41 e500 0000  ...........A....
---
> 00033c0: 0000 0000 0000 0000 1000 fd41 e500 0000  ...........A....
833c833
< 0003400: 1000 ed41 0f01 0000 7772 6573 2f67 7269  ...A....wres/gri
---
> 0003400: 1000 fd41 0f01 0000 7772 6573 2f67 7269  ...A....wres/gri
837c837
< 0003440: 00a4 8140 0100 0077 7265 732f 6772 6964  ...@...wres/grid
---
> 0003440: 00b4 8140 0100 0077 7265 732f 6772 6964  ...@...wres/grid
841c841
< 0003480: 0026 0000 0000 0000 0000 0000 00a4 81b3  .&..............
---
> 0003480: 0026 0000 0000 0000 0000 0000 00b4 81b3  .&..............
847c847
< 00034e0: 00a4 8185 0c00 0077 7265 732f 6772 6964  .......wres/grid
---
> 00034e0: 00b4 8185 0c00 0077 7265 732f 6772 6964  .......wres/grid
851c851
< 0003520: 001f 0000 0000 0000 0000 0000 00a4 81e4  ................
---
> 0003520: 001f 0000 0000 0000 0000 0000 00b4 81e4  ................
856c856
< 0003570: 0000 0000 0000 0000 0000 a481 250f 0000  ............%...
---
> 0003570: 0000 0000 0000 0000 0000 b481 250f 0000  ............%...
862c862
< 00035d0: 0012 0000 0000 0000 0000 0010 00ed 416d  ..............Am
---
> 00035d0: 0012 0000 0000 0000 0000 0010 00fd 416d  ..............Am
866c866
< 0003610: 0029 0000 0000 0000 0000 0000 00a4 819f  .)..............
---
> 0003610: 0029 0000 0000 0000 0000 0000 00b4 819f  .)..............
872c872
< 0003670: 0000 0000 a481 d414 0000 7772 6573 2f67  ..........wres/g
---
> 0003670: 0000 0000 b481 d414 0000 7772 6573 2f67  ..........wres/g
877c877
< 00036c0: 0000 0000 00a4 81df 1600 0077 7265 732f  ...........wres/
---
> 00036c0: 0000 0000 00b4 81df 1600 0077 7265 732f  ...........wres/
883c883
< 0003720: 0000 0000 0000 00a4 81e1 1f00 0077 7265  .............wre
---
> 0003720: 0000 0000 0000 00b4 81e1 1f00 0077 7265  .............wre
889c889
< 0003780: 0000 0000 a481 4422 0000 7772 6573 2f67  ......D"..wres/g
---
> 0003780: 0000 0000 b481 4422 0000 7772 6573 2f67  ......D"..wres/g

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:40:46Z

---

One more thing to try is build it with zulu, see if it's a vendor difference.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:45:05Z

---

Gotta give up for now and move the target to mark this closed.

[wres5-11-again]$ scripts/run_with_java_11.sh "./gradlew clean distZip"
Found existing zulu java at zulu11.48.21-ca-jdk11.0.11-linux_x64
The following should indicate 11 somewhere...
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Zulu11.48+21-CA (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.48+21-CA (build 11.0.11+9-LTS, mixed mode)
Running command ./gradlew clean distZip with java 11, assuming it respects JAVA_HOME...
...
BUILD SUCCESSFUL in 41s
92 actionable tasks: 80 executed, 12 up-to-date
[wres5-11-again]$ sha256sum build/distributions/wres-20210621-c2523ed.zip
4d66ab4c1cc962f5aeb7db45afb39a330b981c0df14a03414276d8e41c7ab165  build/distributions/wres-20210621-c2523ed.zip
[wres5-11-again]$

Maybe an OS difference or something, some native facility used to write the jars? IDK, gotta save it for another day.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2021-06-23T17:47:23Z

---

I wonder if this is a system setting that gradle is somehow picking up on and propagating. I see you have the main things covered w/r to reproducibility, those being timestamp and file order.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:47:36Z

---

Sigh. Even if I run the exact same command with exact same jdk, it doesn't produce the same zip from clean build:

[wres5-11-again]$ sha256sum build/distributions/wres-20210621-c2523ed.zip
229f04f38ed780dcd3a3c7e3a43b864330cd4fb71a21bceda68db9b69da617af  build/distributions/wres-20210621-c2523ed.zip

So putting this back in the backlog.

[epag]

---

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-06-23T17:49:04Z

---

James wrote:

I wonder if this is a system setting that gradle is somehow picking up on and propagating. I see you have the main things covered w/r to reproducibility, those being timestamp and file order.

You're welcome to try it yourself locally. Do a @gradlew clean distZip@ then save that resulting zip somewhere, and repeat. Diff the zips and find the difference.

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2021-06-23T17:50:37Z

---

Jesse wrote:

James wrote:

I wonder if this is a system setting that gradle is somehow picking up on and propagating. I see you have the main things covered w/r to reproducibility, those being timestamp and file order.

You're welcome to try it yourself locally. Do a @gradlew clean distZip@ then save that resulting zip somewhere, and repeat. Diff the zips and find the difference.

Yes, I probably will on a slow day, but I sense an inviting rabbit hole. :)

[epag]

---

Original Redmine Comment Author Name: James (James) Original Date: 2021-06-23T17:59:47Z

---

Second comment is interesting:

https://stackoverflow.com/questions/43993329/why-does-the-same-jar-file-have-different-hash-every-time-i-build-it

make sure the umask is the same on both systems you build (apparently gradle includes the file attributes inside the war/jar files, and I had different SHAs when those attributes were different

Anyway, I'm already dragging myself in. Nope.