Closed Joe-Winchester closed 2 years ago
There is already the provision in Zowe to verify that a downloaded file is an original one.
Files and their keys are in Jfrog https://zowe.jfrog.io/zowe/webapp/#/artifacts/browse/tree/General/libs-release-local/org/zowe/1.10.0
Instructions for their use are here on the Zowe download site: https://d1xozlojgf8voe.cloudfront.net/post_download.html?version=1.10.0 This is documented in the user guide; step beginning … “Verify the integrity of the PAX file”.
However, the post-download doc from cloudfront.net does not describe how to verify the driver authenticity for any files except the convenience build .PAX file. I've raised issue https://github.com/zowe/docs-site/issues/1126 for this.
Support staff will want to verify that the user's runtime directory content is unmodified from the official, released version. The first effort is to verify the USS runtime - verification of the shipped MVS runtime datasets will come later.
One solution is to create a key based on the contents of the USS runtime created during zowe-install-test
. The key should ignore data about the file permissions, ownership, date, and any other attributes apart from file name and contents. The key will be published on JFrog like the current ones.
The support person will download the key for the customer's release, and run a script to walk the customer's runtime directory tree and run sha256sum
on the aggregated files. Then use gpg --verify ...
to verify the aggregated file's signature. A further, simple check will verify that the directories and file names are correct.
Program md5deep looks promising http://md5deep.sourceforge.net/ It's for Windows but they say you can compile it on Cygwin, Linux, FreeBSD, OpenBSD, Mac OS X, OpenSolaris, HP/UX, etc.
Suggested solutions
Phase 1 – compare filenames and file sizes Issues:
SMP/E runtime has an extra SMPE directory with 10 PAX files in it.
Runtime directory can have javacore, heap and snap dumps in it.
These can be excluded from the list to be compared.
Phase 2 – real checksum of all files Method: Step 1 – create a checksum for each file, and aggregate them into a single file Step 2 – create a checksum of the aggregate
Issues:
The hash file could be stored in the runtime directory itself, thus rendering the hash check self-contained.
A script e.g. zowe-verify-runtime-integrity.sh
could be provided to L2/L3. The script's sole parameter would be the runtime pathname.
@Joe-Winchester Phase 2 – real checksum of all files Generating checksums takes some time. Current hash set of zowe runtime will take an estimated 100 minutes to generate.
Example on a real z/OS system
real 40m49.03s
user 21m39.56s
sys 7m13.19s
An improved approach reduced the total time to generate the local hash set to 17 seconds.
Adapted from slack with @jackjia-ibm
The checksum generation is better to stay in zowe-install-packaging build, not zowe-install-test. The zowe-install-test branch may not match the zowe build you are testing. For example, you can use zowe-install-test staging branch to test a zowe build generated by a developer’s own branch.
we actually have expanded the full file system tree with installation when we build SMPE
i’m not sure if that can be the best place to generate checksums
we definitely will build SMPE for each RC and formal releases, so this script will be executed, and can be easily associated so the zowe build matches the checksum
if we build checksums along with zowe-install-packaging, we can also promote the checksum file and put into https://zowe.jfrog.io/zowe/webapp/#/artifacts/browse/tree/General/libs-release-local/org/zowe/1.11.0-RC1 folder
Current proposal.
The zowe-install-packaging
SMP/E build will run zowe-checksum-runtime.sh
on the runtime directory it has just created. This will generate a RefRuntimeHash.txt
file (and the hash program executable) which will be published in JFrog artifactory. The customer can download this RefRuntimeHash.txt
file and verify their own installed runtime directory by running zowe-verify-authenticity.sh
.
The zowe-install-packaging
SMP/E build will also run zowe-verify-authenticity.sh
to check itself.
This test run of those changes was successful https://wash.zowe.org:8443/job/zowe-install-packaging/view/change-requests/job/PR-1316/28/consoleFull Search for
zowe-checksum-runtime.sh started
and
[consumer_2] Deploying artifact: https://zowe.jfrog.io/zowe/libs-snapshot-local/org/zowe/1.11.0-PR-1316/RefRuntimeHash-1.11.0-pr-1316-28-20200505111936.txt
[consumer_2] Deploying artifact: https://zowe.jfrog.io/zowe/libs-snapshot-local/org/zowe/1.11.0-PR-1316/HashFiles-1.11.0-pr-1316-28-20200505111936.class
@John-A-Davies I saw you also publish HashFiles
to Artifactory. I think it makes sense if this file changes a lot, might be different from build to build, release to release. Otherwise, if it's quite stable, storing in repository could be a better place. For example, we stored signing keys in repository https://github.com/zowe/zowe-install-packaging/tree/master/signing_keys because they don't change.
@jackjia-ibm Yes, it's as stable as the rest of the source files. The RefRuntimeHash.txt
file must change with each release, but HashFiles.*
probably won't. I'm happy to follow your guidance on how best to store these items.
We also need to store the RefRuntimeHash.txt
file for each release and make them all available for download, as well as versions of that file for releases prior to v1.12.0
Feedback from my presentation of this topic to the Zowe team
Parameters are better when they are keyword-value pairs, e.g. script.sh -a abc -b def
rather than positional, e.g. script.sh abc def
Could use java to download the RefRuntimeHash.txt
file
Ideally, zowe-verify-authenticity.sh
would need no parameters
Optimise the user experience for >= v1.12.0 at the expense of prior releases
Is SMP/E the only alternative build that creates /usr/lpp/zowe
?
A self-contained version of zowe-verify-authenticity.sh
Either
• Dowload the file RefRuntimeHash.txt with curl, or
• The file RefRuntimeHash.txt could be kept in a separate directory, e.g. /tmp/usr/lpp/zowe/fingerprint
, which is excluded from the hash process
Proposed new parm list format:
zowe-verify-authenticity.sh [-r <runtime-dir>] [-h <HashFiles-dir>] [-t <output-dir>]
-r runtime directory path. Defaults to CWD -h hash path, where HashFiles.class and RefRuntimeHash.txt reside. Defaults to -r/fingerprint -t directory for output files. Defaults to /tmp
For example:
cd downloads/hash
zowe-verify-authenticity.sh -r /tmp/usr/lpp/zowe -h . -t ~/hash
Or you can just type
cd /tmp/usr/lpp/zowe/bin
zowe-verify-authenticity.sh
with no parameters. The zowe-support.sh
script could run it like this.
This check could be falsified, but L2 can download these files to downloads/hash
zowe-verify-authenticity.sh
HashFiles.class
RefRuntimeHash.txt
and run
cd downloads/hash
zowe-verify-authenticity.sh -r /tmp/usr/lpp/zowe -h .
-r parameter shouldn't default to CWD - this script is in <runtime>/<sub-path>/zowe-verify-authenticity.sh
, so we should work out the runtime directory by using dirname $0
-t shouldn't default to /tmp - at least we should default to ${TMPDIR:-/tmp}
as some customers have re-defined TMPDIR to point elsewhere. However I think there is an argument that the output should go to the log directory so that it can be picked up by the support script.
<ROOT_DIR>/bin/zowe-verify-authenticity.sh
should run without any args
points to
<ROOT_DIR>/bin/internal/HashFiles.class
The HashFiles.txt goes into
<ROOT_DIR>/fingerprint/RefRuntimeHash_1.12.0.txt
For out directory write to -l
argument that defaults similar to zowe-install.sh -i
defaulting and uses a new directory, e.g. /fingerprint
see the script setup-log-dir.sh
for defaulting as this is used by other creation scripts, see...
https://docs.zowe.org/stable/user-guide/install-zowe-zos-convenience-build.html#step-3-choose-a-dataset-hlq-for-the-samplib-and-loadlib
Can this be closed ?
Can this be closed ?
No - it's not done
@nkocsis This issue will be closed when the pull request https://github.com/zowe/zowe-install-packaging/pull/1316 with the code gets reviewed, approved and merged. It was demo'd on a few calls recently to the community, but there is work to do be done to get it into 1.12 and also the have it documented. Once the pull request https://github.com/zowe/zowe-install-packaging/pull/1316 that contains the code is closed it gets associated with the issue which will be closed automatically. At the same time the CHANGELOG.md file for the release will be updated to reference that the feature is included in a release, which we're hoping will be 1.12.
Is your feature request related to a problem? Please describe. The Zowe release pipeline creates builds, e.g. 1.5, 1.9, 1....
1.9 represents the Zowe SMP/E FMID AZWE001 - the A letter was chosen to make it vendor neutral - other letters are reserved for specific IBM z/OS vendors. The Zowe drivers can be redistributed as-is by vendors offering enterprise support for Zowe through their own channels. These are enterprise "trusted" SMP/E receive order sites. The Zowe SMP/E FMID installs into the USS location /usr/lpp/zowe. The design is for a single destination which offers beneifts to users and extenders that a single desktop and single APIML can have app to app communication and single sign on and MFA ... across all products that extend Zowe from all vendors, whether from the original vendor that delivered the SMP/E package or not. Traditionally a vendor offering support may only do this if the software they are supporting was obtained by the customer through their channel.
Describe the solution you'd like Provide assistance for a vendor support team to be able to trust a Zowe driver irrespective of where it came from, so the source is based on a customer preference for who their trusted partner for support is but they may choose to install extensions from other vendors into the same base (CLI, APIML, Desktop).
Additional information Node.js has already solved this. This section is pasted from the README.MD that is delivered with Node.js.