Cray module RPMs should work w zypper: finish src changes (chapel-code), merge to master

[awallace-cray]

Finish https://github.com/chapel-lang/chapel/issues/9253 Cray XC RPM for ARM should work with zypper not rpm

Outcome of this task: All Chapel Cray-module RPMs and YAMLs generated in Jenkins will use a new internal RPM Name and Version convention which satisfies the new PE-installer's Name uniqueness requirement (RPM's for ARM) yet remains compatible with the old PE-installer (RPMs for x86_64); including existing RPMs already installed in the field.

Note, all source changes, git branches, etc discussed here apply to Cray's internal chapel-code.git repository, not GitHub's chapel.git.

Tasks to be completed:

• Script changes for Jenkins cray-module build jobs, still exist only on private branch gh9321. These need to be refactored out and merged to master (along with the new src changes below)

  
  # Change RPM Name and RPM Version to comply with zypper uniqueness requirement
  # Change RPM Name and RPM Version to comply with zypper, part 2
  git diff --numstat e9bd4952e3514b1e4181ca7af67b0ab3fd3e1ec0~ 0393ad005a571b1700969bb60523351d7ce56c3f
  3       2       build/common.bash
  18      23      build/cray-module/chapel.bash
  29      28      build/cray-module/chapel.spec
  8       3       build/cray-module/common.bash

Replace all hard-coded "x86_64" with "$CPU" in scripts

git diff --numstat 903747695144898e4cdd63beabd88da1e2cf032f~ 903747695144898e4cdd63beabd88da1e2cf032f 1 1 build/cray-module/build_chapel_flavors.bash # does not count 6 6 build/cray-module/chapel.bash 2 2 build/cray-module/common.bash 2 2 build/cray-module/update_dropoff.bash

* Script changes for Jenkins pe-dropoff jobs have not been developed at all. Up to now I have been creating YAML's and copying files into dropoff by hand. This is an opportunity to substantially rewrite (and simplify) the `update_dropoff.bash` script (234 lines) and parts of `common.bash` (167 lines).  
  * Stop doing the test rpm-install. This activity no longer has any meaning since the real RPM databases that could actually predict PE-install failures, are no longer visible except on the SMW (which is unavailable to Jenkins).  Also, the Jenkins chapel cray module build already runs an rpm-install against a "test" RPM database. 
  * Stop trying to generate YAML files using the ugly PE code from 2012-2014. Instead, check-in some YAML templates (nightly v. release)x(x86 v. ARM), then use sed or bash to parameterize dates and versions in the text. The handbuilt YAML files I have been testing can become these templates. 
  * Get the set_default template from Stash instead of SVN. (email from RCW)
  * Apply the new Chapel RPM Name and Version conventions which will make Chapel RPMs compatible with both old and new PE installer tools.  See https://github.com/awallace-cray/priv/issues/9#issuecomment-389282438.

These script changes need to be tested, merged to master, and finally refactored out of gh9321. 

These source changes were not included in task https://github.com/chapel-lang/chapel/issues/9243 because that task was for chapel.git repository. 

Source changes made in rev 903747695144898e4cdd63beabd88da1e2cf032f (above) were originally meant to be included in ttps://github.com/chapel-lang/chapel/issues/9244, but need to be done with this task. 

[awallace-cray]

2018-05-16 : chapel-code.git : new branches

• gh9533 : this task, limited to x86_64 RPMs
  • git checkout gh9321; git checkout -b gh9355; git rebase -i master # edit out unwanted commits. resolve conflicts.
• gh9533+9321
  • git checkout gh9321; git checkout -b gh9533+9321; git rebase -i gh9533 # edit out the other set of commits. There were no conflicts.

Old branch gh9321 is not changed. It is now basically the same as branch gh9533+9321 except

• the more elaborate chpl-make-max-cpu-count change from master replaces the earlier, simple, hard-coded change

  -num_procs=$(python -c 'import multiprocessing; print(min(24, multiprocessing.cpu_count()))')
  +num_procs=$($CHPL_HOME/util/buildRelease/chpl-make-cpu_count)
  etc

• other peoples recent commits on master are also onboard

[awallace-cray]

2018-05-29 Successful module builds in sandbox Jenkins with current Chapel master branch + chapel-code branch gh9533. TODOs:

• Collect and compare before + after RPMs (Collected OK, file transfers and compares to go)
• Complete commit/PR comments. Commits and eventual PR(s) will need more documentation for review. (50% done)
• Create PR(s) from branch gh9533.
• Review/approve.

[awallace-cray]

Incomplete. Continued in https://github.com/chapel-lang/chapel/issues/9642.

[awallace-cray]

2018-05-29 Post-mortem: I under-estimated this work. Contributing factors:

• As planned, this task was used as an opportunity to make some long-overdue structural changes, not directly related to the "work with zypper" topic- especially in PE-dropoff. See https://github.com/chapel-lang/chapel/issues/9533#issue-323299003 above). However, I did not recognize the size of the task.
• Up to now, all Chapel Cray-modules for ARM, and all previous Chapel Cray-modules with the zypper-compatibility modifications, have been built and tested in "sandbox", which means these RPM and YAML files are kept in separate directories, installed and tested in separate areas, etc. When these changes are merged to production, however, these new types of RPMs and YAMLs will need to peacefully co-exist in some of the same folders with the old (current) types; the scripts will need to be able to correctly identify and work with either type when they appear in an existing RPM database or file folder.
  For example, several Jenkins scripts identify Chapel RPM files in locations like "PE Dropoff" by their filenames; using "ls" with glob patterns, "grep" and "sed" on the resulting expanded filenames, and so on. Up to now, all Chapel RPM files ended with "x86_64". and these globs and regex's could just accept anything ("*") after a certain point in the filename. After this change, the filenames may end with "aarch64" or "x86_64", and those globs and regex's had to be reexamined. I planned to make the necessary improvements as part of this task, but did not recognize the size of this part of it.