Enhance build to include commit count, SHA, and modified in version string

[srorso]

When building Hercules in a Windows environment from a Git repository, the Version string includes the count of commits included in the build, the SHA of the last commit, and the word "modified" if the build includes changes beyond the last commit, in the form:

v.m.dot.count-Gxxxxxxx-modified (e.g., 4.0.0.8473-gfe2c24f)

It would be helpful to include the same information for non-windows builds.

It would be ideal if the implementation of this enhancement did not require a re-run of autogen.sh or configure to pick up updates to the version string. The version string value should be updated by something done in make prior to the first c compiler invocation.

[Fish-Git]

I've managed to figure out how to hack configure.ac to construct a VERSION string identical to the one that makefile.bat constructs for Windows builds (which works great), but unfortunately it of course is only invoked (and thus the VERSION string only changes) whenever you do a ./configure.

I'm currently researching how to move the logic into the makefile, so that it gets invoked each time you do a make (which is our goal).

I think I just need to move the logic into makefile.am somehow, but am not sure about that (nor how precisely to do it either).

I still need to do more research.

[srorso]

Way cool! Are you using a Linux Git tool, and if so, which one(s)?

[Fish-Git]

Well, I've managed to figure it out.

The technique works the same on both Windows and non-Windows but I'm not 100% confident as to whether it is the proper technique or how portable it is (and thus how well received it will be by my fellow developers and the Hercules community at large).

It involves adding a single statement to makefile.am which sets a dummy new variable to $(shell <command>), where command is the name of a bash script I wrote that (here's where it gets a bit kludgy) creates or updates a new dynamically generated .h header file which our existing version.h header file then always #includes.

In my first several dozen attempts I tried like heck to just pass the define on the compiler command line (via several -D options), but I just couldn't manage to pass the define for VERSION correctly since it was a string define. Something (libtool?) kept messing with the quotes.

I'll be committing my changes as soon as I finish typing this. If people don't like it or it causes problems we can always back it out. I have no problem with that.

I invite my fellow developers as well as the rest of the Hercules community at large to please provide feedback as to whether or not it works for you, how well it works, what you think of it and whether you know of a better way to do it. Thanks.

Here goes... Everyone hang onto your butts...

[Fish-Git]

I am re-opening this issue and setting the "More work remains" label because I've discovered a minor glitch regarding the technique (git command) used on both Windows and Nix to detect whether any modifications (uncommitted changes) exist or not which merits further research and eventual resolution.

It seems current techniques used to detect uncommitted changes on both Windows and Nix (which are currently different from another but shouldn't be) does not always work as expected. That is to say, they each report inaccurate information (report that modifications exist when there aren't and vice-versa).

During the effort to look into this issue to try and find out what's going on I have developed two small test scripts (one for Windows the other for Nix) to both illustrate the problem and help in determining why it occurs. The two test scripts are called gitmodtest.cmd and gitmodtest both available for download from my gist on GitHub:

• gitmodtest (Nix)
• gitmodtest.cmd (Windows)

Please download them and give them a try and report both your findings and thoughts as a subsequent comment in this github issue. Thanks.

---

Their usage is quite simple: place the script in the parent directory of your "hercules-0" directory (I hard-coded "hercules-0" as the hyperion clone working directory name; either change each script appropriately or, even easier, just temporarily rename your hyperion directory to "hercules-0") and then enter the command gitmodtest without any arguments.

It will iterate through each of the four different techniques I found during my research (feel free to add other techniques if you know of any!) and displays the results from each.

Alternatively, you can also optionally specify as command line arguments which specific techniques you wish to test and the order you want them run in. I did this since during my research it appeared that a given technique was perhaps interfering with the results of a subsequent technique. The ability to specify the order that each technique should be tried eliminates this possibility.

My own testing has revealed (your own results may differ!) that the current Windows technique FAILS to accurately detect a clean (unmodified) or dirty (modified) state the first time it is run, and the current Nix technique always seems to fail!

For example: starting with a clean, virgin clone, both techniques 1 and 2 (our existing "current" techniques) behave as follows:

1. gitmodtest reports unmodified (which is correct).
2. modify a file, any file.
3. run gitmodtest immediately again. This time it reports modified (which is again correct)
4. undo you changes.
5. Immediately run gitmodtest again. This time it still says "modified!" (which is wrong!)
6. Now run gitmodtest again (a second time) and notice the results!

When I run this on Windows, step 5 above (the first run of the Windows technique 1) reports "modified" at first, but then turns right around and reports "UNmodified" (i.e. clean) in step 6! (the second time it is run).

The same phenomenon occurs (for technique 1) when it is run on Nix as well, so it does not appear to be a Windows issue.

So it would seem technique 1 (our current technique on Windows) shouldn't be used.

Furthermore, technique 2 (the technique we're currently using on Nix) fails to detect any modifications at all on both platforms! (i.e. when run on either Windows or Nix).

So it would appear technique 2 shouldn't be used either.

Technique 3 just doesn't work at all under any situation. It's a technique I found on stackoverflow and thought I'd give a try.

Technique 4 however, appears to be the only technique out of the four that seems to work reliable the first time and every time it is run -- on both platforms. It is the new technique that I'm proposing we use in our new _dynamic_version build script.

You might want to also try adding a new file to the repository (git add foobar (but don't commit it!)) and then undoing the add (git reset foobar) too, during your testing. Technique 4 once again is the only one out of the four that seems to properly detect such changes first and every time.

---

Well, that's it. Until this mystery is resolved I don't feel comfortable marking this issue as having been completed yet. Something weird is going on and I want to get to the bottom of it first.

Thanks for reading this far and thanks for running your tests and for providing any insight and/or feedback regarding this mysterious behavior.

[dasdman]

Running the script with debug turned on, the problem with technique_2 is obvious:

++ echo technique_2 = Current Linux technique: git status ... --porcelain
technique_2 = Current Linux technique: git status ... --porcelain
++ git status --untracked-files=no --porcelain
++ test 0 == 1
++ modified_str=
++ echo '    modified_str = '
    modified_str =
+ for i in '$(seq 1 $num_techniques)'
+ modified_str=

The value of $? is zero regardless of whether or not there are modified files (the trace is the same with or without modified files). However, it is possible to modify lines 32-34 to read:

  result=$(git status --untracked-files=no --porcelain 2>/dev/null | wc -c) 

  if test "$result" != "0"; then

and technique_2 will then work properly.

[srorso]

Windows 10 trials

Starting with a clean, virgin clone,:

1. gitmodtest reports unmodified for all four techniques
2. modify _build.bat to remove first "@"
3. gitmodtest reports modified for 1, 4; unmodified for 2, 3
4. run gitmodtest again: modified for 1, 4; unmodified for 2, 3 (same as above)
5. run gitmodtest again: modified for 1, 4; unmodified for 2, 3 (same as above)
6. run "gitmodtest 2 3 4 1" modified for 1, 4; unmodified for 2, 3 (same as above)
7. undo change to _build.bat by restoring the first "@" (note: I did not restore from a backup file.)
8. gitmodtest reports modified for 1; unmodified for 2, 3, 4
9. run gitmodtest again: unmodified for 1, 2, 3, 4

[Fish-Git]

@srorso Thank you. That is the same behavior I'm seeing on my Windows 7 system.

@dasdman Thank you! I have applied your fix to technique 2.

ALL: my gist has been updated. Please re-download and retry.

• gitmodtest (Nix)
• gitmodtest.cmd (Windows)

STILL REMAINING: choosing which technique to use on both systems. (I want to use the same technique on each). Register your vote for which one you feel is best.

Thanks!

[Fish-Git]

I think I know what's going on and how to fix it. TortoiseGit gave me the answer.

I happened to notice that technique 1 -- the current Windows technique, which would usually fail (say modifications exist when there really weren't and vice-versa) -- would always fix itself whenever I used TortoiseGit to "Check for modifications".

So I downloaded TortoiseGit's source code and tooked at it to see what it was doing.

The very first thing it does when you ask it to "Check for modifications" is call a function called RefreshGitIndex which simply does a git update-index --refresh. That, I believe, is the fix. I think all we need to do is refresh the index immediately before doing our existing git diff-index --quiet HEAD.

I also found another function called CheckCleanWorkTree which performs a series of tests and immediately returns FALSE (work tree not clean) if any of them return with a non-zero return code, or TRUE if all tests succeed (return rc=0).

I've added each of them (including update-index --refresh too!) as new techniques to gitmodtest. Now when I do gitmodtest 6 1 it always reports an accurate modified state.

If you're interested you can download the updated gist and try it for yourself:

• gitmodtest (Nix)
• gitmodtest.cmd (Windows)

If no one objects or provides any feedback to the contrary in the next few days I'll go ahead and commit the fix.

Thanks!

[srorso]

Hi Fish:

Tried the tests again:

Virgin Clone (sounds like racy sci-fi): 1-3 and 5-9 reported no modifications. Technique four could not be found "The system cannot find the batch label specified - technique_4"

Removed "@" from the first line of _build.bat: 1-2 and 5-8 reported "-modified" Three and nine reported no modifications. Four still missing.

Re-ran the script:same results.

Ran the script with 3 9 1 2 5 6 7 8: same results in a different order.

Restored the "@" to the first line of _build.bat using an editor: Test one still reports modified, test four still missing, and the remainder report unmodified.

Reran the script: Tests 1-3 and 5-9 report unmodified; test 4 still missing [ :-) ]

So where did test four go?

As downloaded, gitmodtest.cmd has NL rather than CRLF line ends. I ran it through the following command:

type gitmodtest.cmd | find /v "" > gitmodtest1.cmd

so I could look at it. No visible issues. Ran gitmodtest1.cmd, and it worked. Re-ran the original, four is still missing. Go figure....

For what it's worth, technique four correctly reported the status of the repository...

Best Regards, Steve Orso

[Fish-Git]

Apparently GitHub always uses Unix LF-only line-endings for gists. I've sent them a message asking them for Help. We'll see how they respond.

In the mean time, after downloading the gist (I'm presuming you're using the "Download ZIP" link) and unzipping it, you'll have to run the gitmodtest.cmd file through the unix2dos utility first before trying to use it. (Or, if you don't have the unix2dos or dos2unix tools on your system, just open the file with the wordpad tool that still ships with Widows, and do a "Save". Wordpad always saves text files using Windows CRLF line endings.)

As far as the missing technique_4 goes, I cannot explain it other than you may have been unlucky enough to have downloaded a bad version I had uploaded before I had a chance to fix it. (I did upload a bad version at one time but I quickly fixed it and re-uploaded it a few minutes later.)

Try downloading it again. technique 4 does exist in both scripts! I promise you!

[Fish-Git]

FYI: wordpad can be found under "Accessories" in your Start -> All Programs menu.

[Fish-Git]

My own testing shows 6 followed by 1 seem to work quite reliably on both Windows and Linux, so that's the change I'm going to make (if no one objects).

[srorso]

On May 14, 2016, at 10:53 AM, Fish-Git notifications@github.com wrote:

Apparently GitHub always uses Unix LF-only line-endings for gists. I've sent them a message asking them for Help. We'll see how they respond.

I think a .gitdefaults file could be added to force CRLF for .cmd....could be wrong though.

(I'm presuming you're using the "Download ZIP" link) and unzipping it, you'll have to run the gitmodtest.cmd file through the unix2dos utility...

type fname | find /v "" > outfname

does the LF -> CRLF conversion too, and that's what I did.

Try downloading it again. technique 4 does exist in both scripts! I promise you!

Oh yes, it is. When I got the message, I edited the script in notepad, saw LF, did the type thing above, and ran again. No message; test four ran.

Interesting that Windows would object to LF ending only for test four.

Apologies for not being clear in my earlier post about this; I could have saved you some angst.

Best regards, Steve Orso

+1 610 217 7050

[dasdman]

On 05/14/2016 09:56 AM, srorso wrote:

I think a .gitdefaults file could be added to force CRLF for .cmd....could be wrong though. Actually, what is missing is a properly configured .gitattributes file for Hercules. I could provide one, HOWEVER, folks will need to cognizant that they will most likely have to issue a series of Git commands to flush and rebuild their local Git caches. The rebuild process is required to properly sync Git with the changes, otherwise, in an attempt to post an new individual update one may experience Git error messages indicating that there are changes made and Git will not synchronize with the repository and finish the push process. Just reloading your local workspace from the repository does not properly flush and rebuild the Git cache. (I just went through this on another project I am currently working on.)

Mark

[Fish-Git]

I FREAKING GIVE UP!

No matter what I try, it just doesn't work on Nix!

(Not the gitmosttest issue, the original Hercules dynamic VERSION issue, i.e. this GitHub issue)

The new _dynamic_version script that Hercules is now using during make on Nix just does NOT want to work correctly! No matter what I try.

A freshly cloned Hyperion still shows up as "-modified" in the Hecules VERSION string, even though nothing has changed!

The _dynamic_version script is doing:

  git update-index --refresh -q > /dev/null 2>&1
  git diff-index --quiet HEAD

  if test "$?" == "0"; then
    modified_str=""
  else
    modified_str="-modified"
  fi

When I run both commands manually (on a freshly cloned repository), sure enough, git diff-index --quiet HEAD displays:

    _TODO.txt: needs update
    decNumber/ICU-license.html: needs update
    decNumber/decBasic.h: needs update
    decNumber/decCommon.h: needs update
    decNumber/decContext.c: needs update
    decNumber/decContext.h: needs update
    decNumber/decDPD.h: needs update
    decNumber/decDouble.c: needs update
    decNumber/decDouble.h: needs update
    decNumber/decNumber.c: needs update
    decNumber/decNumber.h: needs update
    decNumber/decNumberLocal.h: needs update
    decNumber/decPacked.c: needs update
    decNumber/decPacked.h: needs update
    decNumber/decQuad.c: needs update
    decNumber/decQuad.h: needs update
    decNumber/decSingle.c: needs update
    decNumber/decSingle.h: needs update
    decNumber/decimal128.c: needs update
    decNumber/decimal128.h: needs update
    decNumber/decimal32.c: needs update
    decNumber/decimal32.h: needs update
    decNumber/decimal64.c: needs update
    decNumber/decimal64.h: needs update
    decNumber/example1.c: needs update
    decNumber/example2.c: needs update
    decNumber/example3.c: needs update
    decNumber/example4.c: needs update
    decNumber/example5.c: needs update
    decNumber/example6.c: needs update
    decNumber/example7.c: needs update
    decNumber/example8.c: needs update
    hao.c: needs update
    html/hercinst.html: needs update
    license_dyn76.txt: needs update

(AUGH!) Why? Why? Why?

What freaking git command works reliably to indicate whether any changes to the repository have been made?!

Someone else is going to have to resolve this issue because I've reached the end of my rope.

[Fish-Git]

CLOSED (finally!) by commit 74938ba6bb588cec916d25e0cbfc785c9480bf10.