search

hercules-390 / hyperion

Hercules 390
Other
252 stars 68 forks source link

Building on Windows from .zip file source fails with missing header #135

Closed Fish-Git closed 8 years ago

Fish-Git commented 8 years ago

Attempting to build on Windows from repository .zip file source (as provided by GitHub) fails with missing _dynamic_version.h header file:

dfp.c
d:\dev\github\hercules-390\hyperion-master\version.h (19): fatal error C1083: file (Include) can not be opened: "_dynamic_version.h": No such file or directory
NMAKE: fatal error U1077: "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe" return code "0x2"
Stop.

The root cause is that all files are treated as Unix (Linux) files with LF-only line ending style instead of Windows CRLF line ending style, and Windows batch files do not function properly with LF-only line endings. (files makefile.bat and _dynamic_version.cmd.)

When you clone the repository the problem does not occur because the Windows version of git performs automatic line ending conversion to "native" host style (CRLF on Windows and LF on non-Windows).

The temporary workaround is to use whatever tool is available to convert the line endings to native Windows CRLF style, and then try the build again. (The default "Wordpad" accessory is one such tool. Open the file in Wordpad and then Save it. Wordpad will automatically convert the file to use normal Windows line endings.)

Having a temporary workaround is good, but we need a more permanent resolution. The permanent resolution seems to be to add a .gitattributes file to the repository which specifies the desired line ending style that git should use for specific files and/or file types:

# Set the default behavior, in case people don't have core.autocrlf set.
* text=auto

# Explicitly declare text files you want to always be normalized
# and converted to native line endings on checkout.
*.c text
*.h text

# Denote all files that are truly binary and should not be modified.
*.aws binary
*.bin binary
*.gif binary
*.het binary
*.ico binary
*.jpg binary
*.png binary

# Declare files that will always have CRLF line endings on checkout.
*.bat text eol=crlf
*.cmd text eol=crlf
*.rc* text eol=crlf
*.sln text eol=crlf
*.vc*proj text eol=crlf

# Declare files that will always have LF line endings on checkout.
*.ac text eol=lf
*.am text eol=lf
*.sh text eol=lf
bldlvlck text eol=lf
_dynamic_version text eol=lf

# NOTE: unsure how to declare entire directories (autoconf, m4, man)
# nor whether doing so is even valid in a .gitattributes file.

The files will then have the proper line endings when GitHub creates the download .zip file (already tested and verified).

BUT... Doing so might not be as simple and straightforward as one would expect, especially for a large repository with many files. See for example the "Refreshing a repository after changing line endings" section of the GitHub "Dealing with line endings" page.

jphartmann commented 8 years ago

To create the correct line endings on UNIX while editing, it depends which editor you use:

For vi:

:set list to see control characters. To add CR at the end of a line while in input mode: ctl-v and ctl-m. Control V tells vi to take the next control character literally.

For KEDIT (http://www.kedit.com/ -- the choice of editor for anyone knowing XEDIT):

set eolout crlf

KEDIT runs under wine. http://vm.marist.edu/~pipeline/keditlnx.html

Ultraedit can also be told how to do it, but don't have that to hand.

Fish-Git commented 8 years ago

I myself use d2u and u2d (symlinks for dos2unix and unix2dos). I have the GNU port on my Windows system too, so I use it both places, on *Nix and on Windows. It's a handy tool.

But of course that is just a temporary workaround. Ideally, one should be able to download the .zip file, unzip it, and immediately begin using it, without having to do any post-unzip processing. Thus my suggestion of the .gitattributes file.

If you have git installed and clone the repository instead, then the problem does not occur. It only occurs when you download the GitHub ZIP file (because they're creating the zip file on a Linux server, and it would be unreasonable to expect them to provide separate zip files for Windows and Linux, when a simple .gitattributes file would resolve the problem by making the repository host O/S agnostic).

Fish-Git commented 8 years ago

FYI: I have tested the following .gitattributes file (slightly modified from Mark's version) with one of my ECPS-VSE forks of Hyperion, and it works fine!

#******************************************************************************
#
#   .gitattributes - Set default attributes for files controlled by Git.
#
#******************************************************************************
#
#   Note:  The handling of files here is to handle cases of files which
#          may be part of a Git repository.  File ignore specifications
#          in .gitignore may render part or all of these definitions mute.
#
#   Note:  The only type of files that need to be specified are those for
#          which the default "text=auto" handling would not suffice.  For
#          example, there's no need to specify "*.cpp text".  Only binary
#          file types and text files that must have specific line endings
#          need to be specified.
#
#******************************************************************************

# Default handling for all file types not specified
*                   text=auto

# Documents
*.doc               binary
*.DOC               binary
*.docx              binary
*.DOCX              binary
*.dot               binary
*.DOT               binary
*.pdf               binary
*.PDF               binary
*.rtf               binary
*.RTF               binary

# Script handling
*.bat               text    eol=crlf
*.cmd               text    eol=crlf
*.sh                text    eol=lf

# CMake Templates
*.bat.in            text    eol=crlf
*.cmd.in            text    eol=crlf
*.sh.in             text    eol=lf

# Graphics
*.ai                binary
*.bmp               binary
*.eps               binary
*.gif               binary
*.ico               binary
*.jpeg              binary
*.jpg               binary
*.jpx               binary
*.jxr               binary
*.png               binary
*.psb               binary
*.psd               binary
*.svg               binary
*.svgz              binary
*.tif               binary
*.tiff              binary
*.wbmp              binary
*.webp              binary

# Audio
*.kar               binary
*.m4a               binary
*.mid               binary
*.midi              binary
*.mp3               binary
*.ogg               binary
*.ra                binary

# Video
*.3gpp              binary
*.3gp               binary
*.as                binary
*.asf               binary
*.asx               binary
*.fla               binary
*.flv               binary
*.m4v               binary
*.mng               binary
*.mov               binary
*.mp4               binary
*.mpeg              binary
*.mpg               binary
*.swf               binary
*.ts                binary
*.webm              binary

# Fonts
*.eot               binary
*.otf               binary
*.ttf               binary
*.woff              binary
*.woff2             binary

# Archives and other binary
*.7z                binary
*.gz                binary
*.rar               binary
*.tar               binary
*.zip               binary

# Distribution binaries
*.bundle            binary
*.cab               binary
*.deb               binary
*.jar               binary
*.msi               binary
*.msm               binary
*.msp               binary
*.rpm               binary

# Executables
*.exe               binary
*.pyc               binary

#******************************************************************************
#
#       C/C++
#
#******************************************************************************

# Compiled Object files
*.slo               binary
*.lo                binary
*.o                 binary
*.obj               binary

# Precompiled Headers
*.gch               binary
*.pch               binary

# Compiled Dynamic libraries
*.so                binary
*.dylib             binary
*.dll               binary

# Compiled Static libraries
*.lai               binary
*.la                binary
*.a                 binary
*.lib               binary

# Executables
*.out               binary
*.app               binary

#******************************************************************************
#
#       Java and NetRexx
#
#******************************************************************************

# Executables
*.class             binary

#******************************************************************************
#
#       Visual Studio
#
#******************************************************************************

*.sln               text    eol=crlf
*.csproj            text    eol=crlf
*.vbproj            text    eol=crlf
*.fsproj            text    eol=crlf
*.dbproj            text    eol=crlf

*.vcproj            text    eol=crlf
*.vcxproj           text    eol=crlf
*.vcxitems          text    eol=crlf
*.props             text    eol=crlf
*.filters           text    eol=crlf

# CMake Templates
*.filters.in        text    eol=crlf
*.vcproj.user.in    text    eol=crlf
*.vcxproj.user.in   text    eol=crlf

#******************************************************************************
#
#   Hercules and SATK specific files not covered by any preceding rule
#
#******************************************************************************

# Build (root directory)
*.ac                text    eol=lf
*.am                text    eol=lf
_build              text    eol=lf
_build.tail         text    eol=lf
_dynamic_version    text    eol=lf

# Build (subdirectories)
autoconf/*          text    eol=lf
m4/*                text    eol=lf
man/*               text    eol=lf
util/bldlvlck       text    eol=lf

# Binary objects
*.aws               binary
*.bin               binary
*.core              binary
*.fkt               binary
*.het               binary
util/scsiboot       binary

# EBCDIC card Images
*.card              binary
*.text              binary

# Miscellaneous
util/dasdlist       text    eol=lf
tests/runtest       text    eol=lf

I also did not have to run any type of special git commands afterwards either! (as outlined in the previously mentioned "Refreshing a repository after changing line endings" section of the GitHub "Dealing with line endings" page).

All I did was add the .gitattributes file to the repository, commit that add, and then push to GitHub.

Done!

It was actually quite painless. :)

One thing I did notice however was that none of the files in the repository that I committed and pushed from (that was originally cloned before the commit + push, obviously) were updated to their new line endings. Instead, I had to re-clone the repository to get the new line endings.

So everything looks good.

Should I go ahead and commit the above .gitattributes file to Hyperion?

prino commented 8 years ago

.RTF is text, but never tried LF only files in Word...

Robert AH Prins robert(a)prino(d)org

Fish-Git commented 8 years ago

.rtf files can contain binary data.

prino commented 8 years ago

Yes,but in hex (20800000003D00F0C4C1E8E2D6E4E3400600003003000004E6C5C5D2D6E4E340) format. RTF is PURE ASCII.

Robert AH Prins robert(a)prino(d)org

prino commented 8 years ago

Store a JPG in an RTF file and look at it with a normal ASCII editor.

Robert AH Prins robert(a)prino(d)org

prino commented 8 years ago

Store a JPG in a text file in Word, save it as RTF and look at it with a text editor.

Robert AH Prins robert(a)prino(d)org

Fish-Git commented 8 years ago

Images are saved in ASCII hex format by default, but .rtf files can contain pure binary data since 1997 (version 1.5):

... defined inclusion of PNG, JPEG and EMF picture types in hexadecimal (the default) or binary format in a RTF file ...

Refer to the \bin keyword in the official specification manual, direct from Microsoft:

I am quite familiar with the .rtf format after having added .rtf file support to my HercPrt product (and having struggled with getting embedding fonts to work (they don't)), and I can assure you .rtf files can contain binary data!

Fish-Git commented 8 years ago

I've uploaded two sample RTF files to my Public FTP site called fishtest-hex.rtf and fishtest-bin.rtf. They're in the misc folder:

The fishtest-hex.rtf file uses ASCII hex (the default) and is 2.4 MB. The fishtest-bin.rtf file uses raw binary and is only 21 KB.

They both contain identical content: just a single jpeg image ("Fast. Good. Cheap... Pick any two!")

Both display just fine in both Word Viewer as well as Microsoft Word itself. (The silly "Wordpad" accessory that comes with Windows doesn't work.)

They both also display just fine in LibreOffice on my CentOS 6.7 Linux virtual machine too.

prino commented 8 years ago

I stand corrected, but the difference in size shows the total incompetence of M$. The hex.rtf should not be more than just over twice the size of the bin.rtf one

Robert AH Prins robert(a)prino(d)org

Fish-Git commented 8 years ago

Getting back to the .gitaatributes issue, it appears the procedure described in the "Refreshing a repository after changing line endings" section of the GitHub "Dealing with line endings" page may be needed after all in certain circumstances.

It's unclear to me what those circumstances might be, but I've been told by Mark the situation described has happened to him. It did not happen to me, but that may have been because after committing and pushing the new (or updated) .gitaatributes file, I immediately deleted my local copy and re-cloned it again from [the GitHub] remote (i.e. I started "fresh" again).

Another possible reason why I didn't have to follow the "hard reset" procedure may be because I'm using TortoiseGit backed by Git for Windows, one of which (or perhaps both) may have internally performed the procedure for me automatically. I say this because I did notice whenever I would commit the .gitaatributes file (whether it was the first time or a subsequent change to it), it would take a lot longer than normal, as if some "behind the scenes" processing was taking place.

Regardless, the following is the final version of the .gitaatributes file I intend to commit. I've re-added the 'text' lines which I thought were redundant (and removed my comment as well) so that it more closely matches Mark's original version:

#******************************************************************************
#
#   .gitattributes - Set default attributes for files controlled by Git.
#
#******************************************************************************
#
#   Note:  The handling of files here is to handle cases of files which
#          may be part of a Git repository.  File ignore specifications
#          in .gitignore may render part or all of these definitions mute.
#
#******************************************************************************

# Default handling for all file types not specified

*                   text=auto

# Documents
*.doc               binary
*.DOC               binary
*.docx              binary
*.DOCX              binary
*.dot               binary
*.DOT               binary
*.pdf               binary
*.PDF               binary
*.rtf               binary
*.RTF               binary

*.adoc              text
*.csv               text
*.markdown          text
*.md                text
*.mdwn              text
*.mdown             text
*.mkd               text
*.mkdn              text
*.mdtxt             text
*.mdtext            text
*.tab               text
*.textile           text
*.tsv               text
*.txt               text
AUTHORS             text
AUTHORS.*           text
CHANGELOG           text
CHANGELOG.*         text
CHANGES             text
CHANGES.*           text
CONTRIBUTING        text
CONTRIBUTING.*      text
COPYING             text
COPYING.*           text
INSTALL             text
INSTALL.*           text
license             text
license.*           text
LICENSE             text
LICENSE.*           text
NEWS                text
NEWS.*              text
readme              text
readme.*            text
README              text
README.*            text
*.README            text
*.README.*          text
TODO                text
TODO.*              text

# Script handling
*.bat               text    eol=crlf
*.cmd               text    eol=crlf
*.sh                text    eol=lf

# Git Configuration
.gitattributes      text
.gitignore          text
.gitconfig          text

# Build Configuration
*.cmake             text
*.ninja             text
.ninja_*            text
Makefile            text
makefile            text

# General Configuration
*.cnf               text
*.conf              text
*.config            text
*.ini               text

# CMake Templates
*.bat.in            text    eol=crlf
*.cmd.in            text    eol=crlf
*.sh.in             text    eol=lf
*.in                text

# Graphics
*.ai                binary
*.bmp               binary
*.eps               binary
*.gif               binary
*.ico               binary
*.jpeg              binary
*.jpg               binary
*.jpx               binary
*.jxr               binary
*.png               binary
*.psb               binary
*.psd               binary
*.svg               binary
*.svgz              binary
*.tif               binary
*.tiff              binary
*.wbmp              binary
*.webp              binary

# Audio
*.kar               binary
*.m4a               binary
*.mid               binary
*.midi              binary
*.mp3               binary
*.ogg               binary
*.ra                binary

# Video
*.3gpp              binary
*.3gp               binary
*.as                binary
*.asf               binary
*.asx               binary
*.fla               binary
*.flv               binary
*.m4v               binary
*.mng               binary
*.mov               binary
*.mp4               binary
*.mpeg              binary
*.mpg               binary
*.swf               binary
*.ts                binary
*.webm              binary

# Fonts
*.eot               binary
*.otf               binary
*.ttf               binary
*.woff              binary
*.woff2             binary

# Archives and other binary
*.7z                binary
*.gz                binary
*.rar               binary
*.tar               binary
*.zip               binary

# Distribution binaries
*.bundle            binary
*.cab               binary
*.deb               binary
*.jar               binary
*.msi               binary
*.msm               binary
*.msp               binary
*.rpm               binary

# Executables
*.exe               binary
*.pyc               binary

#******************************************************************************
#
#       C/C++
#
#******************************************************************************

# Sources
*.c                 text
*.cc                text
*.cxx               text
*.cpp               text
*.c++               text
*.hpp               text
*.h                 text
*.h++               text
*.hh                text

# Compiled Object files
*.slo               binary
*.lo                binary
*.o                 binary
*.obj               binary

# Precompiled Headers
*.gch               binary
*.pch               binary

# Compiled Dynamic libraries
*.so                binary
*.dylib             binary
*.dll               binary

# Compiled Static libraries
*.lai               binary
*.la                binary
*.a                 binary
*.lib               binary

# Executables
*.out               binary
*.app               binary

# CMake Templates
*.c.in              text
*.cc.in             text
*.cxx.in            text
*.cpp.in            text
*.c++.in            text
*.hpp.in            text
*.h.in              text
*.h++.in            text
*.hh.in             text

#******************************************************************************
#
#       Java and NetRexx
#
#******************************************************************************

# Source
*.java              text
*.nrx               text

# Executables
*.class             binary

#******************************************************************************
#
#       Rexx
#
#******************************************************************************

*.cls               text
*.rex               text
*.rexx              text
*.scls              text
*.srex              text
*.srexx             text

#******************************************************************************
#
#       Visual Studio
#
#******************************************************************************

*.sln               text    eol=crlf
*.csproj            text    eol=crlf
*.vbproj            text    eol=crlf
*.fsproj            text    eol=crlf
*.dbproj            text    eol=crlf

*.vcproj            text    eol=crlf
*.vcxproj           text    eol=crlf
*.vcxitems          text    eol=crlf
*.props             text    eol=crlf
*.filters           text    eol=crlf

# CMake Templates
*.filters.in        text    eol=crlf
*.vcproj.user.in    text    eol=crlf
*.vcxproj.user.in   text    eol=crlf

#******************************************************************************
#
#       Web Projects
#
#******************************************************************************

# Source Code
*.coffee            text
*.css               text
*.htm               text
*.html              text
*.inc               text
*.js                text
*.json              text
*.less              text
*.php               text
*.pl                text
*.py                text
*.rb                text
*.sass              text
*.scm               text
*.scss              text
*.sql               text
*.styl              text
*.svg               text
*.ts                text
*.xml               text

# Web Templates
*.dot               text
*.ejs               text
*.haml              text
*.handlebars        text
*.hbs               text
*.hbt               text
*.jade              text
*.latte             text
*.mustache          text
*.phtml             text
*.tmpl              text

# Server Configuration
.htaccess           text

# Code Analysis Configuration
.csslintrc          text
.eslintrc           text
.jshintrc           text
.jscsrc             text
.jshintignore       text
.stylelintrc        text

# Miscellaneous Configurations
*.yaml              text
*.yml               text
.editorconfig       text

# Build Configuration (Jason and NPM)
*.npmignore         text
*.bowerrc           text

# Heroku
Procfile            text
.slugignore         text

#******************************************************************************
#
#   Hercules and SATK specific files not covered by any preceding rule
#
#******************************************************************************

# Build (root directory)
*.ac                text    eol=lf
*.am                text    eol=lf
_build              text    eol=lf
_build.tail         text    eol=lf
_dynamic_version    text    eol=lf
*.msvc              text    eol=crlf

# Build (subdirectories)
autoconf/*          text    eol=lf
m4/*                text    eol=lf
man/*               text    eol=lf
util/bldlvlck       text    eol=lf

msvc.makefile.includes/*   text  eol=crlf

# Binary objects
*.aws               binary
*.bin               binary
*.core              binary
*.fkt               binary
*.het               binary
util/scsiboot       binary

# EBCDIC card Images
*.card              binary
*.text              binary

# Miscellaneous
util/dasdlist       text    eol=lf
tests/runtest       text    eol=lf
tests/*.tst         text
tests/*.subtst      text
tests/*.assem*      text
tests/*.list*       text

 

I'll be committing the above .gitaatributes file to the Hyperion repository later today which should resolve this issue.

Please note however that even with a .gitaatributes file, downloading the .zip file from GitHub will still cause some of your files (such as all of the c and h source files and headers for example) to contain the "wrong" line endings due to the fact that GitHub creates the .zip file on a Linux system. This is not seen as a problem since the important files (e.g. the batch files) will have the proper line endings, which is the only thing we're really concerned with. (C/C++ compilers don't care what line endings you use)

Thanks.