Windows binary distribution

simoncozens commented 7 years ago

We need one. But I don't have the resources to make one.

To make this happen we will need:

[x] A reproducible (hopefully scriptable) Windows build. See also #82. We just made this harder by requiring ICU.
[x] A C wrapper around the SILE Lua script, so it can be more easily run as an EXE. (And without the Lua runtime.)
[ ] To isolate the libraries which are required, so that they can all be shipped as part of a Windows distribution
[ ] To ship all the needed files as a distribution and test them on a virgin system ie with no ming32/msys2 installed.

indiamcq commented 7 years ago

Simon,

I am working in mingw64 on Win 10. I have used your build from source instructions in this environment. I am thinking that this will create a sile.exe file, but as I don't know much about this stuff, I may be off track.

The ./configure is getting stuck on:

checking for FONTCONFIG... yes
checking for lua5.3... /mingw64/bin/lua5.3
checking if /mingw64/bin/lua5.3 is a Lua interpreter... no
configure: error: not a Lua interpreter

I had modified the configure.ac from AX_PROG_LUA([5.1]) to AX_PROG_LUA

Also I modified the ax_lua.m4 to

  m4_define_default([_AX_LUA_INTERPRETER_LIST],
    [lua5.3 lua53 lua5.2 lua52 lua5.1 lua51 lua50 lua])

i.e it now includes lua5.3 that was missing. Though moving lua to the end may be wrong.

Any insights on what may be wrong? If what I am doing is off track then just say so.

Ian

simoncozens commented 7 years ago

I've no idea why that's failing. The test it should be performing is:

$ lua5.3 -e "print('Hello ' .. _VERSION .. '!')"
Hello Lua 5.3!

Can you attach the output of config.log?

simoncozens commented 7 years ago

Well, I just managed to replicate that error! Looking into it now.

simoncozens commented 7 years ago

It's this: https://github.com/Alexpux/MINGW-packages/issues/949 Not sure how to fix it but I think the end goal for Windows is going to involve a custom SILE binary anyway, so I wonder if we could actually get rid of this test.

simoncozens commented 7 years ago

OK, here's the state of SILE building on Windows so far:

I'm building a Lua from source on cygwin/msys2/mingw64, using make mingw. It seems to work.
I can't build the various extensions (lpeg, lxp, etc.) from luarocks, because the linker fails with all of the symbols from Lua being undefined.
I can build them by hand if I add -shared /path/to/lua5.X.dll to the linker line. I don't actually know if this is the right thing to do or not.
They work fine if the produced lpeg.so (.so not .dll because it's mingw) etc. is in the current working directory. If I put them somewhere in Lua's cpath, I get something like "required module not found" when I try to require them. This is obviously a problem.
mingw's zlib doesn't have compress2, so libtexpdf can't build automatically. Spiking the check and building the library by hand seems to work OK.
But then building justenough... is not working, as libtool is struggling to produce a DLL which links to another DLL. We just did this by hand for the other Lua extensions, so it must be possible, but libtool is complaining.

This is all happening in the Appveyor branch, with build logs here: https://ci.appveyor.com/project/simoncozens/sile/build/1.0.73

If anyone who knows Lua-on-Windows (I'm looking at you @pkulchenko) has time to have a look over what I'm doing and sanity check it, particularly appveyor.yml and .appveyor/build.sh, it would be very helpful.

pkulchenko commented 7 years ago

@simoncozens, I won't be able to help with appveyor config, but I can definitely try to build SILE from scratch on Windows, similar to what I did in #82.

A C wrapper around the SILE Lua script, so it can be more easily run as an EXE. (And without the Lua runtime.)

Why would you need this?

simoncozens commented 7 years ago

Thanks! Would be very handy to compare notes.

The idea behind a C wrapper (which I've now written) ties into my end goal for distribution on Windows: rather than installing Cygwin and Lua and Luarocks and everything else and then SILE, the installer just unpacks a punch of stuff into C:\Program Files\SILE, where all the dependency libraries - Mingw, Freetype, Harfbuzz, ICU, Lua and so on - are stashed away and nobody needs to care that they're running Lua and half of Unix when they type "sile foo.sil".

pkulchenko commented 7 years ago

Simon, I've made some progress on getting this to run, but on a snapshot from last year, which predates lfs and justenoughicu requirements.

I got it working using DLLs from mingw32/msys2 distribution; here is the full list of libraries used:

libbz2-1.dll
libexpat-1.dll
libfontconfig-1.dll
libfreetype-6.dll
libgcc_s_dw2-1.dll
libglib-2.0-0.dll
libharfbuzz-0.dll
libiconv-2.dll
libintl-8.dll
libpng16-16.dll
libtexpdf-0.dll
libwinpthread-1.dll
zlib1.dll
lpeg.dll
lxp.dll

The last two are Lua libraries and the rest are DLLs from msys\mingw32\bin folder. I also have justenoughharfbuzz.dll and justenoughlibtexpdf.dll I built using SILE scripts.

There have been couple of issues; first of all, I copied all the DLLs into the local folder from which I run SILE; if this is not done, then there is a high likelihood that one of the libraries will be present in one of the folders in PATH that sits before mingw folders, which is likely to make it fail, simply because it will be a 64bit library (from Ruby23-x64) or a library from a different version of mingw or something similar.

The second issue was with libtexpdf from mingw, which required fonts.conf file (I didn't use libtexpdf from SILE as I ran into issues with compiling it; maybe need to try one more time). It just couldn't find any of the fonts without fonts.conf being present. I fixed this by adding set FONTCONFIG_FILE=D:\full\path\to\sile\fonts.conf before launching SILE with the content of the file taken from https://github.com/Shoes3/shoes3/issues/46#issuecomment-71587063.

This worked fine, but it took a while to complete the first run (probably about 20s) while the fonts were scanned and the cache was compiled. The subsequent runs were without much of a delay.

The next step would be to build a more recent version with justenoughicu and other additions.

pkulchenko commented 7 years ago

Couple of issues I ran into compiling the current master with msys2/mingw32:

  CC       justenoughfontconfig_la-justenoughfontconfig.lo
In file included from justenoughfontconfig.c:10:0:
silewin32.h: In function 'strcasestr':
silewin32.h:9:5: warning: implicit declaration of function 'tolower' [-Wimplicit-function-declaration]
     if (tolower(haystack[i]) != tolower(needle[matchamt]))
     ^
  CCLD     justenoughfontconfig.la
  CC       fontmetrics_la-fontmetrics.lo
  CCLD     fontmetrics.la
  CC       svg_la-svg.lo
  CCLD     svg.la
  CC       justenoughicu_la-justenoughicu.lo
justenoughicu.c:5:29: fatal error: unicode/ustring.h: No such file or directory
 #include <unicode/ustring.h>
                             ^
compilation terminated.
Makefile:653: recipe for target 'justenoughicu_la-justenoughicu.lo' failed
make[1]: *** [justenoughicu_la-justenoughicu.lo] Error 1
make[1]: Leaving directory '/home/Paul/sile/src'
Makefile:510: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

I fixed this by adding: -I/usr/include to DEFAULT_INCLUDES = -I. -I/usr/include.

  CC       justenoughicu_la-justenoughicu.lo
  CCLD     justenoughicu.la

*** Warning: linker path does not have real file for library -licuio.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicuio but no candidates were found. (...for file magic test)

*** Warning: linker path does not have real file for library -licui18n.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicui18n but no candidates were found. (...for file magic test)

*** Warning: linker path does not have real file for library -licuuc.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicuuc but no candidates were found. (...for file magic test)

*** Warning: linker path does not have real file for library -licudata.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicudata but no candidates were found. (...for file magic test)

*** Warning: libtool could not satisfy all declared inter-library
*** dependencies of module justenoughicu.  Therefore, libtool will create
*** a static module, that should work as long as the dlopening
*** application is linked with the -dlopen flag.
cp .libs/justenoughharfbuzz.dll .libs/justenoughfontconfig.dll  .libs/justenoughlibtexpdf.dll .libs/fontmetrics.dll .libs/svg.dll .libs/justenoughicu.dll  ../core
cp: cannot stat ‘.libs/justenoughicu.dll’: No such file or directory
Makefile:935: recipe for target 'all-local' failed
make[1]: *** [all-local] Error 1

This is fixed by installing a proper icu module: pacman -S mingw-w64-i686-icu; this still didn't quite work as it did find some libraries, but not others:

  CC       justenoughicu_la-justenoughicu.lo
  CCLD     justenoughicu.la

*** Warning: linker path does not have real file for library -licui18n.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicui18n and none of the candidates passed a file format test
*** using a file magic. Last file checked: D:/utils/msys64/mingw32/lib/libicuio.dll.a

*** Warning: linker path does not have real file for library -licudata.
*** I have the capability to make that library automatically link in when
*** you link to this library.  But I can only do this if you have a
*** shared version of the library, which you do not appear to have
*** because I did check the linker path looking for a file starting
*** with libicudata and none of the candidates passed a file format test
*** using a file magic. Last file checked: D:/utils/msys64/mingw32/lib/libicuuc.dll.a

*** Warning: libtool could not satisfy all declared inter-library
*** dependencies of module justenoughicu.  Therefore, libtool will create
*** a static module, that should work as long as the dlopening
*** application is linked with the -dlopen flag.
cp .libs/justenoughharfbuzz.dll .libs/justenoughfontconfig.dll  .libs/justenoughlibtexpdf.dll .libs/fontmetrics.dll .libs/svg.dll .libs/justenoughicu.dll  ../core
cp: cannot stat ‘.libs/justenoughicu.dll’: No such file or directory
Makefile:935: recipe for target 'all-local' failed

I fixed this by renaming ICU libraries according to this discussion: ICU_LIBS = -L/usr/lib -licuio -licuin -licuuc -licudt -lpthread -lm.

I also got message ./libtool: line 7916: func_munge_path_list: command not found, but it didn't seem to impact anything.

pkulchenko commented 7 years ago

Here is the list of additional libraries that is needed to decouple the installation from mingw:

libicudt57.dll
libicuin57.dll
libicuuc57.dll
libstdc++-6.dll
libzlib.dll (Lua library)

@simoncozens, I got it all running without issues, but the resulting PDFs have boxes instead of characters. What would be the reason for that? The console output looks correct:

luajit sile -e "require('core/libtexpdf-output')" -d fonts examples\simple.sil
This is SILE 0.9.5-unreleased
<examples\simple.sil>
[fonts] Looking for Gentium Plus;10;400;;normal;;LTR
[fonts] Resolved font family Gentium Plus -> C:/Windows/Fonts/GentiumPlus-R.ttf
[1]

pkulchenko commented 7 years ago

I rolled back changes to the earlier commit that works for me (e9d821d4), but still using the new binaries and with justenoughharfbuzz.dll from the same commit (as I couldn't use the new one as the API has changed) everything works, so it seems to be related to the changes in harfbuzz.

pkulchenko commented 7 years ago

This is what SILE.shaper.debugVersions() shows:

Harfbuzz version: 0.9.40
Shapers enabled: ot, fallback
ICU support enabled

Fonts used:
C:/Windows/Fonts/GentiumPlus-I.ttf:0    Unknown version
C:/Windows/Fonts/GentiumPlus-R.ttf:0    Unknown version

pkulchenko commented 7 years ago

@simoncozens, @alerque, any ideas on why I'm getting boxes instead of characters in the output?

simoncozens commented 7 years ago

OK, that's weird. Strange that it can't get a version string out of the fonts, which makes me suspect that either the fonts are bad or that the opentype parser isn't working on Windows. That shouldn't affect output, however. Could you try:

% ./sile -e 'require("core/debug-output")' examples/simple.sil

This will tell me whether or not the problem is to do with finding the right glyphs in the font (Harfbuzz problem) or outputting the glyph to pdf (libtexpdf problem).

pkulchenko commented 7 years ago

@simoncozens, here is the output:

luajit sile -e "require('core/debug-output')" examples/simple.sil 
<examples/simple.sil>
Set paper size  365.6692953 561.2598486
Begin page
Mx  50.3506
My  65.1061
Set font    Gentium Plus;10;400;;normal;;LTR
T   0   (I)
Mx  61.3644
T   0 0 0 0 (have)
Mx  87.3783
T   0 0 0 0 0 0 (seldom)
Mx  123.3922
T   0 0 0 0 0   (heard)
Mx  154.4061
T   0 0 0   (him)
Mx  175.4200
T   0 0 0 0 0 0 0   (mention)
Mx  216.4339
T   0 0 0   (her)
Mx  237.4478
T   0 0 0 0 0   (under)
Mx  268.4617
T   0 0 0   (any)
Mx  289.4756
T   0 0 0 0 0   (other)
Mx  30.3506
My  77.1061
T   0 0 0 0 (name)
Mx  50.3506
T   0   (.)
My  89.1061
T   0 0 0 0 (SILE)
Mx  77.1162
T   0 0 0 0 0   (comes)
Mx  108.8818
T   0 0 0 0 (with)
Mx  135.6474
T   0   (a)
Mx  147.4131
T   0 0 0 0 0 0 (number)
Mx  184.1787
T   0 0 (of)
Mx  200.9443
T   0 0 0 0 0 0 0 0 (packages)
Mx  247.7100
T   0 0 0 0 0   (which)
Mx  279.4756
T   0 0 0 0 0 0 0   (provide)
Mx  30.3506
My  101.1061
T   0 0 0 0 0 0 0 0 0 0 (additional)
Mx  88.5381
T   0 0 0 0 0 0 0 0 0 0 0 0 0   (functionality)
Mx  153.5381
T   0   (.)
Mx  166.7256
T   0 0 0 0 0   (Lorem)
Mx  199.9131
T   0 0 0 0 0   (ipsum)
Mx  233.1006
T   0 0 0 0 0   (dolor)
Mx  266.2881
T   0 0 0   (sit)
Mx  289.4756
T   0 0 0 0 (amet)
Mx  309.4756
T   0   (,)
Mx  30.3506
My  113.1061
T   0 0 0 0 0 0 0 0 0 0 0   (consectetur)
Mx  91.6541
T   0 0 0 0 0 0 0 0 0 0 0   (adipisicing)
Mx  152.9577
T   0 0 0 0 (elit)
Mx  172.9577
T   0   (,)
Mx  184.2613
T   0 0 0   (sed)
Mx  205.5649
T   0 0 (do)
Mx  221.8684
T   0 0 0 0 0 0 0   (eiusmod)
Mx  263.1720
T   0 0 0 0 0 0 (tempor)
Mx  299.4756
T   0 0 (in)
Mx  309.4756
T   0   (-)
Mx  30.3506
My  125.1061
T   0 0 (ci)
Mx  40.3506
T   0 0 0 0 0 0 (didunt)
Mx  77.4756
T   0 0 (ut)
Mx  94.6006
T   0 0 0 0 0 0 (labore)
Mx  131.7256
T   0 0 (et)
Mx  148.8506
T   0 0 0 0 0 0 (dolore)
Mx  185.9756
T   0 0 0 0 0   (magna)
Mx  218.1006
T   0 0 0 0 0 0 (aliqua)
Mx  248.1006
T   0   (.)
Mx  260.2256
T   0 0 (Ut)
Mx  277.3506
T   0 0 0 0 (enim)
Mx  304.4756
T   0 0 (ad)
Mx  30.3506
My  137.1061
T   0 0 0 0 0   (minim)
Mx  61.8714
T   0 0 0 0 0 0 (veniam)
Mx  91.8714
T   0   (,)
Mx  103.3922
T   0 0 0 0 (quis)
Mx  129.9131
T   0 0 0 0 0 0 0   (nostrud)
Mx  171.4339
T   0 0 0 0 0 0 0 0 0 0 0 0 (exercitation)
Mx  237.9548
T   0 0 0 0 0 0 0   (ullamco)
Mx  279.4756
T   0 0 0 0 0 0 0   (laboris)
Mx  30.3506
My  149.1061
T   0 0 0 0 (nisi)
Mx  55.8089
T   0 0 (ut)
Mx  71.2672
T   0 0 0 0 0 0 0   (aliquip)
Mx  111.7256
T   0 0 (ex)
Mx  127.1839
T   0 0 (ea)
Mx  142.6422
T   0 0 0 0 0 0 0   (commodo)
Mx  183.1006
T   0 0 0 0 0 0 0 0 0   (consequat)
Mx  228.1006
T   0   (.)
Mx  238.5589
T   0 0 0 0 (Duis)
Mx  264.0173
T   0 0 0 0 (aute)
Mx  289.4756
T   0 0 0 0 0   (irure)
Mx  30.3506
My  161.1061
T   0 0 0 0 0   (dolor)
Mx  63.0827
T   0 0 (in)
Mx  80.8148
T   0 0 0 0 0 0 0 0 0 0 0 0 0   (reprehenderit)
Mx  153.5470
T   0 0 (in)
Mx  171.2791
T   0 0 0 0 0 0 0 0 0   (voluptate)
Mx  224.0113
T   0 0 0 0 0   (velit)
Mx  256.7434
T   0 0 0 0 (esse)
Mx  284.4756
T   0 0 0 0 0 0 (cillum)
Mx  30.3506
My  173.1061
T   0 0 0 0 0 0 (dolore)
Mx  65.9398
T   0 0 (eu)
Mx  81.5291
T   0 0 0 0 0 0 (fugiat)
Mx  117.1184
T   0 0 0 0 0   (nulla)
Mx  147.7077
T   0 0 0 0 0 0 0 0 (pariatur)
Mx  187.7077
T   0   (.)
Mx  198.2970
T   0 0 0 0 0 0 0 0 0   (Excepteur)
Mx  248.8863
T   0 0 0 0 (sint)
Mx  274.4756
T   0 0 0 0 0 0 0 0 (occaecat)
Mx  30.3506
My  185.1061
T   0 0 0 0 0 0 0 0 0   (cupidatat)
Mx  82.7412
T   0 0 0   (non)
Mx  105.1318
T   0 0 0 0 0 0 0 0 (proident)
Mx  145.1318
T   0   (,)
Mx  157.5224
T   0 0 0 0 (sunt)
Mx  184.9131
T   0 0 (in)
Mx  202.3037
T   0 0 0 0 0   (culpa)
Mx  234.6943
T   0 0 0   (qui)
Mx  257.0850
T   0 0 0 0 0 0 0   (officia)
Mx  299.4756
T   0 0 (de)
Mx  309.4756
T   0   (-)
Mx  30.3506
My  197.1061
T   0 0 0 0 0 0 (serunt)
Mx  66.3778
T   0 0 0 0 0 0 (mollit)
Mx  102.4050
T   0 0 0 0 (anim)
Mx  128.4323
T   0 0 (id)
Mx  144.4595
T   0 0 0   (est)
Mx  165.4868
T   0 0 0 0 0 0 0   (laborum)
Mx  200.4868
T   0   (.)
Mx  30.3506
My  209.1061
T   0 0 0 0 (SILE)
Mx  55.3506
T   0 0 0 0 0   (comes)
Mx  85.3506
T   0 0 0 0 (with)
Mx  110.3506
T   0   (a)
Mx  120.3506
T   0 0 0 0 0 0 (number)
Mx  155.3506
T   0 0 (of)
Mx  170.3506
T   0 0 0 0 0 0 0 0 (packages)
Mx  215.3506
T   0 0 0 0 0   (which)
Mx  245.3506
T   0 0 0 0 0 0 0   (provide)
Mx  30.3506
My  221.1061
T   0 0 0 0 0 0 0 0 0 0 (additional)
Mx  85.3506
T   0 0 0 0 0 0 0 0 0 0 0 0 0   (functionality)
Mx  150.3506
T   0   (.)
Mx  160.3506
T   0 0 0 0 0   (Lorem)
Mx  190.3506
T   0 0 0 0 0   (ipsum)
Mx  220.3506
T   0 0 0 0 0   (dolor)
Mx  250.3506
T   0 0 0   (sit)
Mx  270.3506
T   0 0 0 0 (amet)
Mx  290.3506
T   0   (,)
Mx  30.3506
My  233.1061
T   0 0 0 0 0 0 0 0 0 0 0   (consectetur)
Mx  90.3506
T   0 0 0 0 0 0 0 0 0 0 0   (adipisicing)
Mx  150.3506
T   0 0 0 0 (elit)
Mx  170.3506
T   0   (,)
Mx  180.3506
T   0 0 0   (sed)
Mx  200.3506
T   0 0 (do)
Mx  215.3506
T   0 0 0 0 0 0 0   (eiusmod)
Mx  255.3506
T   0 0 0 0 0 0 (tempor)
Mx  30.3506
My  245.1061
T   0 0 0 0 0 0 0 0 0 0 (incididunt)
Mx  85.3506
T   0 0 (ut)
Mx  100.3506
T   0 0 0 0 0 0 (labore)
Mx  135.3506
T   0 0 (et)
Mx  150.3506
T   0 0 0 0 0 0 (dolore)
Mx  185.3506
T   0 0 0 0 0   (magna)
Mx  215.3506
T   0 0 0 0 0 0 (aliqua)
Mx  245.3506
T   0   (.)
Mx  255.3506
T   0 0 (Ut)
Mx  270.3506
T   0 0 0 0 (enim)
Mx  295.3506
T   0 0 (ad)
Mx  30.3506
My  257.1061
T   0 0 0 0 0   (minim)
Mx  60.3506
T   0 0 0 0 0 0 (veniam)
Mx  90.3506
T   0   (,)
Mx  100.3506
T   0 0 0 0 (quis)
Mx  125.3506
T   0 0 0 0 0 0 0   (nostrud)
Mx  165.3506
T   0 0 0 0 0 0 0 0 0 0 0 0 (exercitation)
Mx  230.3506
T   0 0 0 0 0 0 0   (ullamco)
Mx  270.3506
T   0 0 0 0 0 0 0   (laboris)
Mx  30.3506
My  269.1061
T   0 0 0 0 (nisi)
Mx  55.3506
T   0 0 (ut)
Mx  70.3506
T   0 0 0 0 0 0 0   (aliquip)
Mx  110.3506
T   0 0 (ex)
Mx  125.3506
T   0 0 (ea)
Mx  140.3506
T   0 0 0 0 0 0 0   (commodo)
Mx  180.3506
T   0 0 0 0 0 0 0 0 0   (consequat)
Mx  225.3506
T   0   (.)
Mx  235.3506
T   0 0 0 0 (Duis)
Mx  260.3506
T   0 0 0 0 (aute)
Mx  285.3506
T   0 0 0 0 0   (irure)
Mx  30.3506
My  281.1061
T   0 0 0 0 0   (dolor)
Mx  60.3506
T   0 0 (in)
Mx  75.3506
T   0 0 0 0 0 0 0 0 0 0 0 0 0   (reprehenderit)
Mx  145.3506
T   0 0 (in)
Mx  160.3506
T   0 0 0 0 0 0 0 0 0   (voluptate)
Mx  210.3506
T   0 0 0 0 0   (velit)
Mx  240.3506
T   0 0 0 0 (esse)
Mx  265.3506
T   0 0 0 0 0 0 (cillum)
Mx  30.3506
My  293.1061
T   0 0 0 0 0 0 (dolore)
Mx  65.3506
T   0 0 (eu)
Mx  80.3506
T   0 0 0 0 0 0 (fugiat)
Mx  115.3506
T   0 0 0 0 0   (nulla)
Mx  145.3506
T   0 0 0 0 0 0 0 0 (pariatur)
Mx  185.3506
T   0   (.)
Mx  195.3506
T   0 0 0 0 0 0 0 0 0   (Excepteur)
Mx  245.3506
T   0 0 0 0 (sint)
Mx  270.3506
T   0 0 0 0 0 0 0 0 (occaecat)
Mx  30.3506
My  305.1061
T   0 0 0 0 0 0 0 0 0   (cupidatat)
Mx  80.3506
T   0 0 0   (non)
Mx  100.3506
T   0 0 0 0 0 0 0 0 (proident)
Mx  140.3506
T   0   (,)
Mx  150.3506
T   0 0 0 0 (sunt)
Mx  175.3506
T   0 0 (in)
Mx  190.3506
T   0 0 0 0 0   (culpa)
Mx  220.3506
T   0 0 0   (qui)
Mx  240.3506
T   0 0 0 0 0 0 0   (officia)
Mx  30.3506
My  317.1061
T   0 0 0 0 0 0 0 0 (deserunt)
Mx  75.3506
T   0 0 0 0 0 0 (mollit)
Mx  110.3506
T   0 0 0 0 (anim)
Mx  135.3506
T   0 0 (id)
Mx  150.3506
T   0 0 0   (est)
Mx  170.3506
T   0 0 0 0 0 0 0   (laborum)
Mx  205.3506
T   0   (.)
[1] Mx  169.9131
My  484.3672
T   0   (1)
End page
Finish

Harfbuzz version: 0.9.40
Shapers enabled: ot, fallback
ICU support enabled

Fonts used:
C:/Windows/Fonts/GentiumPlus-R.ttf:0    Unknown version

The fonts should be good as I can use them with the old version from the same install without issues (I just switch the DLLs and checkout older SILE commit), so it looks like Harfbuzz issue as the results are wrong in my viewer as well (no PDF output).

simoncozens commented 7 years ago

Yeah, all the zeroes are telling me that it's a Harfbuzz issue. But the fact that Harfbuzz can't find any glyphs and the opentype parser can't read the version number makes me suspect something else: harfbuzzShaper:getFace isn't actually picking up the right data - possibly not any data at all. Could you try this on the SILE REPL:

> #(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)

You should get about 362644. If you don't, have a look at:

> SILE.shaper:getFace(SILE.font.loadDefaults({}))

The code I'm suspecting, which looks very innocuous, is:

local fh,e = io.open(face.filename)
if e then SU.error("Can't open "..e) end
face.data = fh:read("*all")

pkulchenko commented 7 years ago

(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)

You should get about 362644. If you don't, have a look at:

I'm getting 91:

> #(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)
91
> SILE.shaper:getFace(SILE.font.loadDefaults({}))
{data= ☺   ↕☺  ♦  LTSH°↓ç╤  ☺,  ♠ºOS/2qócU  ╘   VVDMXgWn┌,  ♣αcmap\♫☺╫  ♫♀  ♦Çcvt ☺åG  ↕î   ,family=Gentium,filename=C:/Windows/Fonts/Gentium-R.ttf,fullname=Gentium,index=0,pointsize=12}
> SILE.font.loadDefaults({})
{direction=LTR,family=Gentium Plus,features=,hyphenchar=-,language=en,script=,size=10,style=,variant=normal,weight=400}

This looks like I only have Gentium installed and nothing else? I checked fonts.conf and the cache that was generated and it seems to report all the fonts that I have installed. Is there anything else I should check?

pkulchenko commented 7 years ago

hm, maybe it needs to read as binary file...

pkulchenko commented 7 years ago

Yep, this fixes it:

diff --git a/core/harfbuzz-shaper.lua b/core/harfbuzz-shaper.lua
index 7a5710a..504705f 100644
--- a/core/harfbuzz-shaper.lua
+++ b/core/harfbuzz-shaper.lua
@@ -53,7 +53,7 @@ SILE.shapers.harfbuzz = SILE.shapers.base {
     local face = fontconfig._face(opts)
     SU.debug("fonts", "Resolved font family "..opts.family.." -> "..(face and face.filename))
     if not face.filename then SU.error("Couldn't find face "..opts.family) end
-    local fh,e = io.open(face.filename)
+    local fh,e = io.open(face.filename, "rb")
     if e then SU.error("Can't open "..e) end
     face.data = fh:read("*all")
     return face

pkulchenko commented 7 years ago

Simon, everything is back to normal so far with the fix, but I noticed one small difference between rules shown in the generated PDF and my preview:

sile-simple-vs-pdf-rule-position

This is how the line information is shown in the output:

Draw line   126.51168904979 160.39748101271 28.346457   0.708661425
Draw line   154.85814604979 160.39748101271 28.346457   0.708661425

The lines are generated using the following: \hrule[width=10mm,height=0.25mm].

Not sure why the line is moved relative to the text, but I've been wondering which one is the correct location. Note that it's quite possible there is something wrong with my positions, but all the other objects (including the image) are properly positioned on the page.

alerque commented 7 years ago

The generated PDF is correct, your preview is off. The height parameter is supposed to specify the thickness of the line, not the raise distance relative to the baseline. Something is off there.

pkulchenko commented 7 years ago

Interesting; thank you Caleb. I'll check my calculations...

pkulchenko commented 7 years ago

@alerque, thank you for checking. It turned out that the rule calculations are correct, but I used to have a small adjustment for the text when I was trying to match the PDF output and it was only done for text. Essentially, when I remove that adjustment, everything looks the same as in the PDF, but it's somehow 2 pixels higher than it should be. Is there a way to check for exact pixel coordinates in generated PDF to match with the ones I'm calculating?

Maybe there is something wrong with the point relative to which my text is drawn? I calculate it as the following:

  local function l2d(v) return v*16/12 end
--
  setFont = function (options)
      local w, h, descent, leading = mdc:GetTextExtent("Text", font)
      fontadj = h-descent
-- 
  moveTo = function (x,y)
    cx = x
    cy = y
  end,
-- and then
  outputHbox = function (value,w)
      mdc:DrawText(value.text, l2d(cx), l2d(cy)-fontadj)

Because the text drawing in my case is relative to the top left corner, I recalculate it from the baseline origin by subtracting height-descent. The rule is in the baseline now as expected, but somehow the entire text is about 2 pixels higher than it should be (comparing to the images that are positioned in absolute coordinates on the page). Any idea why this happens?

harrysummer commented 5 years ago

I think I have done pretty much of the tasks in the main content of this issue. Checkout my fork here: https://github.com/harrysummer/sile/tree/win32

The reproducible Windows build: this is handled by using CMake and Azure Pipeline for continuous integration.
The entry script is wrapped into exe by using srlua -- a small piece of code that concats an binary with lua source code. The binary would look for the lua script in itself.
Isolate the libraries and ship distribution: all dependencies are built in the CMake script and installed along with SILE. The CI build artifacts can be downloaded here

harrysummer commented 5 years ago

Just want to share with you the workspace on Windows I am using. It is quite fascinating that we can configure to use TeXstudio for editting SILE, and then previewing side-by-side.

akavel commented 5 years ago

@harrysummer Oh wow, that is a brilliant idea!

simoncozens commented 5 years ago

Wow. This is superb.

simoncozens commented 5 years ago

Reopening because although we now have the tooling to generate a distribution, we need to:

[ ] Actually generate it and distribute it
[ ] Test it on a Windows system
[ ] Document how to get it and how to use it

harrysummer commented 5 years ago

As for actually generate it and distribute it, are we looking for something like automatically pushing the build artifact to GitHub Release? It can be done by adding an Azure Release Pipeline with GitHub Release Task.

simoncozens commented 5 years ago

Possibly, although I think what we really want is to produce an installer which places the binaries, libraries and SILE classes/packages in appropriate paths on the target system so that it can be easily used. I don't think the build artifacts alone are going to get us something that someone can install and use.

indiamcq commented 5 years ago

I have written a few Inno installer scripts, but I’d need to know where all the bits need to go.

I have looked at Nathan Miles’s implementation of SILE in Windows, but it is not an installation but a relative file structure.

Ian

On 4 Apr 2019, at 6:15 pm, Simon Cozens notifications@github.com wrote:

Possibly, although I think what we really want is to produce an installer which places the binaries, libraries and SILE classes/packages in appropriate paths on the target system so that it can be easily used. I don't think the build artifacts alone are going to get us something that someone can install and use.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simoncozens/sile/issues/410#issuecomment-479964872, or mute the thread https://github.com/notifications/unsubscribe-auth/ABb5aGPAmo3VkQzd50rZiL3WSEf5926Aks5vdiU2gaJpZM4K0grr.

harrysummer commented 5 years ago

Hello, Ian,

These scripts will definitely help. If you don't mind, could you download the latest artifact from here and test the script with the folder? https://simoncozens-github.visualstudio.com/sile/_build/results?buildId=23

After the script is finished, you could just post it here or send to me directly. And I can work it out how to put it into the CI system.

Thanks, Rui

indiamcq notifications@github.com 于2019年4月4日周四下午12:51写道：

I have written a few Inno installer scripts, but I’d need to know where all the bits need to go.

I have looked at Nathan Miles’s implementation of SILE in Windows, but it is not an installation but a relative file structure.

Ian

On 4 Apr 2019, at 6:15 pm, Simon Cozens notifications@github.com wrote:

Possibly, although I think what we really want is to produce an installer which places the binaries, libraries and SILE classes/packages in appropriate paths on the target system so that it can be easily used. I don't think the build artifacts alone are going to get us something that someone can install and use.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/simoncozens/sile/issues/410#issuecomment-479964872>, or mute the thread < https://github.com/notifications/unsubscribe-auth/ABb5aGPAmo3VkQzd50rZiL3WSEf5926Aks5vdiU2gaJpZM4K0grr .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/simoncozens/sile/issues/410#issuecomment-480039263, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKLtaN49YuOnpqKHoDaUjCX3PF1DwDrks5vdlfYgaJpZM4K0grr .

indiamcq commented 5 years ago

Rui,

I am travelling with a Mac and won’t be back to my Win machine until net week.

Ian

On 5 Apr 2019, at 10:57 pm, Rui Xia notifications@github.com wrote:

Hello, Ian,

These scripts will definitely help. If you don't mind, could you download the latest artifact from here and test the script with the folder? https://simoncozens-github.visualstudio.com/sile/_build/results?buildId=23

After the script is finished, you could just post it here or send to me directly. And I can work it out how to put it into the CI system.

Thanks, Rui

indiamcq notifications@github.com 于2019年4月4日周四下午12:51写道：

I have written a few Inno installer scripts, but I’d need to know where all the bits need to go.

I have looked at Nathan Miles’s implementation of SILE in Windows, but it is not an installation but a relative file structure.

Ian

On 4 Apr 2019, at 6:15 pm, Simon Cozens notifications@github.com wrote:

Possibly, although I think what we really want is to produce an installer which places the binaries, libraries and SILE classes/packages in appropriate paths on the target system so that it can be easily used. I don't think the build artifacts alone are going to get us something that someone can install and use.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/simoncozens/sile/issues/410#issuecomment-479964872>, or mute the thread < https://github.com/notifications/unsubscribe-auth/ABb5aGPAmo3VkQzd50rZiL3WSEf5926Aks5vdiU2gaJpZM4K0grr .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/simoncozens/sile/issues/410#issuecomment-480039263, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKLtaN49YuOnpqKHoDaUjCX3PF1DwDrks5vdlfYgaJpZM4K0grr .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simoncozens/sile/issues/410#issuecomment-480419602, or mute the thread https://github.com/notifications/unsubscribe-auth/ABb5aPzkgLCdEpFrRHz4E8hktZqsqmS8ks5vd7iugaJpZM4K0grr.

alerque commented 5 years ago

@harrysummer At this stage is there any point at all in keeping code oriented towards supporting mingw/msys? I'm working on the build/release process and there is quite a bit of cruft tucked around there and there that I don't think actually does anything at the present time, but I don't want to lobotomize it if that's a remotely feasible way to move.

indiamcq commented 5 years ago

Well I finally got around to making an Inno installer script using @harrysummer Win build and some other bits from @simoncozens.

I installed that and it did not work. Do I have to run it in a msys shell?

; Script generated by the Inno Script Studio Wizard.
; SEE THE DOCUMENTATION FOR DETAILS ON CREATING INNO SETUP SCRIPT FILES!
#define AppVer "0.9.5.1"
#define icon "sile-logo.ico"
#define silesource "D:\Publishing\github\sile"
#define silewinsource "D:\Publishing\github\SILE-windows"

[Setup]

AppName=Simon's Improved Layout Engine
AppVerName=SILE {#AppVer}
VersionInfoVersion={#AppVer}
OutputBaseFilename=SILE-{#AppVer}-Windows-Setup

AppVersion={#AppVer}

DefaultDirName={pf}\SILE
DefaultGroupName=Publishing
OutputDir=output

SolidCompression=yes
Compression=lzma2

[Files]
Source: "{#silewinsource}\*.*"; DestDir: "{app}"; Flags: recursesubdirs
Source: "LICENSE.txt" ; DestDir: "{app}";
Source: "{#silesource}\examples\*.*" ; DestDir: "{app}\examples"; Flags: recursesubdirs 
Source: "{#silesource}\tests\*.*" ; DestDir: "{app}\tests"; Flags: recursesubdirs 

[Icons]
Name: "{group}\SILE"; Filename: "explorer {app}"; IconFilename: "{app}\{#icon}"
Name: "{group}\Uninstallers\SILE Uninstall"; Filename: "{uninstallexe}"

harrysummer commented 5 years ago

@harrysummer At this stage is there any point at all in keeping code oriented towards supporting mingw/msys? I'm working on the build/release process and there is quite a bit of cruft tucked around there and there that I don't think actually does anything at the present time, but I don't want to lobotomize it if that's a remotely feasible way to move.

When I was designing the CMake-based building system on Windows platform, I didn't consider supporting msys/mingw. As a compatible layer, msys/mingw does a good job in integrating Linux programs to Windows. However, if we are able to make the program building with Visual Studio, this would be the ultimate solution for Windows. Given that Visual Studio is compatible with GNU C++ better and better, I think there is no need for maintaining a msys/mingw build on Windows if we are already on Visual Studio.

harrysummer commented 5 years ago

Well I finally got around to making an Inno installer script using @harrysummer Win build and some other bits from @simoncozens.

I installed that and it did not work. Do I have to run it in a msys shell?

Thanks for your script. Let me try it without msys. If I understand it correctly, I just need to give Inno Studio this script along with the build and it should generate something like sile-setup.exe.

harrysummer commented 5 years ago

Well I finally got around to making an Inno installer script using @harrysummer Win build and some other bits from @simoncozens. I installed that and it did not work. Do I have to run it in a msys shell?

Thanks for your script. Let me try it without msys. If I understand it correctly, I just need to give Inno Studio this script along with the build and it should generate something like sile-setup.exe.

FYI, I made some progress on the Windows installer in https://github.com/simoncozens/sile/pull/650. You can try the installer in artifact of the PR build here if you have a Windows machine. Based on @indiamcq's script, I added several new features:

Add optional task to add SILE installation directory to PATH
Add optional task to associate *.sil file with SILE
Add optional task to create a desktop shortcut to open SILE command line
Add more start menu items including opening SILE commandline, and opening LUA
Show license before installation
Show "Launch SILE commandline" option after finishing the installation
Get the version from git describe --tags

goyalyashpal commented 2 years ago

I don't think the build artifacts alone are going to get us something that someone can install and use @simoncozens at comment

so. what have we got now? have we got a portlable zip which i can extract and run it? i'd be okay with it even if it doen't install and even if i may have to configure some folder locations thereafter somewhere if that's what'd be needed.

The Resolved one

> * We just made this harder by requiring ICU @simoncozens at [comment](#issue-189858023) what's ICU? International Componenets for Unicode? https://icu.unicode.org **edit:** yeah, verified from the link : [ICU libraries](http://icu-project.org/) in [readme](https://github.com/sile-typesetter/sile#from-source)

alerque commented 2 years ago

so. what have we got now? have we got a portlable zip which i can extract and run it?

We had something like that, but at this point I don't know if the things inside the ZIP are usabel.

It is being built in CI by Azure and the builds are still running (successfully) for most commits and PRs to the project, but none of us are in a position to evaluate them and figure out what works, what doesn't, what needs changing, etc.

You can see recent builds here: https://dev.azure.com/sile-typesetter/sile/_build

It claims to have built an artifact, but what is or isn't in that artifact at this point is anybody's guess.

KrasnayaPloshchad commented 1 year ago

The idea behind a C wrapper (which I've now written) ties into my end goal for distribution on Windows: rather than installing Cygwin and Lua and Luarocks and everything else and then SILE, the installer just unpacks a punch of stuff into C:\Program Files\SILE, where all the dependency libraries - Mingw, Freetype, Harfbuzz, ICU, Lua and so on - are stashed away and nobody needs to care that they're running Lua and half of Unix when they type "sile foo.sil".

This could be easily done via NSIS.

alerque commented 1 year ago

I have no idea what NSIS is, but sure knock yourself out. I'm happy to facilitate contributions from Windows folks making their life easier, I just don't have the interest or time to make it happen myself.

Also medium term there is a very solid chance one option for running SILE will be as a single Rust binary with an embeded Lua interpreter and all the default support files baked in. See #1762 for current progress on that front. Getting the Rust CLI and embeded Lua interpreter is already fully working, but getting everything including the assorted C modules embeded is a bit trickier.

KrasnayaPloshchad commented 1 year ago

I have no idea what NSIS is, but sure knock yourself out. I'm happy to facilitate contributions from Windows folks making their life easier, I just don't have the interest or time to make it happen myself.

That’s shocking. Right now, you can acknowledge NSIS here, which give great help for this issue: https://nsis.sourceforge.io/Main_Page

Omikhleia commented 1 year ago

Why would that be shocking?

AFAIC, I've used NSIS since the early 2000s for some job-related projects. Packaging is only a small part of the problem, though -- last time I could use a Windows machine (more than one year ago), the artifacts from the above-mentioned Azure CI build were lacking things, and I couldn't make them work. I didn't investigate a lot, but it all boils down to what @alerque already said:

It claims to have built an artifact, but what is or isn't in that artifact at this point is anybody's guess.

Someone interested and thinking it's an easy topic is surely welcome to help :smile_cat:

sile-typesetter / sile

Windows binary distribution #410

(SILE.shaper:getFace(SILE.font.loadDefaults({})).data)