search

lyze / xetex-js

Javascript port of XeTeX
Other
13 stars 3 forks source link

Pass CCexe=gcc to freetype make command #4

Open vadimkantorov opened 4 years ago

vadimkantorov commented 4 years ago

This seems to remove the need for copying apinames from native build and maybe makes the native build less necessary

vadimkantorov commented 4 years ago

Passing --disable-shared to libexpat configure was also important to bypass duplicate memcpy errors.

Now it's failing at xdvipdfmx with:

error: undefined symbol: getpass (referenced by top-level compiled C/C++ code)
warning: Link with `-s LLD_REPORT_UNDEFINED` to get more information on undefined symbols
warning: To disable errors for undefined symbols use `-s ERROR_ON_UNDEFINED_SYMBOLS=0`
warning: _getpass may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: popen (referenced by top-level compiled C/C++ code)
warning: _popen may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
Error: Aborting compilation due to previous errors
emcc: error: '/mnt/c/Users/user/emsdk/node/12.18.1_64bit/bin/node /mnt/c/Users/user/emsdk/upstream/emscripten/src/compiler.js /tmp/tmpbhq191nf.txt' failed (1)
make[6]: *** [xdvipdfmx] Error 1
make[5]: *** [all-recursive] Error 1
make[4]: *** [all] Error 2
make[3]: *** [subtexk] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
Makefile:329: recipe for target 'build-js-xetex-toplevel.stamp' failed

I'm trying to repro the build using recent emscripten

getpass issue seems related to http://project.ktug.org/dvipdfmx/mailman/dvipdfmx/2011-February/000250.html

vadimkantorov commented 4 years ago

There was also a suspicious configure: WARNING: Sorry, neither ApplicationServices framework nor fontconfig library: disabling xetex, possibly during some web2c run.

vadimkantorov commented 4 years ago

To bypass the undefined functions, -s ERROR_ON_UNDEFINED_SYMBOLS=0 should be added to EMCC_FLAGS inside build-js-xetex-toplevel.stamp

vadimkantorov commented 4 years ago

Then getting this:

make: make -C build-js/build-xetex/texk/web2c/ -o ctangle -o otangle -o tangle -o tangleboot -o tie -o web2c/fixwrites -o web2c/makecpool -o web2c/splitup -o web2c/web2c xetex
clang-12: warning: argument unused during compilation: '-ffunction-sections' [-Wunused-command-line-argument]
clang-12: warning: argument unused during compilation: '-fdata-sections' [-Wunused-command-line-argument]
clang-12: warning: argument unused during compilation: '-D _REENTRANT' [-Wunused-command-line-argument]
clang-12: warning: argument unused during compilation: '-D U_ENABLE_DYLOAD=0' [-Wunused-command-line-argument]
clang-12: warning: argument unused during compilation: '-D U_HAVE_POPEN=0' [-Wunused-command-line-argument]
clang-12: warning: argument unused during compilation: '-D U_ATTRIBUTE_DEPRECATED=' [-Wunused-command-line-argument]
clang-12: warning: argument unused during compilation: '-D PIC' [-Wunused-command-line-argument]
./out/tmp/icudt51l_dat.s:2:20: error: Expected ,, instead got: -
        .section .note.GNU-stack,"",%progbits
                          ^
./out/tmp/icudt51l_dat.s:3:18: error: Expected ,, instead got:

        .section .rodata
                        ^
./out/tmp/icudt51l_dat.s:5:20: error: Expected label,@type declaration, got: %
        .type icudt51_dat,%object
                          ^
emcc: error: '/mnt/c/Users/user/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -ffunction-sections -fdata-sections -D_REENTRANT -DU_ENABLE_DYLOAD=0 -DU_HAVE_POPEN=0 -DU_ATTRIBUTE_DEPRECATED= -g -O3 -Wno-error=implicit-function-declaration -c -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/icu/icu-51.1/source/common -I../common -DPIC -fPIC -o ./out/tmp/icudt51l_dat.o ./out/tmp/icudt51l_dat.s -c -o ./out/tmp/icudt51l_dat.o -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr' failed (1)
-- return status = 256
Error creating with assembly code. Failed command: ����
Error generating assembly code for data.

This is a false alert, since it happens before replacing icu binaries

vadimkantorov commented 4 years ago

So it seems it's trying to use inline assembly duing non-native build and fails

vadimkantorov commented 4 years ago

Also there's expected error ../bin/icupkg: line 1: syntax error near unexpected token('`, during first attempt before replacing icu binaries

vadimkantorov commented 4 years ago 
In file included from /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/Engine.cpp:120:
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2203:10: error: constant expression evaluates to 69786 which cannot be narrowed to type 'UInt16' (aka 'unsigned short') [-Wc++11-narrowing]
        {0x0000,0x1109A,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2203:10: note: insert an explicit cast to silence this issue
        {0x0000,0x1109A,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
                static_cast<UInt16>( )
In file included from /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/Engine.cpp:120:
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2203:10: error: constant expression evaluates to 69786 which cannot be narrowed to type 'UInt16' (aka 'unsigned short') [-Wc++11-narrowing]
        {0x0000,0x1109A,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2203:10: note: insert an explicit cast to silence this issue
        {0x0000,0x1109A,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
                static_cast<UInt16>( )
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2204:10: error: constant expression evaluates to 69788 which cannot be narrowed to type 'UInt16' (aka 'unsigned short') [-Wc++11-narrowing]
        {0x0000,0x1109C,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2204:10: note: insert an explicit cast to silence this issue
        {0x0000,0x1109C,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
                static_cast<UInt16>( )
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2205:10: error: constant expression evaluates to 69803 which cannot be narrowed to type 'UInt16' (aka 'unsigned short') [-Wc++11-narrowing]
        {0x0000,0x110AB,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/libs/teckit/TECkit-2.5.3/source/NormalizationData.c:2205:10: note: insert an explicit cast to silence this issue
        {0x0000,0x110AB,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000,0x0000},
                ^~~~~~~
                static_cast<UInt16>( )

This seems the only error that stays. And this makes sense, these numbers don't fit into UINT16

vadimkantorov commented 4 years ago

This issue was brought up in gentoo as well: https://bugs.gentoo.org/594660

vadimkantorov commented 4 years ago

Basically TECkit 2.5.3 is buggy with this respect and an update is needed. I'll first stub out these constants and then will try to bump the version

vadimkantorov commented 4 years ago

Then it seems that passed fontconfig path is not propagated to xetex compilation:

em++: error: '/mnt/c/Users/user/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=0 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -nostdsysteminc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libcxx -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libcxxabi/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/compat -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libc/musl/arch/emscripten -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/local/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SSE -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/compiler-rt/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libunwind/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/cache/wasm/include -DEMSCRIPTEN -fignore-exceptions -DHAVE_CONFIG_H -I. -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c -I./w2c -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/texk -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir -DXETEX_BUILD_DATE="2020082517" -DU_STATIC_IMPLEMENTATION -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/icu/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/freetype2/freetype2 -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/freetype2 -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/teckit/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/harfbuzz/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/graphite2/include -DGRAPHITE2_STATIC -DXETEX_OTHER -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/poppler -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/poppler/goo -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/poppler/poppler -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/libpng/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/zlib/include -g -O2 -MT xetexdir/libxetex_a-XeTeXFontMgr.o -MD -MP -MF xetexdir/.deps/libxetex_a-XeTeXFontMgr.Tpo -c -o xetexdir/libxetex_a-XeTeXFontMgr.o /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir/XeTeXFontMgr.cpp -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SDL -c -o xetexdir/libxetex_a-XeTeXFontMgr.o -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr' failed (1)

After patching this, it fails to find w2c/c-auto.h:

clang++ -target wasm32-unknown-emscripten -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=0 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -nostdsysteminc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libcxx -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libcxxabi/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/compat -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libc/musl/arch/emscripten -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/local/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SSE -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/compiler-rt/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libunwind/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/cache/wasm/include -DEMSCRIPTEN -fignore-exceptions -DHAVE_CONFIG_H -I. -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c -I./w2c -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/texk -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk -I/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir -DXETEX_BUILD_DATE="2020082517" -DU_STATIC_IMPLEMENTATION -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/icu/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/freetype2/freetype2 -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/freetype2 -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/teckit/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/harfbuzz/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/graphite2/include -DGRAPHITE2_STATIC -DXETEX_OTHER -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/poppler -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/poppler/goo -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/poppler/poppler -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/libpng/include -I/mnt/c/Users/user/texet/xetex-js/build-js/build-xetex/libs/zlib/include -g -O2 -MT xetexdir/libxetex_a-XeTeXFontInst.o -MD -MP -MF xetexdir/.deps/libxetex_a-XeTeXFontInst.Tpo -c -o xetexdir/libxetex_a-XeTeXFontInst.o /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir/XeTeXFontInst.cpp -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SDL -c -o xetexdir/libxetex_a-XeTeXFontInst.o -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr
In file included from /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir/XeTeXFontInst.cpp:42:
In file included from /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir/XeTeXFontInst.h:51:
In file included from /mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir/XeTeXFontMgr.h:47:
/mnt/c/Users/user/texet/xetex-js/build-sources/xetex-0.9999.3/source/texk/web2c/xetexdir/XeTeX_ext.h:39:10: fatal error: 'w2c/c-auto.h' file not found
#include <w2c/c-auto.h>  /* needed for SIZEOF_LONG and NO_DUMP_SHARE */
         ^~~~~~~~~~~~~~
1 error generated.
lyze commented 4 years ago

Ah thanks for investigating! Sadly, this repository has bitrotted. The artifacts branch has some older compiled binaries from back when things worked. I'll try to take some time this weekend to take a look.

vadimkantorov commented 4 years ago

I'm advancing towards building xetex + dvipdfm-x from scratch from recent TexLive repo: https://tex.stackexchange.com/questions/560046/how-to-build-xetex-and-dvipdfmx-from-source-texlive-2020

I currently have a proof of concept of in-browser-only tex editor (not new, was already tried by SwiftLatex and texlive.js and https://people.math.osu.edu/fowler.291/latex/ - I also try to hack around GitHub API for the usecase of editing CVs or small-ish repos with scientific papers):

My frankenstein (Monaco + xterm.js + pdf.js + emscripten): image

Unfortunately I don't have experience with web programming, so all this is in one https://github.com/vadimkantorov/texet/blob/master/index.html (so that it's easily hackable). Unsolved issues: how to import TypeScript module in JavaScript (xterm.js Fit Addon); simple split-pane UI to resolve issues of pdf.js and xterm.js overlaying everything; xetex + bibtex + automatically downloading and caching texmf files in a worker thread (similar to your repo or https://github.com/manuels/texlive.js/)

vadimkantorov commented 4 years ago

I managed to compile xetex and dvipdfm-x from TexLive (for now with native toolchain). However, creating a format file fails:

wget http://mirrors.ctan.org/macros/latex/base.zip
unzip -o base.zip

TEXINPUTS=base//: $XELATEX_EXE -ini -etex -output-directory=base unpack.ins
# works

TEXINPUTS=base//: $XELATEX_EXE -ini -etex -output-directory=base latex.ltx 
# fails as below
Local config file preload.cfg used

=====================================
(base/preload.cfg (base/preload.ltx
kpathsea: Running mktextfm cmex10
mktextfm: No such file or directory
kpathsea: Appending font creation commands to missfont.log.

! Font \OMX/cmex/m/n/10=cmex10 not loadable: Metric (TFM) file or installed fon
t not found.
<to be read again>
                   \relax
l.41 ...ame OMX/cmex/m/n/10\endcsname=cmex10\relax
vadimkantorov commented 4 years ago

Do you make use of native INSTALL_TL_UNX_ARCHIVE to get around this?

lyze commented 4 years ago

It looks like in this case that's an error specific to the native texlive environment. It's looking for the cmex10 font but it's not found. If you add an entry around here to configure texlive to download an additional font package containing cmex10, then that would work. You may need to run the interactive texlive installation UI to figure out what packages are available.

vadimkantorov commented 4 years ago

You are right, I think (I don't have a native install except for installer used as part of Makefile). I tried the following (texlive.js version which is slightly simpler), but still fails (I'll try again!). I think I need to pass it some paths to my new native install.

How does xelatex -ini command is supposed to know about the path of native install root? TEXINPUTS seems to pass the path to base. Do I need to set some other env variable?

Is profile file only destined for the installer?

TEXLIVE_BASE_URL=http://mirrors.ctan.org/macros/latex/base.zip
TEXLIVE_INSTALLER_URL=http://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz

mkdir -p texlive
echo selected_scheme scheme-basic > texlive/profile.input
echo TEXDIR $PWD/texlive >> texlive/profile.input
echo TEXMFLOCAL $PWD/texlive/texmf-local >> texlive/profile.input
echo TEXMFSYSVAR $PWD/texlive/texmf-var >> texlive/profile.input
echo TEXMFSYSCONFIG $PWD/texlive/texmf-config >> texlive/profile.input
echo TEXMFVAR $PWD/home/texmf-var >> texlive/profile.input

wget $TEXLIVE_INSTALLER_URL
pushd texlive
tar xzvf ../install-tl-unx.tar.gz
./install-tl-*/install-tl -profile profile.input
rm -rf bin readme* tlpkg install* *.html texmf-dist/doc texmf-var/web2c
echo "Done! Please run 'make texlive.lst' now!"
popd

wget $TEXLIVE_BASE_URL
mkdir -p latex_format
pushd latex_format
unzip -o ../base.zip
pushd base
$XELATEX_EXE -ini -etex unpack.ins
$XELATEX_EXE -ini -etex latex.ltx
vadimkantorov commented 4 years ago 
vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/blfstexlive/texlive$ ls
LICENSE.CTAN  README             install-tl-20200826  release-texlive.txt  texmf-dist   texmf-var  texmfcnf.lua
LICENSE.TL    README.usergroups  profile.input        texmf-config         texmf-local  texmf.cnf

vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/blfstexlive/texlive$ find texmf-dist -name cm
texmf-dist/fonts/afm/public/amsfonts/cm
texmf-dist/fonts/map/dvips/cm
texmf-dist/fonts/pk/ljfour/public/cm
texmf-dist/fonts/source/public/cm
texmf-dist/fonts/tfm/public/cm
texmf-dist/fonts/type1/public/amsfonts/cm

So in theory fonts are present. How do I point newly built $XELATEX_EXE to the prepared texlive root?

lyze commented 4 years ago

I believe TFMFONTS is the variable to use to find tfm fonts, and TTFONTS for truetype.

I think setting TEXMFDIST would work. [1]

[1] https://wiki.archlinux.org/index.php/TeX_Live#texmf_trees_and_Kpathsea [2] https://tex.stackexchange.com/questions/507299/why-doesnt-texinputs-work-for-fonts [3] https://tex.stackexchange.com/questions/224867/fontspec-not-searching-in-texinputs

vadimkantorov commented 4 years ago

The full script I currently execute is here: https://gist.github.com/vadimkantorov/501634e4f0c93b2940b43aea071dff03 I did more research in https://tex.stackexchange.com/questions/560336/how-to-create-a-latex-format-file-without-an-existing-texlive-install and it seems that:

somehow kpathsea cannot find mktextfm which is in $TEXLIVE_SOURCE_DIR/texk/texlive/linked_scripts/texlive. If I add it to PATH, then ./prefix/bin/kpsewhich cannot find mktex.opt which is in ./texlive/texmf-dist/web2c/mktex.opt. ./prefix/share/texmf-dist/web2c/texmf.cnf does not know anything about ./texlive. How do I make sure that ./prefix/bin/kpsewhich can discover ./texlive and produce format files?

So yes, the primary problem seems to be in discovery of tools' paths. I'll try TEXMFDIST

vadimkantorov commented 4 years ago

export TEXMFDIST=$PWD/texlive/texmf-dist does not help, ksewhich still cannot discover texlive/texmf-dist/web2c/mktex.opt

vadimkantorov commented 4 years ago

Here is strace of kpsewhich failing:

fonts/tfm//:!!/mnt/c/Users/user/blfstexlive/texmf-dist/fonts/tfm//:/home/vadimkantorov/.texlive2020/texmf-var/fonts/tfm//
vadimkantorov@DESKTOP-4UF8FID:/mnt/c/Users/user/blfstexlive$ strace -fff ./prefix/bin/kpsewhich --format='web2c files' mktex.opt
execve("./prefix/bin/kpsewhich", ["./prefix/bin/kpsewhich", "--format=web2c files", "mktex.opt"], 0x7fffcbf94238 /* 21 vars */) = 0
brk(NULL)                               = 0x7fffd1827000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=68021, ...}) = 0
mmap(NULL, 68021, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbb2cc05000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2030544, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbb2cc00000
mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fbb2c400000
mprotect(0x7fbb2c5e7000, 2097152, PROT_NONE) = 0
mmap(0x7fbb2c7e7000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7fbb2c7e7000
mmap(0x7fbb2c7ed000, 15072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fbb2c7ed000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7fbb2cc01540) = 0
mprotect(0x7fbb2c7e7000, 16384, PROT_READ) = 0
mprotect(0x7fbb2d016000, 4096, PROT_READ) = 0
mprotect(0x7fbb2ca27000, 4096, PROT_READ) = 0
munmap(0x7fbb2cc05000, 68021)           = 0
brk(NULL)                               = 0x7fffd1827000
brk(0x7fffd1848000)                     = 0x7fffd1848000
lstat(".", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("./prefix", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("./prefix/bin", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("./prefix/bin/kpsewhich", {st_mode=S_IFREG|0777, st_size=517032, ...}) = 0
getcwd("/mnt/c/Users/user/blfstexlive", 4097) = 30
stat("/mnt/c/Users/user/blfstexlive/prefix/bin", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
access("/mnt/c/Users/user/blfstexlive/prefix/bin/texmf.cnf", R_OK) = 0
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/texmf.cnf", {st_mode=S_IFREG|0777, st_size=39451, ...}) = 0
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/share/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/share/texmf-dist/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/share/texmf/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/texmf-dist/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/bin/texmf/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
access("/mnt/c/Users/user/blfstexlive/prefix/texmf.cnf", R_OK) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/share/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/share/texmf-dist/web2c", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
access("/mnt/c/Users/user/blfstexlive/prefix/share/texmf-dist/web2c/texmf.cnf", R_OK) = 0
stat("/mnt/c/Users/user/blfstexlive/prefix/share/texmf-dist/web2c/texmf.cnf", {st_mode=S_IFREG|0777, st_size=39451, ...}) = 0
stat("/mnt/c/Users/user/blfstexlive/prefix/share/texmf/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/texmf-dist/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/prefix/texmf/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
access("/mnt/c/Users/user/blfstexlive/texmf.cnf", R_OK) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/share/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/share/texmf-dist/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/share/texmf/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/texmf-local/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/texmf-dist/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/texmf/web2c", 0x7fffd87dfe30) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/mnt/c/Users/user/blfstexlive/prefix/bin/texmf.cnf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=39451, ...}) = 0
read(3, "% original texmf.cnf -- runtime "..., 4096) = 4096
read(3, "aces.\nTEXMFAUXTREES = {}\n\n% List"..., 4096) = 4096
read(3, "st      = $TEXMFDOTDIR;$TEXMF/{t"..., 4096) = 4096
read(3, "//\n\n% XeTeX.\nTEXINPUTS.xelatex  "..., 4096) = 4096
read(3, "TEXPOOL}\nMPPOOL = ${TEXPOOL}\n\n% "..., 4096) = 4096
read(3, "IR;$TEXMF/metapost/support\n\n% Fo"..., 4096) = 4096
read(3, "IR;$TEXMF/scripts/{$progname,$en"..., 4096) = 4096
read(3, "nds are allowed.  Although this\n"..., 4096) = 4096
read(3, "ost = eptex\n\n% These variables s"..., 4096) = 4096
read(3, "curios/page.php/8191.html\nnest_s"..., 4096) = 2587
read(3, "", 4096)                       = 0
close(3)                                = 0
openat(AT_FDCWD, "/mnt/c/Users/user/blfstexlive/prefix/share/texmf-dist/web2c/texmf.cnf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=39451, ...}) = 0
read(3, "% original texmf.cnf -- runtime "..., 4096) = 4096
read(3, "aces.\nTEXMFAUXTREES = {}\n\n% List"..., 4096) = 4096
read(3, "st      = $TEXMFDOTDIR;$TEXMF/{t"..., 4096) = 4096
read(3, "//\n\n% XeTeX.\nTEXINPUTS.xelatex  "..., 4096) = 4096
read(3, "TEXPOOL}\nMPPOOL = ${TEXPOOL}\n\n% "..., 4096) = 4096
read(3, "IR;$TEXMF/metapost/support\n\n% Fo"..., 4096) = 4096
read(3, "IR;$TEXMF/scripts/{$progname,$en"..., 4096) = 4096
read(3, "nds are allowed.  Although this\n"..., 4096) = 4096
read(3, "ost = eptex\n\n% These variables s"..., 4096) = 4096
read(3, "curios/page.php/8191.html\nnest_s"..., 4096) = 2587
read(3, "", 4096)                       = 0
close(3)                                = 0
stat("/mnt/c/Users/user/texmf-local", 0x7fffd87dfd90) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/texmf-config", 0x7fffd87dfd90) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/texmf-var", 0x7fffd87dfd90) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Users/user/blfstexlive/texmf-dist", 0x7fffd87dfd90) = -1 ENOENT (No such file or directory)
mmap(NULL, 516096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbb2cb80000
munmap(0x7fbb2cb80000, 516096)          = 0
brk(0x7fffd1869000)                     = 0x7fffd1869000
stat("/home/vadimkantorov/.texlive2020/texmf-config/web2c", 0x7fffd87dfed0) = -1 ENOENT (No such file or directory)
stat("/home/vadimkantorov/.texlive2020/texmf-var/web2c", 0x7fffd87dfed0) = -1 ENOENT (No such file or directory)
stat("/home/vadimkantorov/texmf/web2c", 0x7fffd87dfed0) = -1 ENOENT (No such file or directory)
exit_group(1)                           = ?
+++ exited with 1 +++
vadimkantorov commented 4 years ago

Yay! There were some stupid path bug on my side. I managed to get a latex format file

vadimkantorov commented 4 years ago

Complete script here: https://gist.github.com/vadimkantorov/501634e4f0c93b2940b43aea071dff03

vadimkantorov commented 4 years ago

Will now try your next steps after the format file is done

vadimkantorov commented 4 years ago

Would you have an advice on how to share FS between several Emscripten MODULARIZE'd programs? https://github.com/emscripten-core/emscripten/issues/12074

E.g. I first want to not deal with objects and have a common FS and manually launch xetex / bibtex (if needed) / dvipdfmx modules (for now without workers). For that it'd be good to have them a shared MEMFS.

lyze commented 4 years ago

I see. I read through the issues referenced. What we really need is a pluggable filesystem or an abstraction to hack away on emscripten-core/emscripten/issues/777.

Depending on the extent of the data, maybe a workaround is to just copy what you need among the modularized filesystems. The common shared files, like the tex distribution itself, would still be accessed lazily with XHR. This may mean more memory use, but the common requests should go to the browser cache, so maybe things would be just fine.

vadimkantorov commented 4 years ago

maybe a workaround is to just copy what you need among the modularized filesystems.

Makes sense! At least until there is some feedback in emscripten repo!

So will now try to build xetex/dvipdfmx with emscripten. Will let you know how it builds now from source (if the issues you described still persist)

vadimkantorov commented 4 years ago

Do I understand correctly that your Makefile does not build a separate kpathsea binary? And somehow builds it into xetex itself?

vadimkantorov commented 4 years ago

So emconfigure of TexLive (from trunk) succeeded. Making currently fails in upmendex at https://github.com/TeX-Live/texlive-source/blob/trunk/texk/upmendex/configure#L3081

Somehow upmendex tries to check the compiler and doesn't detect the .js output (even if it's not required, TexLive still tries to configure everything and does it in making time!). Can't figure out a way to patch it out :( or set some config.site / config.cache presets. Would be glad if you have any advice!

vadimkantorov commented 4 years ago

I managed to continue the build by removing upmendex from texlive-source/texlive-build-wasm/texk/Makefile from CONF_SUBDIRS variable, but definitely a cleaner hack would be great

vadimkantorov commented 4 years ago

Official TexLive building manual suggests actually removing unneeded subdirs prior to configure:

Some libraries and programs require C++11. If you want to build with an older compiler lacking such support, you need to (re)move those source directories; specifying --disable for them does not suffice, unfortunately. Specifically, before running configure:

rm -rf libs/icu libs/poppler libs/graphite2 texk/dvisvgm

But a hack to simply bypass the compiler check in upmendex would be neater (at least, at this point) I guess

vadimkantorov commented 4 years ago

Getting this when compiling fontconfig...

    CCLD     libfontconfig.la

*** Warning: Linking the shared library libfontconfig.la against the
*** static library /mnt/c/Users/user/blfstexlive/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/freetype2/libfreetype.a is not portable!
make[3]: Leaving directory '/mnt/c/Users/user/blfstexlive/fontconfig-2.13.1/build-wasm/src'
make[2]: Leaving directory '/mnt/c/Users/user/blfstexlive/fontconfig-2.13.1/build-wasm/src'
Making all in fc-cache
make[2]: Entering directory '/mnt/c/Users/user/blfstexlive/fontconfig-2.13.1/build-wasm/fc-cache'
  CCLD     fc-cache
error: undefined symbol: FT_Done_Face (referenced by top-level compiled C/C++ code)
warning: Link with `-s LLD_REPORT_UNDEFINED` to get more information on undefined symbols
warning: To disable errors for undefined symbols use `-s ERROR_ON_UNDEFINED_SYMBOLS=0`
warning: _FT_Done_Face may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Done_FreeType (referenced by top-level compiled C/C++ code)
warning: _FT_Done_FreeType may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Done_MM_Var (referenced by top-level compiled C/C++ code)
warning: _FT_Done_MM_Var may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_Advance (referenced by top-level compiled C/C++ code)
warning: _FT_Get_Advance may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_BDF_Property (referenced by top-level compiled C/C++ code)
warning: _FT_Get_BDF_Property may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_First_Char (referenced by top-level compiled C/C++ code)
warning: _FT_Get_First_Char may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_MM_Var (referenced by top-level compiled C/C++ code)
warning: _FT_Get_MM_Var may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_Next_Char (referenced by top-level compiled C/C++ code)
warning: _FT_Get_Next_Char may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_PS_Font_Info (referenced by top-level compiled C/C++ code)
warning: _FT_Get_PS_Font_Info may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_Postscript_Name (referenced by top-level compiled C/C++ code)
warning: _FT_Get_Postscript_Name may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_Sfnt_Name (referenced by top-level compiled C/C++ code)
warning: _FT_Get_Sfnt_Name may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_Sfnt_Name_Count (referenced by top-level compiled C/C++ code)
warning: _FT_Get_Sfnt_Name_Count may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_Sfnt_Table (referenced by top-level compiled C/C++ code)
warning: _FT_Get_Sfnt_Table may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Get_X11_Font_Format (referenced by top-level compiled C/C++ code)
warning: _FT_Get_X11_Font_Format may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Init_FreeType (referenced by top-level compiled C/C++ code)
warning: _FT_Init_FreeType may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Load_Glyph (referenced by top-level compiled C/C++ code)
warning: _FT_Load_Glyph may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Load_Sfnt_Table (referenced by top-level compiled C/C++ code)
warning: _FT_Load_Sfnt_Table may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_New_Face (referenced by top-level compiled C/C++ code)
warning: _FT_New_Face may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Select_Charmap (referenced by top-level compiled C/C++ code)
warning: _FT_Select_Charmap may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Select_Size (referenced by top-level compiled C/C++ code)
warning: _FT_Select_Size may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
error: undefined symbol: FT_Set_Var_Design_Coordinates (referenced by top-level compiled C/C++ code)
warning: _FT_Set_Var_Design_Coordinates may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
Error: Aborting compilation due to previous errors
emcc: error: '/mnt/c/Users/user/emsdk/node/12.18.1_64bit/bin/node /mnt/c/Users/user/emsdk/upstream/emscripten/src/compiler.js /tmp/tmpk2dl1nio.txt' failed (1)
vadimkantorov commented 4 years ago

It seems that it can't find freetype during linking... I'm currently building fontconfig as:

EM_PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig EMCONFIGURE_JS=2 emconfigure ../configure \
    --cache-file $ROOT/config-fontconfig-wasm.cache \
    --prefix=$PREFIX \
    --enable-static \
    --disable-shared \
    --disable-docs \
    FREETYPE_CFLAGS="-g -O3 -I$TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/ -I$TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/freetype2/" \
    FREETYPE_LIBS=$TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/libfreetype.a
vadimkantorov commented 4 years ago

I guess it's issues about:

# Using libfontconfig.a mysteriously fails a configure check (and final link
# step) with:
# AssertionError: Failed to run LLVM optimizations:
LIB_FONTCONFIG = $(FONTCONFIG_BUILD_DIR)src/.libs/libfontconfig.so

For now I tried to build fontconfig only statically

vadimkantorov commented 4 years ago

I managed to compile fontconfig statically:

FONTCONFIG_SOURCE_URL=https://www.freedesktop.org/software/fontconfig/release/fontconfig-2.13.1.tar.gz
wget --no-clobber $FONTCONFIG_SOURCE_URL
tar -xf $(basename $FONTCONFIG_SOURCE_URL)
FONTCONFIG_SOURCE_NAME=$(basename $FONTCONFIG_SOURCE_URL .tar.gz)

mkdir -p $FONTCONFIG_SOURCE_NAME/build-wasm
cd $FONTCONFIG_SOURCE_NAME/build-wasm

patch -d .. -Np1 -i $ROOT/0002-fix-fcstats-emscripten.patch
echo 'all:' > ../test/Makefile.in

FREETYPE_CFLAGS="-I$TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/ -I$TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/freetype2"
FREETYPE_LIBS="-L$TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/ -lfreetype"

CFLAGS=-Duuid_generate_random=uuid_generate EM_PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig EMCONFIGURE_JS=2 FREETYPE_CFLAGS="$FREETYPE_CFLAGS" FREETYPE_LIBS="$FREETYPE_LIBS" emconfigure ../configure \
    --cache-file $ROOT/config-fontconfig-wasm.cache \
    --prefix=$PREFIX \
    --enable-static \
    --disable-shared \
    --disable-docs

emmake make $MAKEFLAGS
emmake make $MAKEFLAGS install
vadimkantorov commented 4 years ago

So the latest problem is that xetex keeps rebuilding tangle, even after I replace it with a native program

vadimkantorov commented 4 years ago

Currently I build it like this:

for f in ctangle otangle tangle tangleboot tie ctangleboot; do
    cp --preserve=mode $TEXLIVE_SOURCE_DIR/texlive-build-native/texk/web2c/$f  $TEXLIVE_SOURCE_DIR/texlive-build-wasm/texk/web2c/
done
for f in fixwrites makecpool splitup web2c; do
    cp --preserve=mode $TEXLIVE_SOURCE_DIR/texlive-build-native/texk/web2c/web2c/$f  $TEXLIVE_SOURCE_DIR/texlive-build-wasm/texk/web2c/web2c/
done
cd texk/web2c

emmake make $MAKEFLAGS xetex
cp --preserve=mode $TEXLIVE_SOURCE_DIR/texlive-build-native/libs/freetype2/ft-build/apinames $TEXLIVE_SOURCE_DIR/texlive-build-wasm/libs/freetype2/ft-build

for f in ctangle otangle tangle tangleboot tie ctangleboot; do
    cp --preserve=mode $TEXLIVE_SOURCE_DIR/texlive-build-native/texk/web2c/$f  $TEXLIVE_SOURCE_DIR/texlive-build-wasm/texk/web2c/
done
for f in fixwrites makecpool splitup web2c; do
    cp --preserve=mode $TEXLIVE_SOURCE_DIR/texlive-build-native/texk/web2c/web2c/$f  $TEXLIVE_SOURCE_DIR/texlive-build-wasm/texk/web2c/web2c/
done

emmake make $MAKEFLAGS xetex
vadimkantorov commented 4 years ago

After specifying -o ctangle -o otangle -o tangle -o tangleboot -o tie -o ctangleboot, the problem is that it cannot find fontconfig includes (despite them being already installed in prefix) and not finding unicode/platform.h.

Doing

DONOTREMAKE="-o ctangle -o otangle -o tangle -o tangleboot -o tie -o ctangleboot"
CFLAGS="-I$PREFIX/include -I$TEXLIVE_SOURCE_DIR/libs/icu/icu-src/source/common"
CFLAGS="$CFLAGS" emmake make $MAKEFLAGS $DONOTREMAKE xetex

does not help. Do I need to reconfigure TexLive to propagate CFLAGS?

Also, it seems that icu did not get built as a dependency. Do I need to build it explicitly?

vadimkantorov commented 4 years ago

If I understand your Makefile, you do not build libicu for wasm (only native), correct?

lyze commented 4 years ago

Right. I didn't do any special handling for wasm in general in the makefile. I didn't consider a wasm binary target at the time.

I believe you need to pass CFLAGS in the emconfigure invocation to propagate that to the the generated makefiles.

vadimkantorov commented 4 years ago

My current scripts are in https://github.com/vadimkantorov/xetex2020.js/

It's very strange. -I/mnt/c/Users/user/xetex2020.js/prefix-wasm/include was propagated.

ls prefix-wasm/include/
expat.h  expat_config.h  expat_external.h  fontconfig  kpathsea

but xetex still can't find fontconfig/fontconfig.h

/mnt/c/Users/user/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I. -I../../../texk/web2c -I./w2c  -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/texk -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texk -I../../../texk/web2c/xetexdir  -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/freetype2/freetype2 -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/teckit/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/harfbuzz/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/graphite2/include -DGRAPHITE2_STATIC -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/libpng/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/zlib/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/pplib/include -I../../../texk/web2c/libmd5     -g -O2 -MT xetexdir/libxetex_a-XeTeXFontInst.o -MD -MP -MF xetexdir/.deps/libxetex_a-XeTeXFontInst.Tpo -c -o xetexdir/libxetex_a-XeTeXFontInst.o `test -f 'xetexdir/XeTeXFontInst.cpp' || echo '../../../texk/web2c/'`xetexdir/XeTeXFontInst.cpp

In file included from ../../../texk/web2c/xetexdir/XeTeXFontInst.cpp:45:
In file included from ../../../texk/web2c/xetexdir/XeTeXFontInst.h:46:
../../../texk/web2c/xetexdir/XeTeXFontMgr.h:41:10: fatal error: 'fontconfig/fontconfig.h' file not found
#include <fontconfig/fontconfig.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../../../texk/web2c/xetexdir/XeTeXFontMgr.cpp:35:
In file included from ../../../texk/web2c/xetexdir/XeTeX_web.h:39:
../../../texk/web2c/xetexdir/XeTeX_ext.h:38:10: fatal error: 'unicode/utypes.h' file not found
#include <unicode/utypes.h>
         ^~~~~~~~~~~~~~~~~~
Makefile:13779: recipe for target 'xetexdir/libxetex_a-XeTeX_pic.o' failed
vadimkantorov commented 4 years ago

Maybe I need to add CPPFLAGS as well?

vadimkantorov commented 4 years ago

Indeed, it was not passing CFLAGS to the C++ compiler, so I had to add CPPFLAGS.

Somehow --with-fontconfig-libdir also did not help, now it's missing fontconfig symbols. Maybe need explicit LDFLAGS

vadimkantorov commented 4 years ago

Hmm, "texlive-build-wasm/texk/web2c/config.log" does have "kpse_cv_fontconfig_libs='-L/mnt/c/Users/user/xetex2020.js/prefix-wasm/lib -lfontconfig'"

vadimkantorov commented 4 years ago

Somehow libtool / linker is not getting them:

libtool: link: /mnt/c/Users/user/emsdk/upstream/emscripten/em++ -g -O2 -o xetex xetexdir/xetex-xetexextra.o synctexdir/xetex-synctex.o xetex-xetexini.o xetex-xetex0.o xetex-xetex-pool.o  libxetex.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/harfbuzz/libharfbuzz.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/graphite2/libgraphite2.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/teckit/libTECkit.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/libpng/libpng.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/freetype2/libfreetype.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/pplib/libpplib.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/zlib/libz.a libmd5.a lib/lib.a /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/texk/kpathsea/.libs/libkpathsea.a
vadimkantorov commented 4 years ago

A problem with manual setting LDFLAGS="-L$PREFIX/lib -lfontconfig -lexpat at TexLive configure time is that these libraries don't exist at that point yet. So compilation fails very early.

Somehow these flags need to be added only for xetex. Any idea for how to configure that directory separately?

vadimkantorov commented 4 years ago

Okay, I found that directly patching xetex_LDFLAGS in build/texk/web2c/Makefile works. Now the problem is that it requires some icu function and I can't figure out what icu library is needed.

And icu build fails because of missing popen. Somehow -s ERROR_ON_UNDEFINED_SYMBOLS=0 does not get propagated to linker commands in Makefile in libs/icu/icu-build.

vadimkantorov commented 4 years ago

Maybe it doesn't get configured with all other modules, though I have --enabled-icu

vadimkantorov commented 4 years ago

It seems that TexLive does not propagate its CFLAGS/CPPFLAGS to icu at all, and neither icu nor em++ provides no way to pass any extra CFLAGS after configure time :(

vadimkantorov commented 4 years ago

Okay, I got around that by doing emmake make CXX="em++ -s ERROR_ON_UNDEFINED_SYBMOLS=0". That works, but now my icu build fails data generation and your trick with copying over libicudata.a and pkgdata/icupkg doesn't work.

Do we actually need that packagedata target? Can we copy over those data files from the native build (like we do with latex format file)?

cp --preserve=mode $TEXLIVE_SOURCE_DIR/texlive-build-native/libs/icu/icu-build/bin/icupkg $TEXLIVE_SOURCE_DIR/texlive-build-native/libs/icu/icu-build/bin/pkgdata bin/
cp $TEXLIVE_SOURCE_DIR/texlive-build-native/libs/icu/icu-build/stubdata/libicudata.a stubdata/
$EMMAKE make $MAKEFLAGS CXX="em++ -s ERROR_ON_UNDEFINED_SYMBOLS=0"
make[0]: Making `all' in `data'
make[1]: Entering directory '/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/icu-build/data'
LD_LIBRARY_PATH=../stubdata:../tools/ctestfw:../lib:$LD_LIBRARY_PATH  ../bin/pkgdata -O ../data/icupkg.inc -q -c -s /mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/icu-build/data/out/build/icudt65l -d ../lib -e icudt65  -T ./out/tmp -p icudt65l -m static -r 65.1 -L icudata ./out/tmp/icudata.lst
pkgdata: /mnt/c/Users/user/emsdk/upstream/emscripten/emcc -s ERROR_ON_UNDEFINED_SYMBOLS=0 -DELIDE_CODE -I/mnt/c/Users/user/xetex2020.js/prefix-wasm/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/include -ffunction-sections -fdata-sections -D_REENTRANT  -DU_HAVE_ELF_H=1 -DU_ENABLE_DYLOAD=0 -DU_HAVE_STRTOD_L=1 -DU_HAVE_XLOCALE_H=1 -DU_HAVE_STRING_VIEW=1  -DU_ATTRIBUTE_DEPRECATED= -s ERROR_ON_UNDEFINED_SYMBOLS=0 -DELIDE_CODE -I/mnt/c/Users/user/xetex2020.js/prefix-wasm/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/include   -c -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/../../../libs/icu/icu-src/source/common -I../common -DPIC -fPIC -o ./out/tmp/icudt65l_dat.o ./out/tmp/icudt65l_dat.S
./out/tmp/icudt65l_dat.S:2:20: error: Expected ,, instead got: -
 .section .note.GNU-stack,"",%progbits
                   ^
./out/tmp/icudt65l_dat.S:3:18: error: Expected ,, instead got:

 .section .rodata
                 ^
./out/tmp/icudt65l_dat.S:8:20: error: Expected label,@type declaration, got: %
 .type icudt65_dat,%object
                   ^
emcc: error: '/mnt/c/Users/user/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=0 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -nostdsysteminc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libcxx -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libcxxabi/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/compat -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/libc -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libc/musl/arch/emscripten -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/local/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SSE -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/compiler-rt/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/lib/libunwind/include -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/cache/wasm/include -DEMSCRIPTEN -fignore-exceptions -DELIDE_CODE -I/mnt/c/Users/user/xetex2020.js/prefix-wasm/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/include -ffunction-sections -fdata-sections -D_REENTRANT -DU_HAVE_ELF_H=1 -DU_ENABLE_DYLOAD=0 -DU_HAVE_STRTOD_L=1 -DU_HAVE_XLOCALE_H=1 -DU_HAVE_STRING_VIEW=1 -DU_ATTRIBUTE_DEPRECATED= -DELIDE_CODE -I/mnt/c/Users/user/xetex2020.js/prefix-wasm/include -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/include -c -I/mnt/c/Users/user/xetex2020.js/texlive-source-9ed922e7d25e41b066f9e6c973581a4e61ac0328/texlive-build-wasm/libs/icu/../../../libs/icu/icu-src/source/common -I../common -DPIC -fPIC -o ./out/tmp/icudt65l_dat.o ./out/tmp/icudt65l_dat.S -Xclang -isystem/mnt/c/Users/user/emsdk/upstream/emscripten/system/include/SDL -c -o ./out/tmp/icudt65l_dat.o -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr' failed (1)
-- return status = 256
Error creating with assembly code. Failed command:  �_��
Error generating assembly code for data.
Makefile:172: recipe for target 'packagedata' failed