Missing dlls on for the build on Windows?

[krassowski]

Issue: We are using tectonic for LSP-latex integration testing with Azure Pipelines. It worked well so far, but a recent change of conda environment seem to have broken the jobs on Windows. Linux and MacOS worked ok. On Windows, it returns 3221225785 exit status, which seem to suggest missing dlls.

Since the previous successful run on Windows the tectonic version did not change, but the build version did:

success:

tectonic 0.1.12 h11def9f_0 conda-forge

failure:

tectonic 0.1.12 h87e793b_1 conda-forge

The h11def9f_0 binary is bigger (2.9 MB) than h87e793b_1 (2.8 MB) and is labelled with cf202003:

It could be related to this rebuild: https://github.com/conda-forge/tectonic-feedstock/pull/20

I changed the channel specifier to get the build 0 (labelled cf202003) as:

conda-forge/label/cf202003::tectonic

in a pull request https://github.com/krassowski/jupyterlab-lsp/pull/320, which made all work again. This was the only change in the environment.

For the details of the run, please see: https://dev.azure.com/krassowskimichal/jupyterlab-lsp/_build/results?buildId=742&view=logs&j=c1ff525e-18e3-5932-41ad-b3b1d5f9697a

Environment (conda list):

The failing jobs: ``` $ conda list # packages in environment at C:\Miniconda: # # Name Version Build Channel _r-mutex 1.0.1 anacondar_1 conda-forge argon2-cffi 20.1.0 py36h779f372_1 conda-forge asn1crypto 1.4.0 py_0 astroid 2.3.3 py36_1 conda-forge atomicwrites 1.4.0 pyh9f0ad1d_0 conda-forge attrs 20.1.0 pyh9f0ad1d_0 conda-forge autopep8 1.5.4 pyh9f0ad1d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.1 py_0 conda-forge bleach 3.1.5 pyh9f0ad1d_0 conda-forge brotlipy 0.7.0 py36he774522_1000 ca-certificates 2020.6.20 hecda079_0 conda-forge cairo 1.16.0 hc79e3cf_1005 conda-forge certifi 2020.6.20 py36h9f0ad1d_0 conda-forge cffi 1.14.0 py36h7a1dbc1_0 chardet 3.0.4 py36_1003 chktex 1.7.6 h301d43c_0 conda-forge colorama 0.4.3 py_0 conda-forge conda 4.8.4 py36_0 conda-package-handling 1.6.1 py36h62dcd97_0 console_shortcut 0.1.1 3 coverage 5.2.1 py36h779f372_0 conda-forge cryptography 2.9.2 py36h7a1dbc1_0 decorator 4.4.2 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge entrypoints 0.3 py36h9f0ad1d_1001 conda-forge firefox 79.0 ha925a31_1 conda-forge flake8 3.8.3 py_1 conda-forge fontconfig 2.13.1 hdcb13c4_1002 conda-forge freetype 2.10.2 hd328e21_0 conda-forge future 0.18.2 py36h9f0ad1d_1 conda-forge geckodriver 0.27.0 h33f27b4_0 conda-forge gettext 0.19.8.1 hb01d8f6_1002 conda-forge glib 2.65.0 he4de6d7_0 conda-forge graphite2 1.3.13 1000 conda-forge harfbuzz 2.7.1 hab87c11_0 conda-forge icu 67.1 h33f27b4_0 conda-forge idna 2.10 py_0 importlib-metadata 1.7.0 py36h9f0ad1d_0 conda-forge importlib_metadata 1.7.0 0 conda-forge iniconfig 1.0.1 pyh9f0ad1d_0 conda-forge intel-openmp 2020.1 216 ipykernel 5.3.4 py36h5ca1d4c_0 conda-forge ipython 7.16.1 py36h95af2a2_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge isort 5.4.2 py36h9f0ad1d_0 conda-forge jedi 0.17.2 py36h9f0ad1d_0 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge json5 0.9.4 pyh9f0ad1d_0 conda-forge jsonschema 3.2.0 py36h9f0ad1d_1 conda-forge jupyter_client 6.1.6 py_0 conda-forge jupyter_core 4.6.3 py36h9f0ad1d_1 conda-forge jupyterlab 2.2.5 py_0 conda-forge jupyterlab_server 1.2.0 py_0 conda-forge lazy-object-proxy 1.5.1 py36h779f372_0 conda-forge libblas 3.8.0 16_mkl conda-forge libffi 3.2.1 ha925a31_1007 conda-forge libiconv 1.16 he774522_0 conda-forge liblapack 3.8.0 16_mkl conda-forge libpng 1.6.37 ha81a0f5_2 conda-forge libsodium 1.0.17 h2fa13f4_0 conda-forge libxml2 2.9.10 h1006b36_2 conda-forge m2-gcc-libs 5.3.0 4 m2-msys2-runtime 2.5.0.17080.65c939c 3 m2-ncurses 6.0.20160220 2 m2w64-bwidget 1.9.10 2 m2w64-bzip2 1.0.6 6 m2w64-expat 2.1.1 2 m2w64-fftw 3.3.4 6 m2w64-flac 1.3.1 3 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gettext 0.19.7 2 m2w64-gmp 6.1.0 2 m2w64-gsl 2.1 2 m2w64-icu 58.2 heb44b8b_2 m2w64-libiconv 1.14 6 m2w64-libjpeg-turbo 1.4.2 3 m2w64-libogg 1.3.2 3 m2w64-libpng 1.6.21 2 m2w64-libsndfile 1.0.26 2 m2w64-libtiff 4.0.6 2 m2w64-libvorbis 1.3.5 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 m2w64-libxml2 2.9.3 4 m2w64-mpfr 3.1.4 4 m2w64-pcre 8.38 2 m2w64-speex 1.2rc2 3 m2w64-speexdsp 1.2rc3 3 m2w64-tcl 8.6.5 3 m2w64-tk 8.6.5 3 m2w64-tktable 2.10 5 m2w64-wineditline 2.101 5 m2w64-xz 5.2.2 2 m2w64-zlib 1.2.8 10 markupsafe 1.1.1 py36h68a101e_1 conda-forge mccabe 0.6.1 py_1 conda-forge menuinst 1.4.16 py36he774522_1 mistune 0.8.4 py36h68a101e_1001 conda-forge mkl 2020.1 216 more-itertools 8.4.0 py_0 conda-forge msys2-conda-epoch 20160418 1 nbconvert 5.6.1 py36h9f0ad1d_1 conda-forge nbformat 5.0.7 py_0 conda-forge nodejs 10.16.3 0 conda-forge notebook 6.1.3 py36h9f0ad1d_0 conda-forge openssl 1.1.1g he774522_1 conda-forge packaging 20.4 pyh9f0ad1d_0 conda-forge pandoc 2.10.1 he774522_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.7.1 pyh9f0ad1d_0 conda-forge pcre 8.44 h6538335_0 conda-forge pickleshare 0.7.5 py36h9f0ad1d_1001 conda-forge pip 20.2.2 py36_0 pixman 0.38.0 hfa6e2cd_1003 conda-forge pluggy 0.13.1 py36h9f0ad1d_2 conda-forge powershell_shortcut 0.0.1 2 prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge prompt-toolkit 3.0.6 py_0 conda-forge py 1.9.0 pyh9f0ad1d_0 conda-forge pycodestyle 2.6.0 pyh9f0ad1d_0 conda-forge pycosat 0.6.3 py36he774522_0 pycparser 2.20 py_2 pydocstyle 5.0.2 py_0 conda-forge pyflakes 2.2.0 pyh9f0ad1d_0 conda-forge pygments 2.6.1 py_0 conda-forge pylint 2.3.1 py36_0 conda-forge pyopenssl 19.1.0 py_1 pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyrsistent 0.16.0 py36h68a101e_0 conda-forge pysocks 1.7.1 py36_0 pytest 6.0.1 py36h9f0ad1d_0 conda-forge pytest-asyncio 0.12.0 py36h9f0ad1d_1 conda-forge pytest-azurepipelines 0.8.0 py_0 conda-forge pytest-cov 2.10.1 pyh9f0ad1d_0 conda-forge pytest-flake8 1.0.6 pyh9f0ad1d_0 conda-forge pytest-runner 5.2 py_0 conda-forge python 3.6.11 h6f26aa1_2_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python-jsonrpc-server 0.3.4 pyh9f0ad1d_1 conda-forge python-language-server 0.34.1 py36h9f0ad1d_0 conda-forge python_abi 3.6 1_cp36m conda-forge pytz 2020.1 pyh9f0ad1d_0 conda-forge pywin32 227 py36he774522_1 pywinpty 0.5.7 py36_0 conda-forge pyzmq 19.0.2 py36h274d426_0 conda-forge r 3.6 r36_1004 conda-forge r-askpass 1.1 r36hda5aaf8_2 conda-forge r-assertthat 0.2.1 r36h6115d3f_2 conda-forge r-backports 1.1.8 r36hda5aaf8_0 conda-forge r-base 3.6.3 h588f722_3 conda-forge r-base64enc 0.1_3 r36hda5aaf8_1004 conda-forge r-boot 1.3_25 r36h6115d3f_1 conda-forge r-callr 3.4.3 r36h6115d3f_1 conda-forge r-class 7.3_17 r36hda5aaf8_1 conda-forge r-cli 2.0.2 r36h6115d3f_1 conda-forge r-clipr 0.7.0 r36h6115d3f_1 conda-forge r-cluster 2.1.0 r36h17ddedb_3 conda-forge r-codetools 0.2_16 r36h6115d3f_1002 conda-forge r-collections 0.3.5 r36hda5aaf8_0 conda-forge r-crayon 1.3.4 r36h6115d3f_1003 conda-forge r-curl 4.3 r36hda5aaf8_1 conda-forge r-cyclocomp 1.1.0 r36h6115d3f_1004 conda-forge r-desc 1.2.0 r36h6115d3f_1003 conda-forge r-digest 0.6.25 r36h796a38f_2 conda-forge r-ellipsis 0.3.1 r36hda5aaf8_0 conda-forge r-evaluate 0.14 r36h6115d3f_2 conda-forge r-fansi 0.4.1 r36hda5aaf8_1 conda-forge r-foreign 0.8_76 r36hda5aaf8_1 conda-forge r-fs 1.5.0 r36h796a38f_0 conda-forge r-glue 1.4.1 r36hda5aaf8_0 conda-forge r-highr 0.8 r36h6115d3f_2 conda-forge r-hms 0.5.3 r36h6115d3f_1 conda-forge r-htmltools 0.5.0 r36h796a38f_0 conda-forge r-httr 1.4.2 r36h6115d3f_0 conda-forge r-irdisplay 0.7 r36_1002 conda-forge r-irkernel 1.1.1 r36h6115d3f_0 conda-forge r-jsonlite 1.7.0 r36hda5aaf8_0 conda-forge r-kernsmooth 2.23_17 r36h2d65612_1 conda-forge r-knitr 1.29 r36h6115d3f_0 conda-forge r-languageserver 0.3.6 r36hda5aaf8_0 conda-forge r-lattice 0.20_41 r36hda5aaf8_2 conda-forge r-lazyeval 0.2.2 r36hda5aaf8_2 conda-forge r-lifecycle 0.2.0 r36h6115d3f_1 conda-forge r-lintr 2.0.1 r36h6115d3f_1 conda-forge r-magrittr 1.5 r36h6115d3f_1003 conda-forge r-markdown 1.1 r36hda5aaf8_1 conda-forge r-mass 7.3_52 r36hda5aaf8_0 conda-forge r-matrix 1.2_18 r36h0adb578_3 conda-forge r-mgcv 1.8_32 r36h0adb578_0 conda-forge r-mime 0.9 r36hda5aaf8_1 conda-forge r-nlme 3.1_148 r36h17ddedb_0 conda-forge r-nnet 7.3_14 r36hda5aaf8_1 conda-forge r-openssl 1.4.2 r36h601d743_0 conda-forge r-pbdzmq 0.3_3 r36h8216dcb_1003 conda-forge r-pillar 1.4.6 r36h6115d3f_0 conda-forge r-pkgbuild 1.1.0 r36h6115d3f_0 conda-forge r-pkgconfig 2.0.3 r36h6115d3f_1 conda-forge r-pkgload 1.1.0 r36h796a38f_0 conda-forge r-praise 1.0.0 r36h6115d3f_1004 conda-forge r-prettyunits 1.1.1 r36h6115d3f_1 conda-forge r-processx 3.4.3 r36hda5aaf8_0 conda-forge r-ps 1.3.4 r36hda5aaf8_0 conda-forge r-purrr 0.3.4 r36hda5aaf8_1 conda-forge r-r.cache 0.14.0 r36h6115d3f_1 conda-forge r-r.methodss3 1.8.0 r36h6115d3f_1 conda-forge r-r.oo 1.23.0 r36h6115d3f_1 conda-forge r-r.utils 2.9.2 r36h6115d3f_1 conda-forge r-r6 2.4.1 r36h6115d3f_1 conda-forge r-rcpp 1.0.5 r36h796a38f_0 conda-forge r-readr 1.3.1 r36h796a38f_1004 conda-forge r-recommended 3.6 r36_1004 conda-forge r-rematch2 2.1.2 r36h6115d3f_1 conda-forge r-remotes 2.2.0 r36h6115d3f_0 conda-forge r-repr 1.1.0 r36h6115d3f_1 conda-forge r-rex 1.2.0 r36h6115d3f_1 conda-forge r-rlang 0.4.7 r36hda5aaf8_0 conda-forge r-rpart 4.1_15 r36hda5aaf8_2 conda-forge r-rprojroot 1.3_2 r36h6115d3f_1003 conda-forge r-rstudioapi 0.11 r36h6115d3f_1 conda-forge r-spatial 7.3_12 r36hda5aaf8_1 conda-forge r-stringdist 0.9.6 r36hda5aaf8_0 conda-forge r-stringi 1.4.6 r36h796a38f_3 conda-forge r-stringr 1.4.0 r36h6115d3f_2 conda-forge r-styler 1.3.2 r36h6115d3f_1 conda-forge r-survival 3.2_3 r36hda5aaf8_0 conda-forge r-sys 3.4 r36hda5aaf8_0 conda-forge r-testthat 2.3.2 r36h796a38f_1 conda-forge r-tibble 3.0.3 r36hda5aaf8_0 conda-forge r-utf8 1.1.4 r36hda5aaf8_1003 conda-forge r-uuid 0.1_4 r36hda5aaf8_1 conda-forge r-vctrs 0.3.2 r36hda5aaf8_0 conda-forge r-withr 2.2.0 r36h6115d3f_1 conda-forge r-xfun 0.16 r36hda5aaf8_0 conda-forge r-xml2 1.3.2 r36h796a38f_1 conda-forge r-xmlparsedata 1.0.3 r36h6115d3f_1 conda-forge r-yaml 2.2.1 r36hda5aaf8_1 conda-forge r-zeallot 0.1.0 r36h6115d3f_1002 conda-forge requests 2.24.0 py_0 robotframework 3.2.1 pyh9f0ad1d_0 conda-forge robotframework-pythonlibcore 2.1.0 py_0 conda-forge robotframework-seleniumlibrary 4.5.0 pyh9f0ad1d_0 conda-forge rope 0.17.0 pyh9f0ad1d_0 conda-forge rpy2 3.3.2 py36r36h9902d54_0 conda-forge ruamel_yaml 0.15.71 py36hfa6e2cd_1000 conda-forge selenium 3.141.0 py36h68a101e_1001 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 49.6.0 py36_0 simplegeneric 0.8.1 py_1 conda-forge six 1.15.0 py_0 snowballstemmer 2.0.0 py_0 conda-forge sqlite 3.33.0 h2a8f88b_0 tectonic 0.1.12 h87e793b_1 conda-forge terminado 0.8.3 py36h9f0ad1d_1 conda-forge testpath 0.4.4 py_0 conda-forge texlab 2.2.0 hf301747_0 conda-forge toml 0.10.1 pyh9f0ad1d_0 conda-forge tornado 6.0.4 py36hfa6e2cd_0 conda-forge tqdm 4.48.2 py_0 traitlets 4.3.3 py36h9f0ad1d_1 conda-forge typed-ast 1.4.1 py36hfa6e2cd_0 conda-forge tzlocal 2.1 pyh9f0ad1d_0 conda-forge ujson 1.35 py36hbd05846_1002 conda-forge urllib3 1.25.10 py_0 vc 14.1 h0510ff6_4 vs2015_runtime 14.16.27012 hf0eaf9b_3 wcwidth 0.2.5 pyh9f0ad1d_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.34.2 py36_0 win_inet_pton 1.1.0 py36_0 wincertstore 0.2 py36h7fe50ca_0 winpty 0.4.3 4 conda-forge wrapt 1.12.1 py36h68a101e_1 conda-forge yaml 0.1.7 hc54c509_2 yapf 0.30.0 pyh9f0ad1d_0 conda-forge zeromq 4.3.2 ha925a31_3 conda-forge zipp 3.1.0 py_0 conda-forge zlib 1.2.11 h62dcd97_4 ```

Details about conda and system ( conda info ):

``` $ conda info active environment : None user config file : C:\Users\VssAdministrator\.condarc populated config files : conda version : 4.8.4 conda-build version : not installed python version : 3.6.11.final.0 virtual packages : base environment : C:\Miniconda (writable) channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Miniconda\pkgs C:\Users\VssAdministrator\.conda\pkgs C:\Users\VssAdministrator\AppData\Local\conda\conda\pkgs envs directories : C:\Miniconda\envs C:\Users\VssAdministrator\.conda\envs C:\Users\VssAdministrator\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.8.4 requests/2.24.0 CPython/3.6.11 Windows/10 Windows/10.0.14393 administrator : True netrc file : None offline mode : False ```

[pkgw]

Hmm ... well, I'm not very sure how to debug these issues in detail, but I've tried submitting an updated build (labeled 0.1.12 h...._2) that should make it to the servers soon. Can you please try it and see if it fixes your problem?

[bollwyvl]

The newer build is looking even stranger, as it apparently has a bunch of python in it, and maybe git (this is the output of diffoscope):

• 

[pkgw]

Weird! Those stray files seem to represent an issue recent versions of the Python package. Reported as conda-forge/python-feedstock#384.

I will check for a workaround.

[pkgw]

OK, we have marked those Python packages as broken and I have just merged a recipe for a new package, build number 3, that should hopefully fix those issues. Please let me know if it works for you.

[krassowski]

Many thanks @pkgw for a very quick reply and the fixes! It seems that the original problem with missing libraries was fixed in h4f32bc4_3 but overall it still does not work great on Windows (while working on Linux and MacOs), failing with error: Zlib error, see below for details.

See more details in the [build on Azure Pipelines](https://dev.azure.com/krassowskimichal/jupyterlab-lsp/_build/results?buildId=767&view=logs&j=c1ff525e-18e3-5932-41ad-b3b1d5f9697a&t=066b17a0-99af-53ba-9478-7b74ddd0c4fc) ``` note: downloading hyph-th.tex note: downloading loadhyph-tr.tex note: downloading hyph-tr.tex note: downloading loadhyph-tk.tex note: downloading hyph-tk.tex note: downloading loadhyph-uk.tex note: downloading hyph-uk.tex note: downloading hyph-quote-uk.tex note: downloading loadhyph-hsb.tex note: downloading hyph-hsb.tex note: downloading loadhyph-cy.tex note: downloading hyph-cy.tex Running TeX ... note: downloading article.cls note: downloading size10.clo note: downloading lmroman10-regular.otf Fontconfig error: Cannot load default config file note: downloading tex-text.tec note: downloading lmroman10-bold.otf Rerunning TeX because "example.aux" changed ... warning: warnings were issued by the TeX engine; use --print and/or --keep-logs for details. Running xdvipdfmx ... note: downloading pdftex.map note: downloading kanjix.map note: downloading ckx.map note: downloading texglyphlist.txt note: downloading pdfglyphlist.txt note: downloading glyphlist.txt error: Zlib error ```

[pkgw]

@krassowski Yikes, I'm not sure what that would be — that's a shouldn't-happen kind of bug. Unfortunately I don't have a ton of bandwidth to investigate these days ... any chance that you would be able to use a debugger to investigate what's going on? I know that's a big ask.

[bollwyvl]

We have a (usually) working binder for that downstream, but perhaps I can PR a more thorough test on this feedstock that will exercise the machinery a bit more, and give us some more detailed info.

[bollwyvl]

Adding a sanity check build to this feedstock over on https://github.com/conda-forge/tectonic-feedstock/pull/24 also demonstrates the windows issue... I'll do some more google trawling, now that we have something reproducible...

by the by: it's really a bummer to have conda do all the "hard" stuff of fulfilling the binary dependencies... to then have to grab seemingly a few hundred files off archive.org the first time tectonic is invoked. Do you think some of this could be pre-cached during the build, and made available as a tectonic-base-deps or something? Pre-activate scripts are the devil, but being able to set the tectonic cache path seems like it would be a big win.

[pkgw]

@bollwyvl Thanks for working on this! All of the data handling should be quite cross-platform so I'm not sure why this problem is only showing up on Windows, but ... that's what debugging is for.

As for your other comment: yes, it's not ideal. My preferred solution would be to create a new Tectonic CLI that has what I call a "Swiss Army knife" style, with subcommands like git, and then implement a command like tectonic bootstrap or tectonic import-cache that could enable a pre-warm of the cache .... as well as many other useful functions that I can't really provide right now because I don't want the CLI to be one of those --activate-totally-bizarre-other-mode --with-strange-info ones. The only barrier to making this happen is just a lack of time on my part. (edit: FWIW if someone else wanted to prototype this I would be ecstatic!)

[pkgw]

Oh, this issue has been fixed for a while, I believe.