Windows build missing ucrtbased.dll and VCRUNTIME140D.dll

[weiji14]

Issue:

Fresh conda installation of gmt==6.0.0 on Windows doesn't seem to work. After installing gmt inside a clean conda environment using conda create -n gmt gmt, running gmt in Anaconda prompt results in the popups below:

Transcription: The code execution cannot proceed because ucrtbased.dll was not found. Reinstalling the program may fix this problem.

Transcription: The code execution cannot proceed because VCRUNTIME140D.dll was not found. Reinstalling the program may fix this problem.

Environment (conda list):

``` $ conda list # Name Version Build Channel boost-cpp 1.70.0 h6a4c333_2 conda-forge bzip2 1.0.8 hfa6e2cd_1 conda-forge ca-certificates 2019.9.11 hecc5488_0 conda-forge certifi 2019.9.11 py38_0 conda-forge cfitsio 3.470 hfa6e2cd_2 conda-forge curl 7.65.3 h4496350_0 conda-forge expat 2.2.5 he025d50_1004 conda-forge ffmpeg 4.2 h6538335_0 conda-forge fftw 3.3.8 nompi_hbc43ac9_1110 conda-forge freetype 2.10.0 h563cfd7_1 conda-forge freexl 1.0.5 hd288d7e_1002 conda-forge gdal 3.0.2 py38h2fee047_2 conda-forge geos 3.7.2 he025d50_2 conda-forge geotiff 1.5.1 ha8299ad_6 conda-forge gettext 0.19.8.1 hb01d8f6_1002 conda-forge ghostscript 9.22 he025d50_1 conda-forge glib 2.58.3 py38hc0c2ac7_1002 conda-forge gmt 6.0.0 h54a23f7_0 conda-forge hdf4 4.2.13 hf8e6fe8_1003 conda-forge hdf5 1.10.5 nompi_ha405e13_1104 conda-forge icu 64.2 he025d50_1 conda-forge intel-openmp 2019.4 245 jpeg 9c hfa6e2cd_1001 conda-forge kealib 1.4.10 hf7dc31f_1005 conda-forge krb5 1.16.3 hdd46e55_1001 conda-forge libblas 3.8.0 14_mkl conda-forge libcblas 3.8.0 14_mkl conda-forge libcurl 7.65.3 h4496350_0 conda-forge libffi 3.2.1 h6538335_1006 conda-forge libgdal 3.0.2 hd349289_2 conda-forge libiconv 1.15 hfa6e2cd_1005 conda-forge libkml 1.3.0 h4ece8bf_1010 conda-forge liblapack 3.8.0 14_mkl conda-forge libnetcdf 4.7.1 nompi_h8d74e2a_101 conda-forge libpng 1.6.37 h7602738_0 conda-forge libpq 11.5 hb0bdaea_2 conda-forge libspatialite 4.3.0a h2b2ca8d_1032 conda-forge libssh2 1.8.2 h642c060_2 conda-forge libtiff 4.1.0 h2e92f26_0 conda-forge libxml2 2.9.10 h9ce36c8_0 conda-forge lz4-c 1.8.3 he025d50_1001 conda-forge m2w64-expat 2.1.1 2 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gettext 0.19.7 2 m2w64-gmp 6.1.0 2 m2w64-libiconv 1.14 6 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 m2w64-xz 5.2.2 2 mkl 2019.4 245 msys2-conda-epoch 20160418 1 numpy 1.17.3 py38hc71023c_0 conda-forge openjpeg 2.3.1 hb24c2e3_1 conda-forge openssl 1.1.1d hfa6e2cd_0 conda-forge pcre 8.43 h6538335_0 conda-forge pip 19.3.1 py38_0 conda-forge poppler 0.67.0 h92819f6_7 conda-forge poppler-data 0.4.9 1 conda-forge postgresql 11.5 h06f7779_2 conda-forge proj 6.2.1 ha7a8c7b_0 conda-forge python 3.8.0 hc9e8b01_3 conda-forge setuptools 41.6.0 py38_1 conda-forge sqlite 3.30.1 hfa6e2cd_0 conda-forge tbb 2018.0.5 he980bc4_0 conda-forge tiledb 1.6.2 h6bfbd54_2 conda-forge tk 8.6.9 hfa6e2cd_1003 conda-forge vc 14.1 h0510ff6_4 vs2015_runtime 14.16.27012 hf0eaf9b_0 wheel 0.33.6 py38_0 conda-forge wincertstore 0.2 py38_1003 conda-forge xerces-c 3.2.2 h6538335_1004 conda-forge xz 5.2.4 h2fa13f4_1001 conda-forge zlib 1.2.11 h2fa13f4_1006 conda-forge zstd 1.4.3 hd8a0e53_0 conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : gmt active env location : C:\Users\username\AppData\Local\Continuum\miniconda3\envs\gmt shell level : 2 user config file : C:\Users\username\.condarc populated config files : C:\Users\username\.condarc conda version : 4.7.12 conda-build version : not installed python version : 3.7.4.final.0 virtual packages : base environment : C:\Users\username\AppData\Local\Continuum\miniconda3 (writable) channel URLs : https://conda.anaconda.org/conda-forge/win-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\username\AppData\Local\Continuum\miniconda3\pkgs C:\Users\username\.conda\pkgs C:\Users\username\AppData\Local\conda\conda\pkgs envs directories : C:\Users\username\AppData\Local\Continuum\miniconda3\envs C:\Users\username\.conda\envs C:\Users\username\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.16299 administrator : False netrc file : None offline mode : False ```

[seisman]

It shouldn't happen. VCRUNTIME140D.dll is a required library for debug version, but the conda-forge package is built with CMAKE_BUILD_TYPE=Release.

[seisman]

Right, I can reproduce the same issue on my Windows machine.

[seisman]

This is the DLL dependency tree reported by Dependency Walker.

It's obvious that the zstd library needs the debug version libraries (i.e. vcruntime140d.dll and ucrtbased.dll), which causes the trouble. This is a bug of the zstd feedstock and was fixed in https://github.com/conda-forge/zstd-feedstock/pull/36.

GMT should work well if the latest zstd (1.4.4, build 1) is installed, but the conda --list shows zstd 1.4.3 is installed. Not sure what we should do here.

[jpgill86]

It's possible that libtiff is pinning zstd to 1.4.3 in your environment. See conda-forge/libtiff-feedstock#44.

[seisman]

That makes sense. Thanks for letting us know.

[jpgill86]

Actually, I just learned it's a global pinning issue:

https://github.com/conda-forge/libtiff-feedstock/pull/45#issuecomment-554495921

[weiji14]

Thanks for the information @jpgill86. Didn't know about global pinning before! I found some documentation at https://conda-forge.org/docs/maintainer/infrastructure.html#conda-forge-pinning and will read up on it now.

[seisman]

The global pinning has been migrated to zstd 1.4.4 (https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/334). @weiji14 Could you try if gmt works now?

[seisman]

If not, we probably need to rebuild the gmt package.

[weiji14]

Ok, I'll try and make a push at https://github.com/GenericMappingTools/pygmt/pull/354 and see how the tests go. Might take ~15min. Edit: Hmm, actually I might drag myself to uni (to a Windows computer) to test things properly. Also might take ~15min.

[jpgill86]

Thanks for the information @jpgill86. Didn't know about global pinning before! I found some documentation at https://conda-forge.org/docs/maintainer/infrastructure.html#conda-forge-pinning and will read up on it now.

We are in exactly the same boat, at the same time! 🤣 I'm currently wondering how long the migration process takes if zstd is used by many packages. Minutes? Days?

[ocefpaf]

Minutes? Days?

Sorry, it will be a few days. But thanks the @CJ-Wright's automation it is way faster than before!

[jpgill86]

Minutes? Days?

Sorry, it will be a few days. But thanks the @CJ-Wright's automation it is way faster than before!

Gotcha. I'm reading about it now and am very impressed by the orchestration required to keep everything running!

[weiji14]

If not, we probably need to rebuild the gmt package.

I've removed the "gmt" conda environment and creating a fresh one using conda create -n gmt gmt still wants to install zstd 1.4.3. Let's wait for https://github.com/conda-forge/libtiff-feedstock/pull/46 😄

[CJ-Wright]

The status page lets you check on the state of the migration. You can also look at the graph plot.

[weiji14]

Cool, thanks everyone! libtiff is now updated but conda create -n gmt gmt won't work directly (yet) as it's still trying to get 4.1.0 build 0 instead of build 1. It'll just take some time for the repodata.json file to update itself and there shouldn't be a need to rebuild the gmt package.

Meanwhile, following this stackoverflow post on installing a specific build of an anaconda package, conda create -n gmt gmt libtiff=4.1.0=h21b02b4_1 or conda install libtiff=4.1.0=h21b02b4_1 works so I'll close this issue (Edit: this works for gmt 5.4.5 but not 6.0.0, see below).

[ocefpaf]

Once the new packages are available on the channel, in a few hours, please let me know if they work.

[jpgill86]

Once the new packages are available on the channel, in a few hours, please let me know if they work.

I'm not using gmt, but the (possibly) related issue that prompted me to open conda-forge/libtiff-feedstock#44 is now fixed by the migration of zstd to 1.4.4. Thanks @ocefpaf!

[ocefpaf]

Thanks for letting us know!

[weiji14]

Sorry, just realized that the workaround above installed gmt 5.4.5 instead of version 6.0.0... There's still some conflicts if we do conda create --name gmt gmt=6.0.0 libtiff=4.1.0=h21b02b4_1 because libgdal depends on tiledb but the zstd rebuild in https://github.com/conda-forge/tiledb-feedstock/pull/48 only updated tiledb 1.7.0 and not 1.6.*

(base) C:\>conda create --name gmt gmt=6.0.0 libtiff=4.1.0=h21b02b4_1
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gmt=6.0.0 -> gdal[version='>=3.0.1,<3.1.0a0'] -> libgdal==3.0.1=h2491c49_10 -> tiledb[version='>=1.6.2,<1.6.3.0a0'] -> zstd[version='>=1.4.0,<1.4.1.0a0']
  - libtiff==4.1.0=h21b02b4_1

(base) C:\>conda create --name gmt gmt=6.0.0 libtiff=4.1.0=h21b02b4_1 gdal=3.0.2
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - gmt=6.0.0 -> gdal[version='>=3.0.1,<3.1.0a0'] -> libgdal==3.0.1=h2491c49_10 -> tiledb[version='>=1.6.2,<1.6.3.0a0'] -> zstd[version='>=1.4.0,<1.4.1.0a0']
  - libtiff==4.1.0=h21b02b4_1

(base) C:\>conda create --name gmt gmt=6.0.0 libtiff=4.1.0=h21b02b4_1 libgdal=3.0.2
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - libgdal=3.0.2 -> tiledb[version='>=1.6.2,<1.6.3.0a0'] -> zstd[version='>=1.4.0,<1.4.1.0a0']
  - libtiff==4.1.0=h21b02b4_1

[ocefpaf]

but the zstd rebuild in conda-forge/tiledb-feedstock#48 only updated tiledb 1.7.0 and not 1.6.*

We can build the older tiledb with the new zstd but we can also update tiledb global pin. I'll work on this tomorrow.

[weiji14]

We can build the older tiledb with the new zstd but we can also update tiledb global pin. I'll work on this tomorrow.

@ocefpaf, I've put in two pull requests to update tiledb on conda-forge, hopefully that helps a bit?

[joa-quim]

Guys, you should not be building GDAL against an external TIF lib. Just let it use it's internal TIF library. This lib is maintained by the same person I (Even Rouault) is is always most up to date. With this the zstd issue will simple go away.

[weiji14]

Could you elaborate a bit more @joa-quim? Do you mean the GMT source code contains the TIF library already? Looking at the meta.yaml file, my impression was that we're not building gdal here (I could be wrong though), just getting the already-built gdal library on conda-forge.

[joa-quim]

No, GMT doesn't have any TIF lib. What I mean is when building GDAL, let it use its own TIF library instead of linking against an external one. See the dependencies of my GDAL (the one shipped with the GMT installer). No TIF, still one can read and create GeoTIFFs with GMT.

[weiji14]

Ok, let me know if this makes sense. So you're saying we shouldn't be using the conda-forge built gdal because it links to an external TIF library. Instead, we should be building our own gdal without that external TIF link?

[joa-quim]

What I'm saying is that your dependency issue is coming from a GDAL that was built against an external TIF (and, btw, a Jpeg one). I don't know how things work on the Conda build and if you have to use the Conda GDAL's or if you can use another build.

Also, though we all would like to have a completely scriptable GMT build, please be aware that the Conda GMT build will rather more limited than the one we provide with the official installer. For example, no MEX and the GDAL build will not come with its GDAL_DATA as well as PROJ_DAT included and automatically detectable by GMT. This mean that the -Jepsg:xxx syntax will not be available with that build.

(Very late here, going to bed)

[weiji14]

Ah I see, good to keep that in mind. Anyways, best to continue that discussion in a separate issue.

I think the upstream dependencies are settled now (i.e. conda create -n gmt gmt works), and I'll close this issue shortly after I run some tests.