build still fails on CRAN's r-release-windows-ix86+x86_64

[gagolews]

https://www.r-project.org/nosvn/R.check/r-release-windows-ix86+x86_64/stringi-00check.html

Related issues: #313, #304

[gagolews]

builds & checks fine on winbuider (release, oldrel, devel), as well CRAN's r-devel-windows, though.

cannot reproduce this myself

[gagolews]

The issue is with loading icudt:

** testing if installed package can be loaded
*** arch - i386
Error: package or namespace load failed for 'stringi' in inDL(x, as.logical(local), as.logical(now), ...):
 ICU init failed: U_FILE_ACCESS_ERROR
Error: loading failed
Execution halted
*** arch - x64
Error: package or namespace load failed for 'stringi' in inDL(x, as.logical(local), as.logical(now), ...):
 ICU init failed: U_FILE_ACCESS_ERROR
Error: loading failed
Execution halted
ERROR: loading failed for 'i386', 'x64'

[jeroen]

Same as https://github.com/gagolews/stringi/issues/170

[gagolews]

hmmm... I don't think so

[jeroen]

Maybe it's better to statically link icudt.a on Windows rather than relocating ICU_DATA at runtime?

[gagolews]

Maybe, but it DID work before, I'd like to know what they changed in the config that I cannot reproduce elsewhere

[jeroen]

I am guessing they are setting a ICU_DATA environment variable somewhere.

[gagolews]

ICU API docs: The data directory is determined as follows: If u_setDataDirectory() has been called, that is it, otherwise if the ICU_DATA environment variable is set, use that, otherwise If a data directory was specified at ICU build time

I call u_setDataDirectory() explicitly in R_init_stringi

[gagolews]

see https://github.com/gagolews/stringi/blob/master/src/stri_stringi.cpp#L250

[jeroen]

Few more things that could be wrong: ICU docs say:

u_setDataDirectory() This function should be called at most once in a process, before the first ICU operation.

However R itself uses libicu as well and also a few other packages. So perhaps something was loaded before stringi called this function.

It could also be an encoding or escaping problem with the const char* libpath argument. Windows paths are very strange, they need to be in native encoding and have ambigous escaping rules. This is often important because R is installed in C:\Program Files\R. We ran into similar problems with libcairo and libgit2 as well.

[jeroen]

Could you post a link to the successful winbuilder output? Then I can try the binary package locally.

[gagolews]

https://win-builder.r-project.org/qkel8h8T0XF6/

[gagolews]

Nah, I don't think the libicus are in conflict. When libicu is built by stringi, all function names have different suffixes.

[jeroen]

CRAN compiles certain packages with custom flags. Compare the two build logs:

• winbuilder: https://win-builder.r-project.org/qkel8h8T0XF6/00install.out
• cran: https://www.r-project.org/nosvn/R.check/r-release-windows-ix86+x86_64/stringi-00install.txt

The first one shows:

installing via 'install.libs.R' to d:/RCompile/CRANguest/R-release/lib/stringi
icudt already downloaded
decompressing icudt archive icu61/data/icudt61l.zip to: d:/RCompile/CRANguest/R-release/lib/stringi/libs
icudt has been installed successfully

But the CRAN version only shows:

installing via 'install.libs.R' to d:/Rcompile/CRANpkg/lib/3.5/stringi

[gagolews]

this could just mean that the message() printing is suppressed ..

[hadley]

Is there anything that I/we can do to help resolve this problem?

[gagolews]

Well, 2 months ago I sent the following email to CRAN:

On 05.08.2018 19:17, Uwe Ligges wrote:
> your package stringi_1.2.4.tar.gz did *not* pass 'R CMD check' on
> Windows and will be omitted from the corresponding CRAN directory.
> Please check the attached log-file and submit a version
> with increased version number that passes R CMD check on Windows.

Dear Uwe, dear CRAN,

apologies for writing about this, but I'm struggling to figure out (for a few months already) what might be the reason why the CRAN win build of stringi fails (note it concerns R-release and oldrel, R-devel is fine).

**I cannot reproduce the error nowhere, it might be specific to the CRAN R-release and R-oldrel build chain.**

When I use the winbuilder, everything is fine, I am pretty sure the key part concerns copying of the icudt file in install.libs.R.

If I run winbuilder (see https://win-builder.r-project.org/3QOwp1Y6IHsn/), I get:

[...]
installing via 'install.libs.R' to d:/RCompile/CRANguest/R-release/lib/stringi
icudt already downloaded
decompressing icudt archive icu61/data/icudt61l.zip to: d:/RCompile/CRANguest/R-release/lib/stringi/libs
icudt has been installed successfully

*** arch - x64
[...]

which is the correct behavior.

On the other hand, the 00check.log I received from you (see also https://www.r-project.org/nosvn/R.check/r-release-windows-ix86+x86_64/stringi-00install.html) states:

[...]
installing via 'install.libs.R' to d:/Rcompile/CRANpkg/lib/3.5/stringi
                       // <------ nothing

*** arch - x64
[...]

Is it the output printing suppressed? Or install.libs.R is not run at all?

Marek

Got a response from Kurt Hornik:

Marek,

This most likely needs Uwe's expertise, but just checking:

* The winbuilder results are for exactly the same package version as
the
 results on
 <https://cran.r-project.org/web/checks/check_results_stringi.html>? 

* If so: the package check results page shows ok for windows and
r-devel
 but errors for windows and the other flavors.  Did you try winbuilder
 also with the r-release series?

Best
-k

I answered:

Yes, indeed, this is the same version and I tried all the 3 winbuider schemes - always with success.

And Kurt replied:

I see, thanks.  This will then need Uwe's attention, but Uwe is mostly
offline this week.

Best
-k

End of story.

[hadley]

Unfortunately it's likely that you'll have to ping CRAN repeatedly to get any additional action.

Have you considered using @jeroen's static library approach? It's a proven technique that seems to remove most of the variability of CRAN, and gives you much greater control and ability to fix problems.

[gagolews]

But the current solution worked well for like 2 years already. And still works on all the 3 winbuilder configs as well as the R-devel check. I'm pretty sure there's something wrong on the CRAN side.

[hadley]

Right, but there's basically nothing you can do to affect the CRAN side except repeatedly emailing them. That might either resolve the problem, or just annoy them.

[jeroen]

One easy fix would be to get rid of your configure.win file and either generate the uconfig_local.h in your Makevars.win or hardcode it for windows and save it in a subdirectory that you include with -Iwindir in your PKG_CPPFLAGS in Makevars.win.

[gagolews]

@jeroen Why do you claim this is a fix anyway? What is wrong with configure.win? Instead of random guessing, I'd like to know what's wrong with the 2 CRAN machines, e.g., what's the diferrence between R-devel and R-release's config.

[jeroen]

If a package includes a configure.win script, R has to assume the build is configured specifically for a single architecture and multilib (building both 32 and 64 bit in one run) is disabled. Packages that have a configure.win require special treatment to build a binary pkg that works for both architectures, e.g. by building with --merge-multiarch or --force-multiarch.

This is where things seem to be going wrong in your case. It looks like the winbuilder uses different custom build flags for stringi than the CRAN release builder. You can see form the log files that for one build it get built inside sub directories src-i386 and src-x64 and the other do not.

In general I strongly recommend to avoid using a configure.win if at all possible, so that your package does not need special treatment from CRAN. What you are doing in your configure.win can easily be done in a make command or just hardcoded via Makevars.win. The entire build process is much less error prone if it consists of a single make call.

[gagolews]

Dear package maintainer,

this notification has been generated automatically.
Your package stringi_1.2.4.tar.gz has been built for Windows and
will be published within 24 hours in the corresponding CRAN directory.
R version 3.5.1 (2018-07-02)

All the best,
Uwe Ligges
(Maintainer of binary packages for Windows)

[gagolews]

Uwe Ligges wrote:

I think I solved the problem during the last week without even knowing it (or maybe some other update or even a MS patch). The regular checks pass now. And I have no idea why they did not pass (reproducibly) before.

Best,
Uwe 

[zkamvar]

I think this is the right place to post this, and I'm also pretty sure you might be aware of the situation, but CRAN is having difficulty building stringi on windows-oldrel and it's propagating cryptic errors through all the reverse imports:

current example at: https://cran.r-project.org/web/checks/check_results_zkamvar_at_gmail.com.html

Error in namespaceExport(ns, exports) : 
     undefined exports: %s!=%, %s!==%, %s+%, %s<%, %s<=%, %s==%, %s===%, %s>%, %s>=%, %stri!=%, %stri!==%, %stri+%, %stri<%, %stri<=%, %stri==%, %stri===%, %stri>%, %stri>=%, stri_datetime_add<-, stri_sub<-, stri_subset<-, stri_subset_charclass<-, stri_subset_coll<-, stri_subset_fixed<-, stri_subset_regex<-, stri_c, stri_c_list, stri_cmp, stri_cmp_eq, stri_cmp_equiv, stri_cmp_ge, stri_cmp_gt, stri_cmp_le, stri_cmp_lt, stri_cmp_neq, stri_cmp_nequiv, stri_compare, stri_conv, stri_count, stri_count_boundaries, stri_count_charclass, stri_count_coll, stri_count_fixed, stri_count_regex, stri_count_words, stri_datetime_add, stri_datetime_create, stri_datetime_fields, stri_datetime_format, stri_datetime_fstr, stri_datetime_now, stri_datetime_parse, stri_datetime_symbols, stri_detect, stri_detect_charclass, stri_detect_coll, stri_detect_fixed, stri_detect_regex, stri_dup, stri_duplicated, stri_duplicated_any, stri_enc_detect, stri_enc_detect2, stri_enc_fromutf32, stri_enc_get, stri_enc_info, stri_enc_isascii, stri_enc_isutf16be, stri_enc_isutf16le, stri_enc_isutf32be, stri_enc_isutf32le, stri_enc_isutf8, stri_enc_list, stri_enc_mark, stri_enc_set, stri_enc_toascii, stri_enc_tonative, stri_enc_toutf32, stri_enc_toutf8, stri_encode, stri_endswith, stri_endswith_charclass, stri_endswith_coll, stri_endswith_fixed, stri_escape_unicode, stri_extract, stri_extract_all, stri_extract_all_boundaries, stri_extract_all_charclass, stri_extract_all_coll, stri_extract_all_fixed, stri_extract_all_regex, stri_extract_all_words, stri_extract_first, stri_extract_first_boundaries, stri_extract_first_charclass, stri_extract_first_coll, stri_extract_first_fixed, stri_extract_first_regex, stri_extract_first_words, stri_extract_last, stri_extract_last_boundaries, stri_extract_last_charclass, stri_extract_last_coll, stri_extract_last_fixed, stri_extract_last_regex, stri_extract_last_words, stri_flatten, stri_info, stri_isempty, stri_join, stri_join_list, stri_length, stri_list2matrix, stri_locale_get, stri_locale_info, stri_locale_list, stri_locale_set, stri_locate, stri_locate_all, stri_locate_all_boundaries, stri_locate_all_charclass, stri_locate_all_coll, stri_locate_all_fixed, stri_locate_all_regex, stri_locate_all_words, stri_locate_first, stri_locate_first_boundaries, stri_locate_first_charclass, stri_locate_first_coll, stri_locate_first_fixed, stri_locate_first_regex, stri_locate_first_words, stri_locate_last, stri_locate_last_boundaries, stri_locate_last_charclass, stri_locate_last_coll, stri_locate_last_fixed, stri_locate_last_regex, stri_locate_last_words, stri_match, stri_match_all, stri_match_all_regex, stri_match_first, stri_match_first_regex, stri_match_last, stri_match_last_regex, stri_na2empty, stri_numbytes, stri_opts_brkiter, stri_opts_collator, stri_opts_fixed, stri_opts_regex, stri_order, stri_pad, stri_pad_both, stri_pad_left, stri_pad_right, stri_paste, stri_paste_list, stri_rand_lipsum, stri_rand_shuffle, stri_rand_strings, stri_read_lines, stri_read_raw, stri_remove_empty, stri_replace, stri_replace_all, stri_replace_all_charclass, stri_replace_all_coll, stri_replace_all_fixed, stri_replace_all_regex, stri_replace_first, stri_replace_first_charclass, stri_replace_first_coll, stri_replace_first_fixed, stri_replace_first_regex, stri_replace_last, stri_replace_last_charclass, stri_replace_last_coll, stri_replace_last_fixed, stri_replace_last_regex, stri_replace_na, stri_reverse, stri_sort, stri_split, stri_split_boundaries, stri_split_charclass, stri_split_coll, stri_split_fixed, stri_split_lines, stri_split_lines1, stri_split_regex, stri_startswith, stri_startswith_charclass, stri_startswith_coll, stri_startswith_fixed, stri_stats_general, stri_stats_latex, stri_sub, stri_subset, stri_subset_charclass, stri_subset_coll, stri_subset_fixed, stri_subset_regex, stri_timezone_get, stri_timezone_info, stri_timezone_list, stri_timezone_set, stri_trans_char, stri_trans_general, stri_trans_isnfc, stri_trans_isnfd, stri_trans_isnfkc, stri_trans_isnfkc_casefold, stri_trans_isnfkd, stri_trans_list, stri_trans_nfc, stri_trans_nfd, stri_trans_nfkc, stri_trans_nfkc_casefold, stri_trans_nfkd, stri_trans_tolower, stri_trans_totitle, stri_trans_toupper, stri_trim, stri_trim_both, stri_trim_left, stri_trim_right, stri_unescape_unicode, stri_unique, stri_width, stri_wrap, stri_write_lines

I'm afraid I don't have anything constructive to offer in terms of a solution, though :/

[jeroen]

@zkamvar this is usually a temporary problem during the transition that will disappear in a day or two.

[bfifield]

Just wanted to ping that what @zkamvar posted is an ongoing issue for us too with fastLink (error log for windows-oldrel at https://www.r-project.org/nosvn/R.check/r-oldrel-windows-ix86+x86_64/fastLink-00install.html) and has not resolved itself over the past week. When checking other packages that import stringi functions, it appears they have the same error on windows-oldrel. All that is to say, thank you for this package - it's been a huge help for us and hopefully this windows-oldrel issue is easy to solve.

[yutannihilation]

Though the problem seems rather temporal, it's a bit worrying that the check keeps failing for weeks. Anyway, discussing on the closed issue is not a good idea. Should we open a new issue if the problem persists?

ERROR: failed to lock directory 'd:/Rcompile/CRANpkg/lib/3.4' for modifying
Try removing 'd:/Rcompile/CRANpkg/lib/3.4/00LOCK-stringi'
In R CMD INSTALL

https://www.r-project.org/nosvn/R.check/r-oldrel-windows-ix86+x86_64/stringi-00install.html

[gagolews]

Thanks for noticing that. This is again some CRAN-side issue. I've emailed CRAN, let's see what happens next.

[gagolews]

Uwe Ligges solved the problem : https://cran.r-project.org/web/checks/check_results_stringi.html

[yutannihilation]

Congratulations!