Closed juanfcocontreras closed 1 year ago
eddelbuettel
commented
1 year ago Hi there and thanks for quick feedback -- I haven't even had time to announce this release (on CRAN as of this morning). And yes, we added some new code from a supposedly known-good source (Google) but apparently M1/M2 bites. Shuckes as this is not part of the test matrix.
Now: did you try the suggested patch?
eddelbuettel
commented
1 year ago Looking at the bombing file crc32c_prefetch.h, and recognising that the crc32c implementation added to digest was meant to be portable (as I also added a new package crc32c with the full cmake logic to detect architectures I think the key is to just not #define HAVE_MM_PREFETCH as we then skip the whole block
#if HAVE_MM_PREFETCH
#if defined(_MSC_VER)
#include <intrin.h>
#else // !defined(_MSC_VER)
#include <xmmintrin.h>
#endif // defined(_MSC_VER)
#endif // HAVE_MM_PREFETCHIf you have a moment you could help in testing a bug fix candidate. Else I can probably use the 'macbuilder' of the R Project as it is M1 based.
eddelbuettel
commented
1 year ago I have a one-line fix for you:
modified src/crc32c/crc32c_config.h
@@ -12,7 +12,7 @@
#define HAVE_BUILTIN_PREFETCH 1
// Define to 1 if targeting X86 and the compiler has the _mm_prefetch intrinsic.
-#define HAVE_MM_PREFETCH 1
+// #define HAVE_MM_PREFETCH 1
// Define to 1 if targeting X86 and the compiler has the _mm_crc32_u{8,32,64}
// intrinsics.
With the change (and version up'ed to a pre-release 0.6.32.1) it builds and checks at the M1-based Macbuilder (and I am including a copy as the machine will prune its logs), expand 'details' below to see.
benz0li
commented
1 year ago Not surprisingly, the same happens for linux/arm64/v8:
docker run --rm -ti glcr.b-data.ch/r/ver install2.r digest
```
trying URL 'https://cloud.r-project.org/src/contrib/digest_0.6.32.tar.gz'
Content type 'application/x-gzip' length 177411 bytes (173 KB)
==================================================
downloaded 173 KB
* installing *source* package ‘digest’ ...
** package ‘digest’ successfully unpacked and MD5 sums checked
** using staged installation
** libs
using C compiler: ‘gcc (Debian 12.2.0-14) 12.2.0’
using C++ compiler: ‘g++ (Debian 12.2.0-14) 12.2.0’
g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c SpookyV2.cpp -o SpookyV2.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c aes.c -o aes.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c blake3.c -o blake3.o
In file included from /usr/include/string.h:535,
from blake3.c:11:
In function ‘memcpy’,
inlined from ‘compress_subtree_to_parent_node’ at blake3.c:357:5,
inlined from ‘blake3_hasher_update.part.0’ at blake3.c:535:7:
/usr/include/aarch64-linux-gnu/bits/string_fortified.h:29:10: warning: ‘__builtin_memcpy’ reading 64 bytes from a region of size 32 [-Wstringop-overread]
29 | return __builtin___memcpy_chk (__dest, __src, __len,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
30 | __glibc_objsize0 (__dest));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
blake3.c: In function ‘blake3_hasher_update.part.0’:
blake3.c:353:11: note: source object ‘out_array’ of size 32
353 | uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
| ^~~~~~~~~
In function ‘memcpy’,
inlined from ‘compress_parents_parallel’ at blake3.c:244:5,
inlined from ‘compress_subtree_to_parent_node’ at blake3.c:356:9,
inlined from ‘blake3_hasher_update.part.0’ at blake3.c:535:7:
/usr/include/aarch64-linux-gnu/bits/string_fortified.h:29:10: warning: ‘__builtin_memcpy’ writing 32 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
29 | return __builtin___memcpy_chk (__dest, __src, __len,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
30 | __glibc_objsize0 (__dest));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
blake3.c: In function ‘blake3_hasher_update.part.0’:
blake3.c:353:11: note: at offset 32 into destination object ‘out_array’ of size 32
353 | uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
| ^~~~~~~~~
In function ‘memcpy’,
inlined from ‘compress_subtree_to_parent_node’ at blake3.c:357:5,
inlined from ‘blake3_hasher_update.part.0’ at blake3.c:535:7:
/usr/include/aarch64-linux-gnu/bits/string_fortified.h:29:10: warning: ‘__builtin_memcpy’ reading 64 bytes from a region of size 32 [-Wstringop-overread]
29 | return __builtin___memcpy_chk (__dest, __src, __len,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
30 | __glibc_objsize0 (__dest));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
blake3.c: In function ‘blake3_hasher_update.part.0’:
blake3.c:353:11: note: source object ‘out_array’ of size 32
353 | uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
| ^~~~~~~~~
In function ‘memcpy’,
inlined from ‘compress_parents_parallel’ at blake3.c:244:5,
inlined from ‘compress_subtree_to_parent_node’ at blake3.c:356:9,
inlined from ‘blake3_hasher_update.part.0’ at blake3.c:535:7:
/usr/include/aarch64-linux-gnu/bits/string_fortified.h:29:10: warning: ‘__builtin_memcpy’ writing 32 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
29 | return __builtin___memcpy_chk (__dest, __src, __len,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
30 | __glibc_objsize0 (__dest));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
blake3.c: In function ‘blake3_hasher_update.part.0’:
blake3.c:353:11: note: at offset 64 into destination object ‘out_array’ of size 32
353 | uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
| ^~~~~~~~~
In function ‘memcpy’,
inlined from ‘compress_subtree_to_parent_node’ at blake3.c:357:5,
inlined from ‘blake3_hasher_update.part.0’ at blake3.c:535:7:
/usr/include/aarch64-linux-gnu/bits/string_fortified.h:29:10: warning: ‘__builtin_memcpy’ reading 96 bytes from a region of size 32 [-Wstringop-overread]
29 | return __builtin___memcpy_chk (__dest, __src, __len,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
30 | __glibc_objsize0 (__dest));
| ~~~~~~~~~~~~~~~~~~~~~~~~~~
blake3.c: In function ‘blake3_hasher_update.part.0’:
blake3.c:353:11: note: source object ‘out_array’ of size 32
353 | uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
| ^~~~~~~~~
gcc -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c blake3_dispatch.c -o blake3_dispatch.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c blake3_portable.c -o blake3_portable.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c crc32.c -o crc32.o
g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c crc32c.cpp -o crc32c.o
g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I. -I/usr/local/include -fPIC -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c crc32c_portable.cpp -o crc32c_portable.o
In file included from crc32c_portable.cpp:10:
crc32c/crc32c_prefetch.h:18:10: fatal error: xmmintrin.h: No such file or directory
18 | #include
ivanek
commented
1 year ago This fixed it on M1 Mac.
─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
setting value
version R version 4.3.1 Patched (2023-06-26 r84601)
os macOS Ventura 13.4.1
system aarch64, darwin22.5.0
ui X11
language (EN)
collate C
ctype en_US.UTF-8
tz Europe/Zurich
date 2023-06-27
pandoc 3.1.4 @ /opt/homebrew/bin/pandoc
─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.0)
digest * 0.6.32 2023-06-27 [1] local
sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0)
[1] /Library/Frameworks/R.framework/Versions/4.3/Resources/libraryI have a one-line fix for you:
modified src/crc32c/crc32c_config.h @@ -12,7 +12,7 @@ #define HAVE_BUILTIN_PREFETCH 1 // Define to 1 if targeting X86 and the compiler has the _mm_prefetch intrinsic. -#define HAVE_MM_PREFETCH 1 +// #define HAVE_MM_PREFETCH 1 // Define to 1 if targeting X86 and the compiler has the _mm_crc32_u{8,32,64} // intrinsics.With the change (and version up'ed to a pre-release 0.6.32.1) it builds and checks at the M1-based Macbuilder (and I am including a copy as the machine will prune its logs), expand 'details' below to see.
juanfcocontreras
commented
1 year ago This fixed it on M1 Mac.
Same here! I think you can close this issue.
Thank you very much!
eddelbuettel
commented
1 year ago Thanks for the confirmations. I also had good luck on the M1 build service.
I'll get a new version to CRAN in the next few days.
eddelbuettel
commented
1 year ago I'll get a new version to CRAN in the next few days.
I actually sent it late yesterday, but per the usual CRAN Incoming Dashboard it and by now a good 100 other packages are stuck in 'pre-test'. I am sure it will get uncorked in a due course but for now we are stuck.
The github (and r-universe) version works, and do the preceding versions.
alecristia
commented
1 year ago Thanks for this package & fix! I tried the recommended command in r-universe
install.packages('digest', repos = c('https://eddelbuettel.r-universe.dev', 'https://cloud.r-project.org'))
yields
Error in download.file(url, destfile, method, mode = "wb", ...) : cannot open URL 'https://eddelbuettel.r-universe.dev/src/contrib/digest_0.6.32.1.tgz' In addition: Warning message: In download.file(url, destfile, method, mode = "wb", ...) : cannot open URL 'https://eddelbuettel.r-universe.dev/src/contrib/digest_0.6.32.1.tgz': HTTP status was '404 Not Found' Warning in download.packages(pkgs, destdir = tmpd, available = available, : download of package ‘digest’ failed
I also tried installing older versions from gzip but got this error:
install.packages("~/Desktop/digest_0.6.31.tar.gz", repos = NULL, type = "source")
Installing package into ‘/Users/acristia/Documents/gitrepos/broad-strokes-lena/renv/library/R-4.3/aarch64-apple-darwin20’ (as ‘lib’ is unspecified) Error in startsWith(contriburl, "file:") : non-character object(s)
And I can't install the github version because trying to install devtools requires digest.
I can try the hot fix given above if you could walk me through it?
Or I can totally be patient and wait for the release- in which case feel free to ignore this comment!
eddelbuettel
commented
1 year ago Could that be you / your network?
edd@rob:/tmp$ wget https://eddelbuettel.r-universe.dev/bin/macosx/contrib/4.3/digest_0.6.32.1.tgz
--2023-06-29 12:37:03-- https://eddelbuettel.r-universe.dev/bin/macosx/contrib/4.3/digest_0.6.32.1.tgz
Resolving eddelbuettel.r-universe.dev (eddelbuettel.r-universe.dev)... 2606:4700:3030::ac43:ac1a, 2606:4700:3033::6815:37b4, 172.67.172.26, ...
Connecting to eddelbuettel.r-universe.dev (eddelbuettel.r-universe.dev)|2606:4700:3030::ac43:ac1a|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cdn.r-universe.dev/f9038448926268fd337670a5df55257f/digest_0.6.32.1.tgz [following]
--2023-06-29 12:37:03-- https://cdn.r-universe.dev/f9038448926268fd337670a5df55257f/digest_0.6.32.1.tgz
Resolving cdn.r-universe.dev (cdn.r-universe.dev)... 2606:4700:3033::6815:37b4, 2606:4700:3030::ac43:ac1a, 104.21.55.180, ...
Connecting to cdn.r-universe.dev (cdn.r-universe.dev)|2606:4700:3033::6815:37b4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 205132 (200K) [application/x-gzip]
Saving to: ‘digest_0.6.32.1.tgz’
digest_0.6.32.1.tgz 100%[=====================================================================================================================>] 200.32K --.-KB/s in 0.02s
2023-06-29 12:37:03 (9.68 MB/s) - ‘digest_0.6.32.1.tgz’ saved [205132/205132]
edd@rob:/tmp$ md5sum digest_0.6.32.1.tgz
f9038448926268fd337670a5df55257f digest_0.6.32.1.tgz
edd@rob:/tmp$That was the binary (and you may have to tell install.packages() you want one -- I am not a macOS user). Source works for me too:
edd@rob:/tmp$ wget https://eddelbuettel.r-universe.dev/src/contrib/digest_0.6.32.1.tar.gz
--2023-06-29 12:38:43-- https://eddelbuettel.r-universe.dev/src/contrib/digest_0.6.32.1.tar.gz
Resolving eddelbuettel.r-universe.dev (eddelbuettel.r-universe.dev)... 2606:4700:3033::6815:37b4, 2606:4700:3030::ac43:ac1a, 104.21.55.180, ...
Connecting to eddelbuettel.r-universe.dev (eddelbuettel.r-universe.dev)|2606:4700:3033::6815:37b4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cdn.r-universe.dev/9e4b578071f00218c8dcc355f62b5a7a/digest_0.6.32.1.tar.gz [following]
--2023-06-29 12:38:44-- https://cdn.r-universe.dev/9e4b578071f00218c8dcc355f62b5a7a/digest_0.6.32.1.tar.gz
Resolving cdn.r-universe.dev (cdn.r-universe.dev)... 2606:4700:3033::6815:37b4, 2606:4700:3030::ac43:ac1a, 104.21.55.180, ...
Connecting to cdn.r-universe.dev (cdn.r-universe.dev)|2606:4700:3033::6815:37b4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 325510 (318K) [application/x-gzip]
Saving to: ‘digest_0.6.32.1.tar.gz’
digest_0.6.32.1.tar.gz 100%[=====================================================================================================================>] 317.88K --.-KB/s in 0.04s
2023-06-29 12:38:44 (8.18 MB/s) - ‘digest_0.6.32.1.tar.gz’ saved [325510/325510]
edd@rob:/tmp$ md5sum digest_0.6.32.1.tar.gz
9e4b578071f00218c8dcc355f62b5a7a digest_0.6.32.1.tar.gz
edd@rob:/tmp$ ls -l digest_0.6.32.1.t*
-rw-rw-r-- 1 edd edd 325510 Jun 27 22:30 digest_0.6.32.1.tar.gz
-rw-rw-r-- 1 edd edd 205132 Jun 27 22:30 digest_0.6.32.1.tgz
edd@rob:/tmp$ Maybe check with folks around you if there is a known networking issue?
digest is being processed at CRAN so this should be resolved 'soon'. There is also the Archive section at CRAN to fetch the previous version.
[Minutes later]
But I see now that the GitHub website is laggy / down. Maybe it is them .... and it is best to try later.
benz0li
commented
1 year ago Current status:
waiting: the CRAN team is waiting for an answer from you, e.g. because issues are present that CRAN cannot automatically check for, such as maintainer changes (check your e-mail …)
eddelbuettel
commented
1 year ago Yes. I responded half a day ago. Given the large-ish number of reverse depends, one sometimes gets false positives where something else blew up for an unrelated reason, or a new check tilts the comparison to before. I argued that that was the case here. We shall see once Europe gets up and to work, I am hopeful we will know more "soon".
alecristia
commented
1 year ago wget https://eddelbuettel.r-universe.dev/bin/macosx/contrib/4.3/digest_0.6.32.1.tgz
this worked for me:
1) manually download tgz (for mac) from r-universe
2) run within rstudio:
install.packages("~/Desktop/digest_0.6.32.1.tgz", repos = NULL, type = .Platform$pkgType)
shashanksinghnain
commented
1 year ago wget https://eddelbuettel.r-universe.dev/bin/macosx/contrib/4.3/digest_0.6.32.1.tgzthis worked for me:
- manually download tgz (for mac) from r-universe
- run within rstudio:
install.packages("~/Desktop/digest_0.6.32.1.tgz", repos = NULL, type = .Platform$pkgType)
Not working for me, I get the following error when I run a code which uses the package (M2 Mac Pro) --
(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
eddelbuettel
commented
1 year ago That is something you need to sort out with the R-universe site. The package installs fine from source so do
install.packages('digest', repos = c('https://eddelbuettel.r-universe.dev', 'https://cloud.r-project.org'), type="source")
schlichtanders
commented
1 year ago Running into this on Arm8 Ampere Altra. It seems CRAN has still the old version, right? Anyone any estimation when the fix will arrive in CRAN?
eddelbuettel
commented
1 year ago For situations like this, CRAN as an Archive/ directory for every package. You can always go back to a working version.
The answer to your second question is in this thread, and at CRAN. The package was uploaded. We are all waiting. Maybe if you ask them they will tell you, they have not told me.
The most obvious answer to your problems is also here. The version in this very repo works, as does the (identical) version we have been talking about at length that you get via r-universe: It simply copies from here.
In other words the pending 0.6.33 at CRAN is identical to the 0.6.32.1 you find here or at r-universe -- apart from the update to DESCRIPTION (and then ChangeLog) as I just do not commit the version number change before acceptance at CRAN.
Lastly, I am truly sorry for the M1/M2/Arm64 breakage. That code was added and commited here months ago but clearly we a) could do with such hardware for tests at CI and/or b) it would be nice if those of you with access to such more exotic (to me) hardware tried the main branch here every now and then. We all would have known sooner :crying_cat_face:
eddelbuettel
commented
1 year ago Current state of the world:
Email from CRAN to me dated June 27: "Please correct before 2023-07-11 to safely retain your package on CRAN."
Upload to CRAN dated June 28 after which they were 'effectively' closed for a day or two, they then ran reverse depends, sent me an email with two perfectly false positives to which I of course responded immediately -- and I have not heard since. I have sent two more email, they remain unanswered.
<sarcasm mode> At least we are guaranteed to find out one way or the other by June 11. </sarcasm mode>
eddelbuettel
commented
1 year ago Word is that (at least part of) the delay is due to some hardware woes in Dortmund, but there may be a resolution tomorrow.
Ashastry2
commented
1 year ago Hi, is this issue resolved? I am unable to load other packages such as devtools because of this error
eddelbuettel
commented
1 year ago @Ashastry2 Please read all the messages on this page.
eddelbuettel
commented
1 year ago Still nothing new. State of the world is that
This is more than a little irritating but even with additional polite behind the scenes probe I have not heard anything. At this point I have to assume the package may be thrown off CRAN on July 11 despite a new version in incoming.
eddelbuettel
commented
1 year ago Always lovely to see:
Thanks, on its way to CRAN.
Of course, standard (presumably scripted) form reply without any hint as to why this took nine days.
schlichtanders
commented
1 year ago https://cran.r-project.org/web/packages/digest/index.html
Version 0.6.33 is live.
As it took soo long for this Bugfix to arrive, I was looking into alternatives just today to make R deployment more stable. As the Microsoft time travel R cran was shutdown this month, I felt a bit lost but luckily found today that another time travel CRAN exists. For those others who experienced a similar shock, this an easy way to make sure your R setup cannot be broken by standard CRAN.
juanfcocontreras
commented
1 year ago CRAN version 0.6.33 installed here without any issue :-)
eddelbuettel
commented
1 year ago CRAN version 0.6.33 installed here without any issue :-)
Given that it only exists in source right now at CRAN and no binaries are listed , you must have installed from source, and I hope you are aware that you have done so any of the last seven or eight days by using this repo, or r-universe, or the CRAN Archive links....
I had planned to keep this open til the binaries are built but there is really no point, and the M1 builder for macOS had been consistently slower than those for Windows or x86_64 macOS so I am changing my and will close this as completed now.
Sorry for the all the "pain and suffering". It was an honest mistake / oversight committed in March here and available for early bird testers since then. The fix was a one-liner but sadly this time the passage was little delayed. All good now, so ... onwards!
Maybe the fix shown here could be also applicable?
https://bugzilla.mozilla.org/show_bug.cgi?id=1767947
Full trace: