build_clang.sh cannot find source tarballs and fails

[cschol]

++ grep -E '(clang|cfe)-12.0.1.src'
++ head -n 1
+ CLANG_PKG=
+ '[' -n '' ']'
+ popd
+ '[' -z '' ']'
+ echo 'Release 12.0.1 not found!'
Release 12.0.1 not found!
+ exit 1
+ _exit
+ EC=1
+ '[' 1 -ne 0 ']'
+ test -z build_clang.sh
+ echo ''
+ echo 'exiting with abnormal exit code (1)'
+ test -n 1

exiting with abnormal exit code (1)

+ '[' -n '' ']'
+ echo ''
+ test build_clang.sh = build.sh
make: *** [Makefile:69: /home/build/rack-plugin-toolchain/local/osxcross] Error 1

CLANG_PKG variable is empty and the release cannot be found.

Same for LLVM_PKG (earlier in log):

<snip>
https://docs.github.com                                                    
https://services.github.com                                             
https://github.blog                                                    
https://github.com/about'                                     
++ grep llvm-12.0.1.src                                       
++ head -n 1                                                                                                                                               
+ LLVM_PKG=                                                                                                                                                                                                                                                                                                           
++ echo 'https://github.githubassets.com                           
https://avatars.githubusercontent.com
https://github-cloud.s3.amazonaws.com                                                                                                                      
https://user-images.githubusercontent.com/   
<snip>

Looking at the page source of https://github.com/llvm/llvm-project/releases/tag/llvmorg-12.0.1, it looks like there are no links to artifacts in it. Maybe the page rendering on Github has changed?

[cschol]

Example for LLVM 12.0.1:

https://api.github.com/repos/llvm/llvm-project/releases/tags/llvmorg-12.0.1

returns the project release data with links to assets at the key browser_download_url.

[cschol]

diff --git a/build_clang.sh b/build_clang.sh
index c854fde..99b02ad 100755
--- a/build_clang.sh
+++ b/build_clang.sh
@@ -36,20 +36,20 @@ function set_package_link()
   pushd $BUILD_DIR &>/dev/null

   DOWNLOAD_PAGE=llvmorg-$CLANG_VERSION
-  download https://github.com/llvm/llvm-project/releases/tag/$DOWNLOAD_PAGE &>/dev/null
+  download https://api.github.com/repos/llvm/llvm-project/releases/tags/$DOWNLOAD_PAGE &> /dev/null

   if [[ $(file $DOWNLOAD_PAGE) == *gzip* ]]; then
     mv $DOWNLOAD_PAGE $DOWNLOAD_PAGE.gz
     require gzip
     gzip -d $DOWNLOAD_PAGE.gz
   fi
-  links=$(cat $DOWNLOAD_PAGE | grep -Po '(?<=href=")[^"]*' | grep -v "\.sig")
+  links=$(cat $DOWNLOAD_PAGE | grep 'browser_download_url')
   rm -f $DOWNLOAD_PAGE
   LLVM_PKG=$(echo "$links" | grep "llvm-$CLANG_VERSION.src" | head -n 1 || true)
   CLANG_PKG=$(echo "$links" | grep -E "(clang|cfe)-$CLANG_VERSION.src" | head -n 1 || true)
   if [ -n "$LLVM_PKG" ] && [[ $LLVM_PKG != https* ]]; then
-    LLVM_PKG="https://github.com/$LLVM_PKG"
-    CLANG_PKG="https://github.com/$CLANG_PKG"
+    LLVM_PKG=$(echo $LLVM_PKG | cut -d\: -f 2-)
+    CLANG_PKG=$(echo $CLANG_PKG | cut -d\: -f 2-)
   fi
   popd &>/dev/null
 }

[akien-mga]

Is there a specific reason why the script parses the HTML instead of constructing the URL directly? As long as LLVM doesn't change the format for their tags and tarball naming, it should work fine with:

VERSION=12.0.1
curl -LO https://github.com/llvm/llvm-project/releases/download/llvmorg-${VERSION}/llvm-${VERSION}.src.tar.xz

If they do change their naming scheme, it would break, but so would the current script AFAICT.

[Relintai]

The root cause of the issue is that the assets list is loaded via javascript after the page now on github, and neither curl nor wget will handle this properly.

If you open a tag like https://github.com/llvm/llvm-project/releases/tag/llvmorg-13.0.1 in your browser you will see a small spinner icon for a short amount of time before the links appear.

Edit: This is where the assets content resides now for the same link: https://github.com/llvm/llvm-project/releases/expanded_assets/llvmorg-13.0.1 I'm not sure whether github will mind using these links directly though.