Download libs script update (adds #7907 wget2) - params and fixes deploy logs

danoli3 commented 1 month ago

Cherry picks: @dimitre https://github.com/openframeworks/openFrameworks/pull/7907 (add wget2 and parallel downloads)

Further Updated the downloader script to support changes for parallel downloads

Added params to downloader script to support no ssl, multiple downloads in list
Validates download type
New Param: -k / --no-ssl (can be used to bypass SSL encryption if network or router or something MITM - use cautiously )
Silent param: -s
Fixed override to remove just target download type libraries (not all types as was before)
Fixed logging more clear as to what is happening
Validated wget2 / curl / wget working

Cleaned up download_libs with better logs and overwrite settings to the target download type

dimitre commented 1 month ago

Hello @danoli3 I'm testing now in osx and wget2 calls are being called once for each download. the whole point of using wget2 is download all three urls as a unique parameter, so they are being all downloaded simultaneously echoing commands

 [downloader] ... 
Downloading [openFrameworksLibs_bleeding_osx_1.tar.bz2] @ [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2]
/opt/homebrew/bin/wget2
wget2 --progress=bar https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2
Downloading [openFrameworksLibs_bleeding_osx_2.tar.bz2] @ [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2]
/opt/homebrew/bin/wget2
wget2 --progress=bar https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2
Downloading [openFrameworksLibs_bleeding_osx_3.tar.bz2] @ [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2]
/opt/homebrew/bin/wget2
wget2 --progress=bar https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2

EDIT: in the other PR

openFrameworksLibs_b  11% [============>                                                                                                        ]    9.60M    1.04MB/s
openFrameworksLibs_b  39% [=============================================>                                                                       ]    7.73M    1.19MB/s
openFrameworksLibs_b  15% [=================>                                                                                                   ]    8.33M    1.24MB/s
                          [Files: 0  Bytes: 0  [0 B/s] Redirects: 3  Todo: 3  Errors: 0                                                         ]

danoli3 commented 1 month ago

Yeah!! 2>/dev/null was preventing the visualisation!

cURL's parallel needed some work though, fixed that up properly

danoli3 commented 1 month ago

cURL

--Downloading using cURL urls [https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios1.tar.bz2 -o openFrameworksLibs_master_ios1.tar.bz2 -k https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios2.tar.bz2 -o openFrameworksLibs_master_ios2.tar.bz2 -k https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios3.tar.bz2 -o openFrameworksLibs_master_ios3.tar.bz2 -k https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios4.tar.bz2 -o openFrameworksLibs_master_ios4.tar.bz2 ]
DL% UL%  Dled  Uled  Xfers  Live Total     Current  Left    Speed

--  --      0     0     4     4  --:--:-- --:--:-- --:--:--     0      
--  --      0     0     4     4  --:--:-- --:--:-- --:--:--     0      
 30 --  36.9M     0     4     4   0:00:03  0:00:01  0:00:02 35.3M      
 53 --  65.4M     0     4     2   0:00:02  0:00:01  0:00:01 42.3M      
 76 --  93.3M     0     4     1   0:00:02  0:00:02 --:--:-- 45.5M      
 99 --   122M     0     4     1   0:00:02  0:00:02 --:--:-- 47.9M      
100 --   122M     0     4     0   0:00:02  0:00:02 --:--:-- 47.9M

wget2

dimitre commented 1 month ago

woww that's great! I'm testing now in macOS and it is running great with wget2. download_libs.sh 100% and download_latest_libs.sh is downloading them twice. here is the output

d@zen osx % ./download_latest_libs.sh
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [downloader] ... 
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2

/opt/homebrew/bin/wget2
openFrameworksLibs_b 100% [====================================================================================================================>]   18.52M    1.20MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]   24.50M    2.00MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]    6.92M    1.20MB/s
                          [Files: 3  Bytes: 49.95M [2.66MB/s] Redirects: 3  Todo: 0  Errors: 0                                                  ]

 Removing old libraries
 ------ 
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
opencv/lib/osx/opencv.xcframework/macos-arm64_x86_64/libopencv_combined.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [downloader] ... 
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2

/opt/homebrew/bin/wget2
openFrameworksLibs_b 100% [====================================================================================================================>]   18.52M    1.30MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]   24.50M    1.87MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]    6.92M    1.02MB/s
                          [Files: 3  Bytes: 49.95M [2.65MB/s] Redirects: 3  Todo: 0  Errors: 0                                                  ]

 ------ 
 Uncompressing libraries [osx]_[arm64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[arm64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
opencv/lib/osx/opencv.xcframework/macos-arm64_x86_64/libopencv_combined.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments

dimitre commented 1 month ago

Great. downloading once now. it seems to be downloading x86_64 here instead of arm and some other errors uncompressing bzip

 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
opencv/lib/osx/opencv.xcframework/macos-arm64_x86_64/libopencv_combined.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments

danoli3 commented 1 month ago

Been testing on a few computers.

Yeah on Windows wget2 getting a strange timeout:

B%20filename%3DopenFrameworksLibs_bleeding_vs_64_2.zip&response-content-type=application%2Foctet-stream
[1] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/69d920c8-c16d-40d4-90fe-5a187476ed90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T002455Z&X-Amz-Expires=300&X-Amz-Signature=1184c22008fb3bff11bfec47782c5ff72e96be8448323bb695c80aa173041478&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_vs_64_2.zip&response-content-type=application%2Foctet-stream' ...
[0] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/b9c15e1d-a4e2-41e3-8112-81b4118a52c3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T002455Z&X-Amz-Expires=300&X-Amz-Signature=c3e2cec46b247b46dcc20b004dd032a7427a2008596391324f088210a1735f3c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_vs_64_1.zip&response-content-type=application%2Foctet-stream' ...
Saving 'openFrameworksLibs_bleeding_vs_64_2.zip'
Saving 'openFrameworksLibs_bleeding_vs_64_1.zip'
HTTP response 200  [https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/69d920c8-c16d-40d4-90fe-5a187476ed90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T002455Z&X-Amz-Expires=300&X-Amz-Signature=1184c22008fb3bff11bfec47782c5ff72e96be8448323bb695c80aa173041478&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_vs_64_2.zip&response-content-type=application%2Foctet-stream]
Error on or near line ; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments

I was watching the cache files as well, it was downloading them and got stuck and some sort of a loop and stopped getting my bytes, looks like remote hung up on the transfer. Prior calls were fine with wget2, even to the same binaries

Also need to fix these random logs:

/opt/homebrew/bin/wget2

Side effect of checking if command function exists >= null should have handled that

Also if failure the in progress files go into the script dev directory... we should allocate a scrub folder

I think once it starts getting .1 our code to extract them is broke

danoli3 commented 1 month ago

Downloader updates

fixes command determination artefact now tracks installed of curl/wget2/wget in variable
added wget2 params for override or continue last download (-nc)
t20 retries
console formatting
added error message if no commands installed

dimitre commented 1 month ago

have you seen this one? https://amber-lang.com

dimitre commented 1 month ago

I've just tested it seems to be working great here. the script now is showing like it is downloading x86_64 libs, but I'm on arm64

 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_2.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_3.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_3.tar.bz2] to [/libs]

a nice glob command to test all libs architectures

cd $ofw/libs
file **/*.a

and the results

FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive] [arm64]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a (for architecture x86_64):   current ar archive
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a (for architecture arm64):    current ar archive
brotli/lib/osx/brotli.xcframework/macos-arm64_x86_64/brotli.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
brotli/lib/osx/brotli.xcframework/macos-arm64_x86_64/brotli.a (for architecture x86_64):    current ar archive random library
brotli/lib/osx/brotli.xcframework/macos-arm64_x86_64/brotli.a (for architecture arm64): current ar archive random library
cairo/lib/osx/cairo.xcframework/macos-arm64_x86_64/libcairo.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64]
cairo/lib/osx/cairo.xcframework/macos-arm64_x86_64/libcairo.a (for architecture x86_64):    current ar archive random library
cairo/lib/osx/cairo.xcframework/macos-arm64_x86_64/libcairo.a (for architecture arm64): current ar archive random library
curl/lib/osx/curl.xcframework/macos-arm64_x86_64/curl.a:                Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
curl/lib/osx/curl.xcframework/macos-arm64_x86_64/curl.a (for architecture x86_64):  current ar archive random library
curl/lib/osx/curl.xcframework/macos-arm64_x86_64/curl.a (for architecture arm64):   current ar archive random library
fmt/lib/osx/fmt.xcframework/macos-arm64_x86_64/libfmt.a:                Mach-O universal binary with 2 architectures: [x86_64:current ar archive] [arm64:current ar archive]
fmt/lib/osx/fmt.xcframework/macos-arm64_x86_64/libfmt.a (for architecture x86_64):  current ar archive
fmt/lib/osx/fmt.xcframework/macos-arm64_x86_64/libfmt.a (for architecture arm64):   current ar archive
freetype/lib/osx/freetype.xcframework/macos-arm64_x86_64/libfreetype.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
freetype/lib/osx/freetype.xcframework/macos-arm64_x86_64/libfreetype.a (for architecture x86_64):   current ar archive random library
freetype/lib/osx/freetype.xcframework/macos-arm64_x86_64/libfreetype.a (for architecture arm64):    current ar archive random library
glew/lib/osx/glew.xcframework/macos-arm64_x86_64/libGLEW.a:             Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64]
glew/lib/osx/glew.xcframework/macos-arm64_x86_64/libGLEW.a (for architecture x86_64):   current ar archive random library
glew/lib/osx/glew.xcframework/macos-arm64_x86_64/libGLEW.a (for architecture arm64):    current ar archive random library
glfw/lib/osx/glfw.xcframework/macos-arm64_x86_64/libglfw3.a:            Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
glfw/lib/osx/glfw.xcframework/macos-arm64_x86_64/libglfw3.a (for architecture x86_64):  current ar archive random library
glfw/lib/osx/glfw.xcframework/macos-arm64_x86_64/libglfw3.a (for architecture arm64):   current ar archive random library
libpng/lib/osx/libpng.xcframework/macos-arm64_x86_64/libpng.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
libpng/lib/osx/libpng.xcframework/macos-arm64_x86_64/libpng.a (for architecture x86_64):    current ar archive random library
libpng/lib/osx/libpng.xcframework/macos-arm64_x86_64/libpng.a (for architecture arm64): current ar archive random library
openFrameworksCompiled/lib/osx/libopenFrameworks.a:                     current ar archive
pixman/lib/osx/pixman.xcframework/macos-arm64_x86_64/libpixman-1.a:     Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
pixman/lib/osx/pixman.xcframework/macos-arm64_x86_64/libpixman-1.a (for architecture x86_64):   current ar archive random library
pixman/lib/osx/pixman.xcframework/macos-arm64_x86_64/libpixman-1.a (for architecture arm64):    current ar archive random library
pugixml/lib/osx/pugixml.xcframework/macos-arm64_x86_64/libpugixml.a:    Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
pugixml/lib/osx/pugixml.xcframework/macos-arm64_x86_64/libpugixml.a (for architecture x86_64):  current ar archive random library
pugixml/lib/osx/pugixml.xcframework/macos-arm64_x86_64/libpugixml.a (for architecture arm64):   current ar archive random library
rtAudio/lib/osx/rtAudio.xcframework/macos-arm64_x86_64/librtaudio.a:    Mach-O universal binary with 2 architectures: [x86_64:current ar archive] [arm64:current ar archive]
rtAudio/lib/osx/rtAudio.xcframework/macos-arm64_x86_64/librtaudio.a (for architecture x86_64):  current ar archive
rtAudio/lib/osx/rtAudio.xcframework/macos-arm64_x86_64/librtaudio.a (for architecture arm64):   current ar archive
tess2/lib/osx/tess2.xcframework/macos-arm64_x86_64/libtess2.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
tess2/lib/osx/tess2.xcframework/macos-arm64_x86_64/libtess2.a (for architecture x86_64):    current ar archive random library
tess2/lib/osx/tess2.xcframework/macos-arm64_x86_64/libtess2.a (for architecture arm64): current ar archive random library
uriparser/lib/osx/uriparser.xcframework/macos-arm64_x86_64/uriparser.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
uriparser/lib/osx/uriparser.xcframework/macos-arm64_x86_64/uriparser.a (for architecture x86_64):   current ar archive random library
uriparser/lib/osx/uriparser.xcframework/macos-arm64_x86_64/uriparser.a (for architecture arm64):    current ar archive random library
zlib/lib/osx/zlib.xcframework/macos-arm64_x86_64/zlib.a:                Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
zlib/lib/osx/zlib.xcframework/macos-arm64_x86_64/zlib.a (for architecture x86_64):  current ar archive random library
zlib/lib/osx/zlib.xcframework/macos-arm64_x86_64/zlib.a (for architecture arm64):   current ar archive random library

dimitre commented 1 month ago

it would be great if we had ./scripts/osx/download_libs.sh using new downloader also :)

danoli3 commented 1 month ago

You should run file */.a on the ultra mega xcframework with the macos scripts, its massif.

it would be great if we had ./scripts/osx/download_libs.sh using new downloader also :)

Its already powering that too

Fixed: [osx]_[x86_64]

dimitre commented 1 month ago

I had this error now:

./scripts/osx/download_latest_libs.sh
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [openFrameworks downloader] ... 

[0] Downloading 'https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2' ...
HTTP response 302  [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2]
Adding URL: https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream
[0] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream' ...
Saving 'openFrameworksLibs_bleeding_osx_3.tar.bz2'
HTTP response 200  [https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream]

 Removing old libraries
 ------ 
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1

after removing all libs folder it stopped working. download_libs.sh worked ok

and subsequent calling download_latest_libs.sh returns

./scripts/osx/download_latest_libs.sh
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [openFrameworks downloader] ... 

Nothing to do - goodbye

 Removing old libraries
 ------ 
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1

danoli3 commented 1 month ago

Should be fixed now with command -nc overwrite file or resume if file is there and can be.

On Wed, 29 May 2024 at 6:02 AM, Dimitre @.***> wrote:

I had this error now:

./scripts/osx/download_latest_libs.sh -----https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2 [openFrameworks downloader] ...

[0] Downloading 'https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2' ... HTTP response 302 [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2] Adding URL: https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream [0] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream' ... Saving 'openFrameworksLibs_bleeding_osx_3.tar.bz2' HTTP response 200 [https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream]

Removing old libraries

Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2] FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input tar: Error exit delayed from previous errors. Error on or near line 262; exiting with status 1

after removing all libs folder it stopped working. download_libs.sh worked ok

and subsequent calling download_latest_libs.sh returns

./scripts/osx/download_latest_libs.sh -----https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2 [openFrameworks downloader] ...

Nothing to do - goodbye

Removing old libraries

Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2] FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input tar: Error exit delayed from previous errors. Error on or near line 262; exiting with status 1

— Reply to this email directly, view it on GitHub https://github.com/openframeworks/openFrameworks/pull/8002#issuecomment-2136013217, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK2HBKMHEVUNTBZVNIV7DZETPGDAVCNFSM6AAAAABIKURUPKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZWGAYTGMRRG4 . You are receiving this because you were mentioned.Message ID: @.***>

danoli3 commented 1 month ago

Since it has the old _[x86_64] still in there it was the outdated version. I've added a version number so we can trace it

danoli3 commented 1 month ago

Download script v3.0.1 - fixes for cURL in silent mode cURL now has progress bar Version in echo Wget2/wget Verbose turned off Download_libs silent now passed as bool

Download_libs.sh fixed Overwrite

Overwrite / remove old libs was not working at all - not looping correctly for core libs - rewrote loop
Overwrite now removes old libs for that target type for the download... will remove only osx if download osx etc rather than all platforms for that library (need to validate some issues with headers, maybe remove those will check)
Overwrite Addons now will work if folders are deleted

dimitre commented 1 month ago

Good idea! version is great

[openFrameworks downloader v3.0.1] ... 

Nothing to do - goodbye

 Overwrite - Removing old libraries for [osx]
 Removing old libraries: [boost/lib/osx]
 Removing old libraries: [cairo/lib/osx]
 Removing old libraries: [curl/lib/osx]
 Removing old libraries: [FreeImage/lib/osx]
 Removing old libraries: [fmod/lib/osx]
 Removing old libraries: [freetype/lib/osx]
 Removing old libraries: [glew/lib/osx]
 Removing old libraries: [glfw/lib/osx]
 Removing old libraries: [rtAudio/lib/osx]
 Removing old libraries: [tess2/lib/osx]
 Removing old libraries: [uriparser/lib/osx]
 ------ 
 Uncompressing libraries [osx] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 263; exiting with status 1
d@zen of % ./scripts/osx/download_latest_libs.sh
 openFrameworks download_libs.sh v2.0
 -----
 [openFrameworks downloader v3.0.1] ...

danoli3 commented 1 month ago

Nothing to do - goodbye this means theres a local file downloaded already and its corrupted, hmmmm okay another check

dimitre commented 1 month ago

Yes maybe only because I've tested scripts under constructions, but it would be great to handle those also, in the case it happens somewhere else.

danoli3 commented 1 month ago

I thought I'd add another folder to libs

downloader 3.1.0 wget/wget2 - now checks remote file vs local if found and optimises - won't download if same

download_libs 2.1 -

files downloaded (zip/tar.bz2) are now stored in openFrameworks/libs/download/
not removing dl (zip/tar.bz2) now after extraction
code to remove (zip/tar.bz2) on start also done - disabled

danoli3 commented 1 month ago

cURLs progress bar

dimitre commented 1 month ago

Beautiful. install complete here. didnt have to remove any file. Installed latest libs, worked great installed classic libs, all good Terminal output is pretty!

All very fast here in a slow rural internet connection via radio.

Suggestions: I would love to have a different starting name for both scripts, to complete with tab, like "latest_libs.sh" or something

Thanks for all the work. it is great here in macOS. I can test later in Windows

dimitre commented 1 month ago

Good to go. only reminding you of file permissions https://github.com/openframeworks/openFrameworks/pull/7907#discussion_r1602583767

danoli3 commented 1 month ago

Linux 24.04 issues on restart https://github.com/actions/runner-images/issues/9937

danoli3 commented 1 month ago

Sweet merging as fixes linux actions as well

dimitre commented 1 month ago

Yes! I've fixed it by removing sudo apt update which calls some services to restart, including the github actions. I'll do the same for projectGenerator

openframeworks / openFrameworks

Download libs script update (adds #7907 wget2) - params and fixes deploy logs #8002

Removing old libraries

Removing old libraries