feature: aria2 multi-threaded downloader with fallback to curl

[jnorthrup]

aria2 has been earning a participation badge in a number of network staples such as in a custom apt-get and in the family of youtube-dl tools.

in each case, aria2 handles less than 100% of the scenarios and there is a fallback for the various non-success conditions, however when it is working it can really eliminate latency overheads and for instance make the transfer experience of a cross-global download a whole lot more pleasan (like for pacific islands, and presumably mars).

the aria2c utility comes with the ability to handle a list of mirrors along with a list of assets and will divide the transfers among the allocated per-connection and per-instance connections, and supports keepalives and http pipelining. there is disk pre[f]alloc to and transfer continuation to whatever value as well.

this suggestion comes on the heels of my experience updating CLR over a 100 megabit connection:

jim@clr-6434c82b828f464fadaa53c054872081~ $ sudo swupd update --verbose -W20 Password: Update started Preparing to update from 35000 to 35920 [...] Starting download of remaining update content. This may take a while... [100%]

Update took 2440.5 seconds, 525 MB transferred [...]

[lebensterben]

libcurl has C API and is both API and ABI stable.

libaria2 only has C++ API and is not stable.

[jnorthrup]

going off my initial CLR update experience, getting (525 megabytes) / (2440 seconds) = 215.163934 kBps, basically ISDN update speeds in 2022 from whatever configuration of distro/mirrors are defaults, I don't really find linker and ABI stability a salient point for avoiding what can be done by exec call.

[fenrus75]

I fully agree update speed matters. a lot. that is what swupd is really supposed to be about.

we don't need aria (C++ which is a problem for C programs) to go multithreaded, we should be able to do that ourselves... .. but that assumes threading would help; it's not automatically clear that it will of course depending on what the bottleneck is

On Mon, Feb 21, 2022 at 5:29 AM Jim Northrup @.***> wrote:

going off my initial CLR update experience, getting (525 megabytes) / (2440 seconds) = 215.163934 kBps, basically ISDN update speeds in 2022 from whatever configuration of distro/mirrors are defaults, I don't really find linker and ABI stability a salient point for avoiding what can be done by exec call.

— Reply to this email directly, view it on GitHub https://github.com/clearlinux/swupd-client/issues/1599#issuecomment-1046882882, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ54FOPGLOFJ5SUE756HNTU4I43DANCNFSM5O5VPOPA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[jnorthrup]

I have no idea how a language bias has anything to do with an expedient fix to constrained distro hosting.

aria2c could run from shell and report 0 or failure in a batch of known objects and mirrors without supervision.

swupd could box this option into a switch, and even make threatening claims about c++ to the hapless swupd user in flashing bold warnings when the switch is selected, but this packaged executable could also open up bitorrent mirroring with almost no pain here if https was not cool enough already.

[lebensterben]

it has nothing to do with language bias.

C doesn't have Class. also very importantly libaria2 doesn't use C style name mangling.

Calling aria2 in a shell is unnecessary. Calling it as a subprocess is doable but still not efficient.

[fenrus75]

this is not about language bias; swupd client is a program written in C which means you can't easily call into C++ code. simple as that.

On Mon, Feb 21, 2022 at 10:54 AM Jim Northrup @.***> wrote:

I have no idea how a language bias has anything to do with an expedient fix to constrained distro hosting.

aria2c could run from shell and report 0 or failure in a batch of known objects and mirrors without supervision.

swupd could box this option into a switch, and even make threatening claims about c++ to the hapless swupd user in flashing bold warnings when the switch is selected, but this packaged executable could also open up bitorrent mirroring with almost no pain here if https was not cool enough already.

— Reply to this email directly, view it on GitHub https://github.com/clearlinux/swupd-client/issues/1599#issuecomment-1047153143, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ54FI7SCMD2TWELO7AV33U4KC7JANCNFSM5O5VPOPA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

[jnorthrup]

Try man exec

[insilications]

I think the CLR (speed) update experience can be improved without resorting to aria.

[lebensterben]

@jnorthrup

but what is the actionable point in the future to revisit any hypothetical improvements for a remote network location like Southeast Asia

You can make PR to this repo, or you may fork swupd-client to add aria2. They are both actionable and are in fact the fundamental practices in OSS.

[jnorthrup]

I think the CLR (speed) update experience can be improved without resorting to aria.

godspeed