Wrapping pacman to optimize for IPFS usage

[guysv]

So, as I understood it from following this repo's issues, I think we could use a pacman wrapper to fix some of the shortcomings we currently face.

• no need to wait for pacman 6.0's parallel downloads, use aria2c for that instead
• optimize ipfs hash lookups as suggested in the pacman mailing list, but externally.
  • have a (separate/extended) repo database ("core.db") for ipfs hash lookups,
  • add the hashes to the ipfs dir,
  • download the hashes as part of the update and use them to build the download URLs feeded to the downloader
• traditional pacman is still accessible easily this way

I wrote a tiny POC (upgrade-only) wrapper for the first point:

#!/bin/sh
# ipfs-pacman-upgrade.sh

A2FLAGS="--conditional-get=true --continue=true --auto-file-renaming=false --remove-control-file=true -x 16"

if [ ! -f /etc/pacman.d/ipfsgateway ]; then
        echo "/etc/pacman.d/ipfsgateway does not exist"
        exit
fi

ipfs_gateway=$(cat /etc/pacman.d/ipfsgateway)

# download repo dbs
# horrible hack lol
expac -S "${ipfs_gateway}%r/%r.db" | sort -u | aria2c $A2FLAGS -i - -d /var/lib/pacman/sync || exit $?

if [ ! "$(pacman -Quq)" ]; then
        echo 'nothing to upgrade'
        exit
fi

echo "packages to upgrade: $(pacman -Quq | tr '\n' ' ' | fmt --width="$(tput cols)")"
printf "Continue? [Y/n]: "
read -r continue
if [ -n "$continue" ] && [ "$continue" != "y" ]; then
        exit
fi

# download packages
pacman -Quq | xargs expac -S "${ipfs_gateway}%r/%f" -- | aria2c $A2FLAGS -i - -d /var/cache/pacman/pkg || exit $?

# install packages
pacman -Su

For the second point, we need you to add that hashes index to the repo, I'd be happy to discuss that!

[RubenKelevra]

Hey cool! Thanks for looking into that!

Have you tried the mirror in the last days (since the 0.8 update)? Most issues seems to be gone actually.

Not sure where the performance issue was.

Sometimes a download stalls, but not sure if that's not an issue with my LTE connection.

To get the hashes from the files you can do something like this:

@guysv wrote

For the second point, we need you to add that hashes index to the repo, I'd be happy to discuss that!

ipfs ls --size=false --resolve-type=false /ipns/x86-64.archlinux.pkg.pacman.store/community/

Takes like 2 seconds here. :)

Let me know if this is not what you were searching for.

[RubenKelevra]

Ah and you could also run pacman in a crontab like this every 5 minutes or so. It requires nearly no traffic as only when there are changes the database files get updated. Else IPFS figures out that it got them already.

pacman -Syuw --noconfirm

Additionally this helps obviously with the distribution of the packages as soon as they are released, since most people have a bit higher delay to fetch them - in this case you can already offer the files to them.