lf-
commented
2 months ago I think the cleanest way to implement this in a maximally generic way is to implement Nix's immutable tarball protocol as used by flakehub and as of yesterday, forgejo, which is a trivial amount of code to possibly just upstream into all the forges.
With this you can just give https://some-forgejo/user/repo/archive/main.tar.gz, which is a URL that does not need to be inspected at all.
Here is some hastily written Python I wrote to implement it today:
import subprocess
import tempfile
from pathlib import Path
import re
import dataclasses
from typing import Literal
import urllib.parse
import json
@dataclasses.dataclass
class PinSerialized:
kind: str
rev: str | None
nar_hash: str
@dataclasses.dataclass
class TarballPinSerialized(PinSerialized):
kind: Literal['tarball']
locked_url: str
url: str
class PinSpec:
def do_pin(self) -> dict[str, str]:
raise ValueError('unimplemented')
@dataclasses.dataclass
class TarballPinSpec(PinSpec):
url: str
def do_pin(self) -> TarballPinSerialized:
return lock_tarball(self.url)
@dataclasses.dataclass
class LinkHeader:
url: str
rev: str | None
LINK_HEADER_RE = re.compile(r'<(?P<url>.*)>; rel="immutable"')
def parse_link_header(header) -> LinkHeader | None:
matched = LINK_HEADER_RE.match(header)
if not matched:
return None
url = matched.group('url')
parsed_url = urllib.parse.urlparse(url)
parsed_qs = urllib.parse.parse_qs(parsed_url.query)
return LinkHeader(url=url, rev=next(iter(parsed_qs.get('rev', [])), None))
def lock_tarball(url) -> TarballPinSerialized:
"""
Prefetches a tarball using the Nix immutable tarball protocol
"""
import requests
resp = requests.get(url)
with tempfile.TemporaryDirectory() as td:
td = Path(td)
proc = subprocess.Popen(["tar", "-C", td, "-xvzf", "-"],
stdin=subprocess.PIPE)
assert proc.stdin
for chunk in resp.iter_content(64 * 1024):
proc.stdin.write(chunk)
proc.stdin.close()
if proc.wait() != 0:
raise RuntimeError("untarring failed")
children = list(td.iterdir())
# FIXME: allow different tarball structures
assert len(children) == 1
child = children[0].rename(children[0].parent.joinpath('source'))
sri_hash = subprocess.check_output(
["nix-hash", "--type", "sha256", "--sri", child]).decode().strip()
path = subprocess.check_output(
["nix-store", "--add-fixed", "--recursive", "sha256",
child]).decode().strip()
link_info = parse_link_header(resp.headers['Link'])
print(sri_hash, path)
return TarballPinSerialized(kind='tarball',
nar_hash=sri_hash,
locked_url=link_info.url if link_info else url,
rev=link_info.rev if link_info else None,
url=url)
andir
I somehow just remembered that there are other ways to host your software other than GitHub and GitLab. Of course we support all generic git repositories, nevertheless there are more forges which could benefit from the additional support.
Potential forges:
Questions that need answering for each one:
user/repostructure for all repositories like GitHub or is it more free-form like GitLab?To generally keep the scope down, the current restrictions will also be required for new candidates:
git ls-remoteetc.)curl. Things that require adding a library for the specific API implementation are out of scope.