Define a publishing workflow, possibly with `alr publish` support.

[mosteo]

For the final push towards public beta, I'm going to keep here at the top our target workflow. Older issue information is kept below.

Workflow

Given what we discussed on gitter, I envision two possible workflows; one in which we simply verify input data and generate the index manifest, and an extended one in which we assist the user in gathering information as a previous step. I think we can work both into reusable parts for a single pipeline. For the beta there are many things that can be omitted, but I want to keep the final picture in mind. For the beta I guess a barebones phase 2 could be enough, perhaps some things from phase 1 too.

1. Information gathering phase (tbd during beta? or before?).
   1. [x] The user issues alr publish --tar within an Alire workspace (#529).
   2. [x] We verify the version does not exist in the community index (prevent forgotten version bump).
      1. [x] The version is presented to be confirmed.
   3. [x] We request missing info interactively (maintainer, etc)
      1. [x] And offer to store this info in the global/local config or local manifest for future reuse
   4. [x] We generate the complete user manifest (still alire.toml), bundle it with sources and create the source archive.
   5. [x] Ask the user to upload the file somewhere and to give back the URL (#529).
      1. [ ] Eventually, offer to upload ourselves to a release in the user github account.
2. Verification phase (beta requisite) (second entry point into the process of publishing).
   1. [x] We arrive here with the tarball result of phase 1 as argument, or
   2. [x] the user issues alr publish without URL and we deduce from the local repository (#519), or
   3. [x] the user issues alr publish <URL> [<hash>] (#512).
      • The URL points to the final remote tarball or repository (it contains a complete user manifest, missing only the origin).
        • [x] If given a repository, check against the whitelist (#514).
   4. [x] We extract/clone the sources into a temporary and check it builds successfully (#517).
   5. [x] We recheck manifest completeness if not continuing from phase 1 (#516).
   6. [x] Present final report (informed fields, warn about missing recommended optionals, like tags) (#516).
   7. [x] Emit the index manifest file (received manifest, plus origin) with its proper name (crate-version.toml) (#512).
   8. [x] Give instructions on where this file has to be placed in an index clone (#512).
3. PR creation phase (after beta) (third possible entry point, giving the index manifest as parameter)
   1. [x] Continue from step 2, or start here with alr publish --submit <index manifest file>
   2. [x] Recheck build and manifest if not coming from phase 2.
   3. [x] Fork the community index if the user hasn't a fork.
      1. [x] If it has a fork we should check there isn't a PR already for the same version somehow, and update it in that case.
   4. [x] Clone the user fork, put the index manifest in place, commit back.
   5. [x] Create the PR.

We could aim to make the steps navigable so one can go back/forward, debian-installer style. Not pressing, though.

Notes:

• From 2.x onwards, the user needs not to be in an Alire workspace (we only need the manifest included in the sources).
• From 3.x onwards, we only need the index manifest file.

Other PRs

• [x] Publishing instructions document for the website.
• [x] User-visible changes entry.
• [x] Long help in alr publish.

Older information

For reference, the opam docs:

• https://opam.ocaml.org/doc/Packaging.html
• https://github.com/ocaml/opam-publish

I've received here a reply about using tagged releases which is not entirely reassuring. Still, opam is using these releases. Let's hope the reply comes from an extra-cautious person.

2020-07-21 edit: patches on the way to automated publishing (after talks in #466, #468) (not in order):

• [x] New per-release manifest index format (#478)
  • [x] One release per file in the index
  • [x] Add name, version fields
  • [x] Merge [general] with [x.x.x] and move to top-level
  • [x] Use [[depends-on]] instead of [depends-on]
• [x] Be able to load these new manifests (#479)
• [x] Update templates in alire-index/templates (e90f28f)
• [x] Modify alr with to work in append-only mode (#484)
• [x] Don't require/reject index-only fields in local manifests (origin) (#493)
• [x] Move manifest and lock files to root of sources (#494)
  • [x] Experiment with single manifest + lock file (#490)
• [x] On alr get, rename any user-provided manifest for safekeeping, but use the index-provided one for builds (#487)
  • This ensures repeatability and prevents hidden traps in the upstream manifest
• [x] Update documentation
  • [x] Changes to the manifest format
  • [ ] Changes to workflows
  • [x] Visible user changes entry

2020-04-30 edit: steps we may want to automate:

• [x] Create the tarball
• [x] Compute the hash (fill-in the toml file)
• [ ] Upload the tarball
  • [x] Or use a git origin (via commit/tag)
• [x] Add a release in the .toml file
• [ ] Open/update PR on alire-index

2020-08-04 edit: options to review for PR creation

• [ ] homebrew: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/dev-cmd/bump-formula-pr.rb
• [ ] Check portability/easiness of installing of hub tool
• [x] Check how OPAM is doing it
• [ ] Review github REST API for direct use

Other TODOs:

• [x] ~Check portability/easiness of installing of hub tool~ (moved to previous list)
• [x] ~TOML pretty-printing~ (not needed once manifest is manual only)

[mosteo]

Now I got a clear answer. For whatever reason they're keeping a escape clause open, so we should at least make a recommendation to generate your own tarball, and where possible automate it.

[Fabien-Chouteau]

Should we consider generating the archives with alr publish and hosting them in the index repo or another repo?

I don't know what the size limit is github.

[mosteo]

Should we consider generating the archives with alr publish and hosting them in the index repo or another repo?

I don't think we can allow users to do so; at most we could help with creating the release/uploading the tarball to the user's repo (if it's in github). And we should be careful to avoid that we are spammed with automated PRs.

I don't know what the size limit is github.

According to https://help.github.com/en/github/administering-a-repository/about-releases there's no limit on the global size of releases; only 2GB per file.

Also for reference, here's the release API: https://developer.github.com/v3/repos/releases/

[Fabien-Chouteau]

I don't think we can allow users to do so;

Why not?

at most we could help with creating the release/uploading the tarball to the user's repo (if it's in github).

Maybe with this indeed: https://developer.github.com/v3/repos/releases/

And we should be careful to avoid that we are spammed with automated PRs.

I am not sure this is a problem considering that creating a PR requires a GitHub account. I think GitHub already has an anti spam account protection.

I don't know what the size limit is github.

According to https://help.github.com/en/github/administering-a-repository/about-releases there's no limit on the global size of releases; only 2GB per file.

I mean if we host the archives in a GitHub repo, as checked-in files. How big the repo can be? How many archives can we host?

[mosteo]

OK, I had misunderstood you. Here are the limits: 100MB per file, 100GB per repo, but we enter the radar at 1GB. So, I don't think GitHub would look happily at expressly using a repo for binaries, even if we don't run afoul of any written rule. LFS quotas in the free tier (the 1GB BW) seem too restricted: https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage

I guess to play it safe we should go for what they advise, which is using releases and uploading binaries there. That requires that users publish from a github repo, or directly from a tarball stored elsewhere.

Understood about the spam.

[Fabien-Chouteau]

I didn't know about this GitHub CLI tool called hub.

There is a command to create a release on a repo: https://hub.github.com/hub-release.1.html

This can be useful because it can upload attachments to the release.

$ git archive --format=zip -o my_project-1.0.0.zip --prefix=my_project-1.0.0/ HEAD
$ hub release create -m "version 1.0.0" -a my_project-1.0.0.zip v1.0.0

[TamaMcGlinn]

So, alr publish [index] from within an alire project should put a tar.gz file at the index location, and write/append to the toml file, such as he/helloworld.toml, correct?

[TamaMcGlinn]

For nuget, there is a nuget.spec file that defines which files go into the nuget package file. That saves the system from having hardcoded ignore lists; I understand we don't want the alire directory, but we can't reasonably know which other files (i.e. .gitignore, /bin/) should be excluded.

[mosteo]

So, alr publish [index] from within an alire project should put a tar.gz file at the index location,

Actually the tarball should be put somewhere online, tarballs don't go into the index repository. Our current idea for automation is as a resource of a release in GitHub (unless the packager wants to override that) to ensure immutability.

and write/append to the toml file, such as he/helloworld.toml, correct?

Ideally, yes.

[mosteo]

For nuget, there is a nuget.spec file that defines which files go into the nuget package file. That saves the system from having hardcoded ignore lists; I understand we don't want the alire directory, but we can't reasonably know which other files (i.e. .gitignore, /bin/) should be excluded.

Thanks for the pointer, I will check the nuget specs more closely. I understand .nuget files are maintained by hand? Do the Microsoft tools also help in populating these files?

[TamaMcGlinn]

I write .nuget files by hand, but there may be tools that help with this.

Where should the tarball be rooted? I.e. when I do tar -xf [tarball] should it immediately contain the .gpr file? Does alire extract to the packagename_version_filesystem directory, and then add that to GPR_PROJECT_PATH?

I looked at an example; the Ada PDF Writer package refers to a .zip file (so evidently alire supports zip as well?) that contains a single directory called apdf. Does that mean with statements will need to refer to 'apdf/some_gprfile.gpr' instead of some_gprfile.gpr directly?

[mosteo]

Your tarball/zipball must contain a single folder (like the apdf you mention), name not important, that in turn contains your sources.

Does that mean with statements will need to refer to 'apdf/some_gprfile.gpr' instead of some_gprfile.gpr directly?

No, you must simply with some_gpr.gpr directly. When you define locations of gpr files in the manifest, you must give the complete relative path without including that archival top folder (that is, relative to your sources root previous to zipping).

You'll find more detailed instructions at publishing.md and catalog-format-spec.md

[mosteo]

Completed as of 2.0