[infrastructure] Automate releases.llvm.org

tru commented 1 year ago

After some ad-hoc discussions with @tstellar and @asl in various places, I wanted to summarize the current questions and the path forward in an issue since I think we should automate this part of the release process asap.

Current process

Currently, we have the llvm/www-release repo where we change a js and add the docs to the subdir, commit, and push
This, unfortunately, doesn't update the actual page. You then have to ssh into the machine and pull. This might fail because of permissions.
A license file also needs to be copied to the right place and is often forgotten. See #57672

This is a suboptimal process that usually fails because of permissions or because we forget to update stuff.

Suggestions

Automatic deploys

The lowest hanging fruit is to deploy from the repo to the server via rsync automatically. There is a rsync plugin for GitHub actions already available. https://github.com/marketplace/actions/action-rsync. This needs someone to help setup the ssh keys for an automated deploy. I don't have enough access on the CDN/www machine to help with this.

This would fix the permission problem.

Autogenerated page

As part of a PoC I converted the page to Hugo here: https://github.com/tru/releases.llvm.org, which means that the release pages would be autogenerated and no more editing html/js files.
This is almost done but requires some fixes to be really deployable: The current DIRECTORY (not in the repo) contains OLD binaries from very old releases. These shouldn't be added to a Git repo since they are big. My suggestion would be to have a new webpage/subpage where we can place these statically and just refer from the Hugo pages since they will no longer be updated. I would again need help to set this up since I don't have access to that part of the infrastructure. documentation should also not be checked-in to the repo, really. We should have a similar way to handle those, docs.llvm.org/<version>/index.html or similar.

TLDR

I am happy to help getting both the auto-deploy and the hugo generated pages working as long as I can get someone to commit to help me with the infrastructure parts that we need and I can't do, these items are:

SSH key for deploying from github actions to a (preferably) new directory on the www server that's only writeable from this new service user.
New directories for docs and old binaries that can be referenced from the page
When everything works someone needs to repoint releases.llvm.org to the new directory.

Thoughts @tstellar @asl?

llvmbot commented 1 year ago

@llvm/issue-subscribers-infrastructure

tru commented 1 year ago

ping @kbeyls and @rnk as well.

asl commented 1 year ago

My suggestion would be to have a new webpage/subpage where we can place these statically and just refer from the Hugo pages since they will no longer be updated.

I think what is important here is not to break URLs of these old releases. Everything else certainly could be flexible.

tru commented 1 year ago

I think what is important here is not to break URLs of these old releases

Hmm. If you think that's important we should find a solution to that, I doubt many people still access these - but maybe my feeling is off. We might be able to handle that with Location headers so they don't get dead links but still store the artifacts on another url.

tstellar commented 1 year ago

Can we host releases.llvm.org an GitHub pages rather than trying to sync it to the server?

brad0 commented 1 year ago

It would be nice to have some resolution to these issues.

tru commented 1 year ago

Can we host releases.llvm.org an GitHub pages rather than trying to sync it to the server?

I don't see any technical reason why? @asl do you?

I would like to move forward on this one since the 17.x release will start soon. What can I do to help out?

tstellar commented 1 year ago

@tru I think the only reason we haven't done this for other LLVM websites is because those use server-side includes which github pages does not support. So if you aren't using server-side includes then I think we should do it. Can you enable github pages for your repo, so we can see what it looks like.

tru commented 1 year ago

I added github pages upload and automation: https://tru.github.io/releases.llvm.org/ it looks good, I think. Note that I haven't added 16.x docs yet.

This streamlines the updating of that page. We just need to figure out what to do with old binaries and docs.

My suggestion:

Point releases.llvm.org to github so it can be used from a repo under the llvm-project.
Rename releases.llvm.org to old-releases.llvm.org and add Location header redirects from the new github page.
Ideally we would have a new place (docs.llvm.org) where we just add all the docs per release instead of checking them into the git repo, it makes deploying take quite some time. but this is optional, we can check it in for now and move to something else in the future.

tstellar commented 1 year ago

@tru the page looks good.

Point releases.llvm.org to github so it can be used from a repo under the llvm-project.

Rename releases.llvm.org to old-releases.llvm.org and add Location header redirects from the new github page.

@asl What do you think about this?

tru commented 1 year ago

Ping on this again

brad0 commented 1 year ago

Personally I don't care about the pages being auto-generated so much as I do about the content being updated automatically. That's the immediate issue. You still have to update the https://llvm.org page too.

I see the other websites point to an IP of 54.67.122.174 which is at Amazon. releases.llvm.org is fronted by a CDN, but is the content on the backend on the same 54.67.122.174 host? I would think it is on something else.

brad0 commented 11 months ago

I don't understand how it is possible for the www-releases repo content to be different from what is being displayed on the actual https://releases.llvm.org/ website.

tru commented 11 months ago

I don't understand how it is possible for the www-releases repo content to be different from what is being displayed on the actual https://releases.llvm.org/ website.

The site on the server is not deployed from the repo because of permissions issues that I have been trying to get someone to fix for a long time. That means that I have to commit my change to the repo and the ssh to the server and manually edit the html files on the server with vim. Which of course leads to mistakes and problems.

llvm / llvm-project