search

racket / racket-pkg-website

A frontend for the Racket Package Catalog.
Other
11 stars 16 forks source link
racket

readme

The Racket Package Catalog Server

The Racket Package Catalog comprises two pieces of software that work in tandem:

Both frontend and backend run in the same Racket process, but as separate server threads. Each also has additional threads for periodic and internal tasks. The frontend sends package-change requests to the backend, and it watches for periodic updates (especially new checksums) from the pkg-index backend. The servers were originally implemented separately, and there is some value to keeping them conceptually separate.

Although the implementation here is not necessarily tied to the main Racket deployment's configuation, various configuration options make the most sense in terms of the main deployment's structure:

Prerequisites

In addition to Racket v8.14.0.3 or later, you will need to install the following Racket packages:

raco pkg install --skip-installed \
     'https://github.com/racket/infrastructure-userdb.git#main' \
     reloadable \
     aws \
     s3-sync \
     plt-service-monitor

Configuration

You can run the server in test mode with make run or ./run or racket src/main.rkt with no further configuration, except that

By default, all package state and generated files go into compiled in the current directory. (The current directory when you run the server doesn't have to be the top of this Git repo checkout.)

An advantage of using make run or ./run is that it sets PLTSTDERR to turn on lots of logging. You can run just the frontend as src/website/main.rkt or just the backend with src/pkg-index/main.rkt. Running just the frontend requires running the backend at least once to generate its output for the frontend, though, or configuring the frontend to use another source via pkg-index-url as described below.

When you use make run or ./run, it actually runs configs/${CONFIG}.rkt, so you can set CONFIG as an environment variable or makefile variable to pick a configuration there. If CONFIG is not defined, testing is used (which is an empty configuration). For example, to select configs/live.rkt, set CONFIG to live. A good place to do this is in the run-prelude file; see the description of run-prelude below.

Within a configuration file, configuration details are to be given as a hashtable to main. Whenusing the testing configuration of ./run or when using racket src/main.rkt, you can supply a --config argument to specify a module that exports a config hashtable.

Keys useful for deployment:

Keys useful for development:

Backend keys for a pkg-index configuration table within the main configuration:

Development Setup

Adding packages

Instead of manually adding packages to a fresh instance of the package web server, use raco pkg catalog-copy to copy an existing catalog into a directory tree. Then, move the pkg directory (no "s") in the catalog copy to be root/pkgs (with "s") where root is the server's root directory — so, "compiled/root/pkgs" by default.

  $ raco pkg catalog-copy https://pkgs.racket-lang.org compiled/pkgs-copy
  $ mkdir -p compiled/root/pkgs
  $ mv compiled/pkgs-copy/pkg/* compiled/root/pkgs/

Beware, however, that the backend will start by updating the checksum of every package, so consider using a specific package name in place of * or a glob that selects a small set of packages.

Warm up

When the server is started, the backend starts by looking for new checksums, while the frontend immediately checks for backend updates. Since those run concurrently, you may not immediately see updates via the frontend even when the backend has completed its scan (which you might infer from logging output). At that point, restarting is the fastest way to warm up the frontend.

Adding users

You can use the website frontend to add a user. Email for a new user is sent via sendmail, so if you don't have that configured, just watch the logs to see the token that would have been sent.

Automatic code reloading

If you would like to enable the automatic code-reloading feature, set the environment variable SITE_RELOADABLE to a non-empty string or set the reloadable? configuration variable to #t.

You must also delete any compiled code .zo files. Otherwise, the system will not be able to correctly replace modules while running.

Therefore, when using automatic code reloading, use just

make run

and make sure to run make clean beforehand, if you've run make compile at all previously.

Deployment

Static Content

The site can be set up to run either

  1. entirely dynamically, generating package pages on-the-fly for each request;
  2. both statically and dynamically, with HTML renderings of package pages stored on and served from disk like other static resources such as Javascript and CSS; or
  3. both statically and dynamically, as the previous option, but additionally replicating both static and generated content to a local file-system directory and invoking an optional update hook that can be used to further replicate the content to S3 or a remote host.

The default is mixed static/dynamic, with no additional replication.

For a fully dynamic site, set configuration variable disable-cache? to #t.

To enable replication, set configuration variable static-content-target-directory to a non-#f value, and optionally set static-content-update-hook to a string containing a shell command to execute every time the static content is updated.

S3 Content

To set up an S3 bucket — let's call it s3.example — for use with this site, follow these steps:

  1. Create the bucket ("s3.example")
  2. Optionally add a CNAME record to DNS mapping s3.example to s3.example.s3-website-us-east-1.amazonaws.com. If you do, static resources will be available at http://s3.example/; if not, at the longer URL.
  3. Enable "Static Website Hosting" for the bucket. Set the index document to index.html and the error document to not-found.

Then, under "Permissions", click "Add bucket policy", and add something like the following.

{
  "Id": "RacketPackageWebsiteS3Policy",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RacketPackageWebsiteS3PolicyStmt1",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::s3.example",
                   "arn:aws:s3:::s3.example/*"],
      "Principal": {
        "AWS": ["<<<ARN OF THE USER TO WHOM ACCESS SHOULD BE GRANTED>>>"]
      }
    }
  ]
}

The user will need to be able to read and write objects and set CORS policy. (CORS is configured automatically by code in src/static.rkt.)

Supervision

Startable using djb's daemontools; symlink this directory into your services directory and start it as usual. The run script starts the program, and log/run sets up logging of stdout/stderr.

If the file run-prelude exists in the current directory on startup, it will be dotted in before racket is invoked. A prelude is useful to update PATH for a locally-built racket bin directory or to select an appropriate CONFIG setting.

On Debian, daemontools can be installed with apt-get install daemontools daemontools-run, and the services directory is /etc/service/.

Control signals

You can send signals to the running service by creating files in /etc/service/webservice/signals/. For example:

See src/signals.rkt for details of the available signals.

So long as sudo chmod 0777 /etc/service/webservice/signals, these are useful for non-root administrators to control the running service.

In particular, a git post-receive hook can be used to create the .pull-required signal in order to update the service on git push.

Copyright and License

Copyright © 2014 Tony Garnock-Jones

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.