Murakami is a tool for creating an automated Internet measurement service,
running in a Docker container. A Murakami measurement container will
automatically run supported tests using a randomized schedule a configurable
number of times a day, and can be configured to export each test result to a
local storage device, to one or more remote servers via SCP, or to a Google
Cloud Storage bucket. Results are saved as individual files in JSON new line
format (.jsonl
).
M-Lab welcomes your contribution to Murakami development. We recommend forking the repository and submitting pull requests to address issues we've identified. If there is a feature you would like to contribute that isn't in our list of issues, please create an issue so that it can be discussed and integrated into our roadmap.
For more information about each supported test see: Supported Tests Runners.
To add new internet measurement tools to Murakami, see: Adding New Measurement Tools to Murakami
Murakami supports Linux operating systems like Ubuntu, Debian, etc. Windows is not supported. Mac OS may work, but is yet untested.
A Murakami container can be configured flexibly depending on the deployment
scenario. If you simply run a Murakami container using M-Lab's latest
image
on Dockerhub, by default all
tests are configured to run four times daily at randomized intervals, but with
no data exporters enabled. To use Murakami and save test results, either using
an individual device or a fleet of managed devices, you'll use one of the
options below to configure the container at runtime, or build your own local
container image.
The Murakami container can be configured/customized using:
configs/murakami.config.example
configuration file and
passing the variables it contains on the command line when running the
Docker containerconfigs/murakami.toml.example
configuration file and including it in a
deployed Docker containerSee CONFIGURING-MURAKAMI for more information on all configuration variables, and in our documentation specific to each deployment scenario.
Murakami supports two types of Docker container deployments on supported systems:
It is also possible to install Murakami directly on supported systems without Docker, however currently documenting direct system installation of Murakami is beyond our project scope. Future testing and documentation is needed to test and confirm supported systems and requirements.
Measurement Lab publishes a pre-built Murakami Docker container image
on Dockerhub tagged latest
. Please visit our repo on
Dockerhub for more details.
If you are interested in building your own Murakami Docker images, please see our BUILD instructions.
Murakami is periodically updated with new measurements and features, as well as
updates to existing measurement services. M-Lab recommends keeping your Murakami installations or
container images up to date with changes in the main branch of this repository,
or with the latest
image on Dockerhub. See the appropriate section on
Updating Murakami in the supported Deployment Scenarios documents
linked above for more information.
Utility scripts that provide some automation to setup Google Cloud resources and
provisioning new devices for a Balena Cloud fleet are available in utilities/
.
These are designed to support M-Lab demo fleets, so please inspect and customize
them to your liking.
setup_bq.sh
- Intended to be run once to setup GCP resources for a new
measurement program or demo. The Google Cloud SDK must be installed and
authorized with an account that has IAM permissions to create GCS buckets
and BigQuery datasets, tables and views.setup_balena_device.sh
- This script can be used to provision new Murakami devices in
your Balena Cloud fleet. It requires the balena-cli to be installed and
authorized.bq_load_murakami.sh
- Intended to be scheduled daily using scheduled jobs or
cron, this script will load murakami test results files within a GCS bucket
from the previous day into BigQuery tables.For more details, see Murakami on Google Cloud Demo.
An optional data visualization service, Murakami-Viz, can be used to receive test results from Murakami measurement devices if desired. Please see USING-MURAKAMI-VIZ for more information.
Two convenience utilities are provided with Murakami:
scripts/convert.py
jsonl
test results to CSV.scripts/upload.py
What do we talk about when we talk about running tests? The M-Lab team talks about this a lot, and the name "Murakami" pays homage to the renown author Haruki Murakami, who's book What I Talk About When I Talk About Running provided us inspiration.
M-Lab would like to thank the Institute for Museum and Library Services (IMLS), and Simmons University, whose partnership on IMLS Award #LG-71-18-0110-18 supported the development of many current Murakami features, as well as Murakami-Viz.