Web app for creating and conducting AB and ABX listening tests.
ABX app is available at abxtests.com. You can use this to create your own listening tests by creating a simple configuration file, saving it to a publicly hosted location along with the audio samples and sending the participants link with the configuration file. A listening test link looks like this:
https://abxtests.com/?test=https://www.dropbox.com/s/c00dylhi7ekqkfw/demo.yml?dl=0
The link after ?test=
is the Dropbox share link of the configuration file. Replace the link with your own.
Dropbox works nicely for this purpose but so does any other hosting solution. The only requirements are that the service actually sends the file with correct content type. Dropbox share links are meant for humans and come with the whole Dropbox UI, but ABX ap will automatically turn the link into a correct download link. Unfortunately Google Drive doesn't provide correct content types and therefore cannot be used for hosting the configuration files and audio clips.
The configuration file itself is written with YAML syntax. This is rather human friendly and should be easy enough for you to get started right away without any prior experience with it. If you need to make changes during the study, you should always create a new separate file and distribute the new links to the participants instead of editing the existing file! This keeps things consistent.
Here's an example file:
name: Demo test
welcome:
description: |-
#### ABX Demo
This is a demo listening test showing how ABX listening tests app works from the listener's perspective.
In this listening test you'll be given three different options with different compression methods
and your task is to choose your preferred option. There are two different clips with 10 iterations
each. This is an AB test. ABX tests where the listener needs to tell if sample X is the same as
sample A or sample B. Greater number of sample options is supported although it's recommended to
keep the number relatively low to avoid overloading the listener.
You'll see your results at the end. It would also be possible to have the test send the results to
the researcher by supplying an email in the configuration file but this test does not do so. The form
below is an example of what can be done to gather statistics about the different demographics. The
email sent to the research would also include the form data.
form:
- name: Age
inputType: number
- name: Gender
inputType: select
options:
- Female
- Male
- Other
- name: Experience
inputType: select
options:
- Trained listener
- Audio engineer
- Audio retailer
- Audio reviewer
- Self-proclaimed audiophile
- Musician
- None of the above
results:
description: |-
### Results
Here are your results
options:
- name: Sons of Winter and Stars (lossless)
audioUrl: https://www.dropbox.com/s/9e92quf9tr7aj8s/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%20lossless.wav?dl=0
tag: lossless
- name: Sons of Winter and Stars (64 kbps MP3)
audioUrl: https://www.dropbox.com/s/epvit0keu9w7emp/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%2064%20kbps.mp3?dl=0
tag: 64 kbps
- name: Sons of Winter and Stars (32 kbps MP3)
audioUrl: https://www.dropbox.com/s/6y0av9tydwsfv6y/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%2032%20kbps.mp3?dl=0
tag: 32 kbps
- name: Hotel California (lossless)
audioUrl: https://www.dropbox.com/s/9j3zdynxrfipezn/Eagles%20-%20Hotel%20California%20lossless.wav?dl=0
tag: lossless
- name: Hotel California (64 kbps MP3)
audioUrl: https://www.dropbox.com/s/5p1gayvfmb3fc0z/Eagles%20-%20Hotel%20California%2064%20kbps.mp3?dl=0
tag: 64 kbps
- name: Hotel California (32 kbps MP3)
audioUrl: https://www.dropbox.com/s/bu5aa181sk3q2ca/Eagles%20-%20Hotel%20California%2032%20kbps.mp3?dl=0
tag: 32 kbps
- name: Bird on a Wire (lossless)
audioUrl: https://www.dropbox.com/s/5froi1pxux8pns0/Jennifer%20Warnes%20-%20Bird%20on%20a%20Wire%20lossless.wav?dl=0
tag: lossless
- name: Bird on a Wire (64 kbps MP3)
audioUrl: https://www.dropbox.com/s/7jb5cif5o5bda9n/Jennifer%20Warnes%20-%20Bird%20on%20a%20Wire%2064%20kbps.mp3?dl=0
tag: 64 kbps
- name: Bird on a Wire (32 kbps MP3)
audioUrl: https://www.dropbox.com/s/2qfxv4x346p1xdi/Jennifer%20Warnes%20-%20Bird%20on%20a%20Wire%2032%20kbps.mp3?dl=0
tag: 32 kbps
tests:
- testType: ABX
name: Jennifer Warnes - Bird on a Wire
description: Select the most preferred option
options:
- Bird on a Wire (lossless)
- Bird on a Wire (64 kbps MP3)
- Bird on a Wire (32 kbps MP3)
repeat: 10
- testType: ABX
name: Eagles - Hotel California
description: Select the most preferred option
options:
- Hotel California (lossless)
- Hotel California (64 kbps MP3)
- Hotel California (32 kbps MP3)
repeat: 10
- testType: AB
name: Wintersun - Sons of Winter and Stars
description: Select the most preferred option
options:
- Sons of Winter and Stars (lossless)
- Sons of Winter and Stars (64 kbps MP3)
- Sons of Winter and Stars (32 kbps MP3)
repeat: 10
email: youremail@gmail.com
You can start by copying this file, changing some fields, saving it to your Dropbox and getting the share link to test.
If you're just looking for a super simple starting point, then maybe this one will serve you better. This is the minimal YAML file you can have. Only one test with two different audio samples.
name: Minimal listening test
welcome:
description: |-
### Minimal listening test
options:
- name: lossless
audioUrl: https://www.dropbox.com/s/9e92quf9tr7aj8s/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%20lossless.wav?dl=0
- name: 64 kbps
audioUrl: https://www.dropbox.com/s/epvit0keu9w7emp/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%2064%20kbps.mp3?dl=0
tests:
- name: Lossless vs 64 kbps
testType: ABX
options:
- lossless
- 64 kbps
name
specifies the name of the listening test. This is only to make it easier for you to tell which results come
from which listening tests.
welcome
details the welcome screen which is shown to the participants before the actual listening tests are launched.
description
can be used for giving instructions to the participants. The
description uses Markdown syntax. Note the strange |-
after the
description:
, this tells YAML that the following is a multi-line text and without this the line breaks will be
replaced with spaces.
form
is optional and can be used as a survey for collecting demographics information from the participants. You should
avoid collecting personally identifiable information. Form can have any number of input fields, these are listed with the
bullet point syntax. Each input field needs to know a name and an inputType
. Supported types are text
, number
and
select
(dropdown). List the set of available options for select
type input with the same bullet point syntax.
results
section only has an optional description
field, which is similar to the welcome screen description. The
results are shown after all of the tests have been completed. The description is displayed at the top of the results
screen.
options
list the audio samples used in all of the listening tests. First you declare the options in this section
and then in the tests sections you can simply reference them with their names. Each option has the name as the identifier
and can to have two additional fields: audioUrl
for specifying where the audio clip is hosted and tag
for grouping
options together for statistical analysis. tag
is optional and if you omit it, there won't be aggregated statistics
in the results. Keep in mind to only group together audio clips which should be analyzed together. In this example all
of the 32 kpbs MP3 clips are grouped together, as are 64 kbps and lossless clips. The results include aggregated
statistics for each combination of tags that appeared together in the tests. If you have one test with samples with tags
32 kbps
and lossless
and another with tags 64 kbps
and lossless
, these two tests won't have shared aggregated
statistics because they had different combinations of tags.
tests
section lists the actual listening tests. There can be any number of tests but in practice this should be quite
limited to avoid exhausting the participants. Each test has four different fields:
testType
tells if the listening test is AB or ABX.name
is the title of the test and is displayed to the user at the top of each test screen and in the results page.description
of the test is a single piece of text shown to the user below the title. Markdown is not supported here.options
list the audio clip names. The clips will be randomly shuffled in each test iteration.repeat
tells how many iterations of the test should be done. Optional, defaults to 10 iterations.email
specifies the email address where the results should be sent to. This is optional and can be omitted if you
don't wish to collect the results. The results will be provided as JSON documents.
ABX only plays the audio samples defined in the configuration file. There is no processing (apart from volume slider) so all the effects under test need to be applied to the audio samples themselves.
All audio samples MUST be exactly the same length, down to a sample, because the samples are playing in loop and if the samples are of different lengths, they will get out of sync. All audio samples also MUST have the same sample rate. This is a limitation of the current technical implementation and might change in the future (or might not).
The audio samples should be looping friendly, meaning that if the sample starts and end abruptly, the users will be greeted with a loud pop sound whenever the audio loops back to the beginning. Applying fade-in in the beginning of each sample and fade-out at the end is a good way to avoid this.
Read on if you're interested in deploying and hosting ABX yourself.
Easiest way to deploy ABX is with Docker. The entire application is configured with environment variables. Here's an example of how to the deployment works for the official abxtests.com domain:
docker run -d --restart on-failure --name abx \
-p 80:80 -p 443:443 \
-v /abx:/abx:ro \
-v /etc/letsencrypt/live/abxtests.com:/etc/letsencrypt/live/abxtests.com:ro \
-v /etc/letsencrypt/archive/abxtests.com:/etc/letsencrypt/archive/abxtests.com:ro \
-e NODE_ENV=production \
-e HTTP_PORT=80 \
-e HTTPS_PORT=443 \
-e SSL_CERT_PATH=/etc/letsencrypt/live/abxtests.com/fullchain.pem \
-e SSL_PRIVATE_KEY_PATH=/etc/letsencrypt/live/abxtests.com/privkey.pem \
-e EMAIL_FROM_ADDRESS=noreply@abxtests.com \
-e EMAIL_FROM_NAME="ABX" \
-e MAILJET_API_KEY_PUBLIC=/abx/mailjet_api_key_public \
-e MAILJET_API_KEY_PRIVATE=/abx/mailjet_api_key_private \
jaakkopasanen/abx:latest
-d
runs the container in detached mode
---restart on-failure
makes the Docker engine restart the container if it
crashes--name abx
sets the container name-p 80:80 -p 443:443
exposes ports 80 (for HTTPS) and 443 (for HTTPS)-v /abx:/abx:ro
binds directory /abx
on the host to the container in the same location. This is where the API
keys for Mailjet reside.-v /etc/letsencrypt/live/abxtests.com:/etc/letsencrypt/live/abxtests.com:ro
and
-v /etc/letsencrypt/archive/abxtests.com:/etc/letsencrypt/archive/abxtests.com:ro
bind letsencrypt certificate
directories to the container. These are needed for HTTPS.-e NODE_ENV=production
tells NodeJS to run in production mode-e HTTP_PORT=80
tells the HTTP server to listen port 80-e HTTPS_PORT=443
tells the HTTPS server to listen port 443-e SSL_CERT_PATH=/etc/letsencrypt/live/abxtests.com/fullchain.pem
tells the HTTPS server where to find SSL
certificate public key-e SSL_PRIVATE_KEY_PATH=/etc/letsencrypt/live/abxtests.com/privkey.pem
tells the HTTPS server where to find the SSL
certificate private key-e EMAIL_FROM_ADDRESS=noreply@abxtests.com
tells the emailer which from address should be set to the emails-e EMAIL_FROM_NAME="ABX"
tells the emailer which name should be set to the emails-e MAILJET_API_KEY_PUBLIC=/abx/mailjet_api_key_public
tells the emailer where to find the Mailjet public API key-e MAILJET_API_KEY_PRIVATE=/abx/mailjet_api_key_private
tells the emailer where to find the Mailjet private API keyjaakkopasanen/abx:latest
this specifies the Docker image. Here we are using the latest version but there are also
tags available with semantic versioning (such as 1.1.0).Build your own Docker image if you don't want to use one from jaakkopasanen. Docker build includes production build of the React app and all other necessary steps.
docker build -t yourdockerhubusername/abx:latest .
docker push yourdockerhubusername/abx:latest
Read on if you're interested in developing ABX.
npm run start
Runs the app in the development mode. Open http://localhost:3000 to view it in the browser.
The page will reload if you make edits. You will also see any lint errors in the console.
npm run dev
Launches the backend server in development mode. This too has automatic reloads.
npm run test
Launches the test runner in the interactive watch mode.
Production build is done in the Docker build but if you'd like to create production build manually outside of Docker, run:
npm run build
This builds the app for production to the build
folder. It correctly bundles React in production mode and optimizes
the build for the best performance.
The build is minified and the filenames include the hashes. ABX is ready to be deployed!