How does FRBSTATS work?

[astrogewgaw]

I had a few questions about how **FRBSTATS** works. I am not very familiar with web design in any way, which is why I have absolutely no clue about how the site works or (more importantly) how it is deployed. Could you walk me through that? I think it would be nice to have those details in this issue for others to have a look as well, especially if they are thinking of contributing.

[0xCoto]

Sure, so let's begin with the Catalogue page, which is where all FRB data is displayed.

Catalogue

There is a spreadsheet where one (with edit permissions) may modify FRB parameters, remove and add new events. Once a modification is made, it's time to sync it to the FRBSTATS Catalogue.

To do that, I've set up a GitHub Action, which sends a GET request to https://www.herta-experiment.org/gitpull.php.

This is the content of gitpull.php:

<?php
// Plot or not plot - User input: /gitpull.php?plot={1,2}
$plot = $_GET['plot'] ?? 0;

if ($plot == "1" || $plot == 1) {
    $plot = True;
} else {
    $plot = False;
}

exec("cd /home/herta-experiment/public_html/frbstats && git reset --hard HEAD && git pull");
exec("cd /home/herta-experiment/public_html/frbstats && python3 fetch_catalogue.py");
exec("cd /home/herta-experiment/public_html/frbstats && python3 parser.py");
exec("cd /home/herta-experiment/public_html/frbstats && python main.py");
exec("cd /home/herta-experiment/public_html/frbstats && python repeater_parser.py");

if ($plot) {
    // DM
    echo "Plotting DM set...\n";
    exec("cd frbstats/figs && python3 dm_fluence.py");
    exec("cd frbstats/figs && python3 dm_flux.py");
    exec("cd frbstats/figs && python3 dm_frequency.py");
    exec("cd frbstats/figs && python3 dm_width.py");
    exec("cd frbstats/figs && python3 dm_long.py");
    exec("cd frbstats/figs && python3 dm_lat.py");

    // Fluence
    echo "Plotting Fluence set...\n";
    exec("cd frbstats/figs && python3 fluence_dm.py");
    exec("cd frbstats/figs && python3 fluence_flux.py");
    exec("cd frbstats/figs && python3 fluence_frequency.py");
    exec("cd frbstats/figs && python3 fluence_width.py");
    exec("cd frbstats/figs && python3 fluence_long.py");
    exec("cd frbstats/figs && python3 fluence_lat.py");

    // Flux
    echo "Plotting Flux set...\n";
    exec("cd frbstats/figs && python3 flux_dm.py");
    exec("cd frbstats/figs && python3 flux_fluence.py");
    exec("cd frbstats/figs && python3 flux_frequency.py");
    exec("cd frbstats/figs && python3 flux_width.py");
    exec("cd frbstats/figs && python3 flux_long.py");
    exec("cd frbstats/figs && python3 flux_lat.py");

    // Frequency
    echo "Plotting Frequency set...\n";
    exec("cd frbstats/figs && python3 frequency_dm.py");
    exec("cd frbstats/figs && python3 frequency_fluence.py");
    exec("cd frbstats/figs && python3 frequency_flux.py");
    exec("cd frbstats/figs && python3 frequency_width.py");
    exec("cd frbstats/figs && python3 frequency_long.py");
    exec("cd frbstats/figs && python3 frequency_lat.py");

    // Width
    echo "Plotting Width set...\n";
    exec("cd frbstats/figs && python3 width_dm.py");
    exec("cd frbstats/figs && python3 width_fluence.py");
    exec("cd frbstats/figs && python3 width_flux.py");
    exec("cd frbstats/figs && python3 width_frequency.py");
    exec("cd frbstats/figs && python3 width_long.py");
    exec("cd frbstats/figs && python3 width_lat.py");

    // Long
    echo "Plotting Gal. Longitude set...\n";
    exec("cd frbstats/figs && python3 long_dm.py");
    exec("cd frbstats/figs && python3 long_fluence.py");
    exec("cd frbstats/figs && python3 long_flux.py");
    exec("cd frbstats/figs && python3 long_frequency.py");
    exec("cd frbstats/figs && python3 long_width.py");

    // Lat
    echo "Plotting Gal. Latitude set...\n";
    exec("cd frbstats/figs && python3 lat_dm.py");
    exec("cd frbstats/figs && python3 lat_fluence.py");
    exec("cd frbstats/figs && python3 lat_flux.py");
    exec("cd frbstats/figs && python3 lat_frequency.py");
    exec("cd frbstats/figs && python3 lat_width.py");

    // Repeaters
    echo "Iterating Repeaters...\n";
    exec("cd frbstats/figs/repeaters && python3 iterate.py");
}

echo "OK\n";
?>

I won't get into a ton of technical details, but essentially what gitpull.php does is it fetches the latest commits from the repository so everything is up-to-date and in sync with whatever's up on the repository. fetch_catalogue.py downloads the spreadsheet as a CSV and parser.py transforms the CSV file to a JSON format. main.py initiates and loads the database (MongoDB).

The rest of the gitpull.php handles the repeaters page and the plots. Before we go there, let's look at the main page (index).

Index

The Events Observed, FRB Repeaters and Event Coveragae are all computed using JavaScript (client-side).

The code for the Event Count and FRB Class plots is found here - they're both updated dynamically (just like the aforementioned statistics, they are directly computed using catalogue.csv and repeaters.json).

Repeaters

The repeaters.json file is currently manually modified, it associates the repeater parents with children bursts. I'm hoping to implement a clustering algorithm soon, so that this is also fully dynamic and automatically up-to-date (#33).

I've added a JS function so that clicking on an FRB repeater, it displays the time series of each burst (partially resolving #27).

Plots

This page lets you select two parameters, and when you click on "Plot Parameters", it fetches parameterY_parameterX.svg. All these figures are found here. They are generated from the corresponding .py files, which are executed by gitpull.php (e.g. every time the catalogue changes etc.).

API

The API is pretty self-explanatory with the listed documentation on the page. Source code for the API that allows the user to query FRBs is written in PHP and can be found here.

---

Hope that helps a bit, let me know if there's any clarification you'd like me to expand on!

[astrogewgaw]

@0xCoto Thanks for the explanation 😁 ! It has cleared a lot of things up for me. I will let you know if there are any other doubts that I have 👍🏾 . Leaving the issue open for now. Maybe at some point this can be re-purposed and added to the contributing guidelines 🤔 ?

[0xCoto]

Yeah, we could probably add some elaboration in contributors.md or somewhere in the documentation (to be expanded in the future?) regarding the structure of how everything works etc.

[0xCoto]

Inactivity; closing for now.