PyHa

A tool designed to convert audio-based "weak" labels to "strong" moment-to-moment labels. Provides a pipeline to compare automated moment-to-moment labels to human labels. Current proof of concept work being fulfilled on Bird Audio clips using Microfaune predictions.

This package is being developed and maintained by the Engineers for Exploration Acoustic Species Identification Team in collaboration with the San Diego Zoo Wildlife Alliance.

PyHa = Python + Piha (referring to a bird species of our interest known as the screaming-piha)

Contents

• Installation and Setup
• Functions
• Examples

Installation and Setup

1. Navigate to a desired folder and clone the repository onto your local machine. git clone https://github.com/UCSD-E4E/PyHa.git

• If you wish to reduce the size of the repository on your local machine you can alternatively use git clone https://github.com/UCSD-E4E/PyHa.git --depth 1 which will only install the most up-to-date version of the repo without its history.

2. Install Python 3.8, Python 3.9, or Python 3.10
3. Create a venv by running python3.x -m venv .venv where python3.x is the appropriate python.
4. Activate the venv with the following commands:

• Windows: .venv\Scripts\activate
• macOS/Linux: source .venv/bin/activate

5. Install the build tools: python -m pip install --upgrade pip poetry
6. Install the environment: poetry install
7. Here you can download the Xeno-canto Screaming Piha test set used in our demos: https://drive.google.com/drive/u/0/folders/1lIweB8rF9JZhu6imkuTg_No0i04ClDh1
8. Run jupyter notebook while in the proper folder to activate the PyHa_Tutorial.ipynb notebook and make sure PyHa is running properly. Make sure the paths are properly aligned to the TEST folder in the notebook as well as in the ScreamingPiha_Manual_Labels.csv file

Functions

This image shows the design of the automated audio labeling system.

isolation_parameters

Many of the functions take in the isolation_parameters argument, and as such it will be defined globally here.

The isolation_parameters dictionary definition depends on the model used. The currently supported models are BirdNET-Lite, Microfaune, and TweetyNET.

The BirdNET-Lite isolation_parameters dictionary is as follows:

isolation_parameters = {
    "model" : "birdnet",
    "output_path" : "",
    "lat" : 0.0,
    "lon" : 0.0,
    "week" : 0,
    "overlap" : 0.0,
    "sensitivity" : 0.0,
    "min_conf" : 0.0,
    "custom_list" : "",
    "filetype" : "",
    "num_predictions" : 0,
    "write_to_csv" : False,
    "verbose" : True
}

The Microfaune isolation_parameters dictionary is as follows:

isolation_parameters = {
    "model" : "microfaune",
    "technique" : "",
    "threshold_type" : "",
    "threshold_const" : 0.0,
    "threshold_min" : 0.0,
    "window_size" : 0.0,
    "chunk_size" : 0.0,
    "verbose" : True
}

The technique parameter can be: Simple, Stack, Steinberg, and Chunk. This input must be a string in all lowercase.
The threshold_type parameter can be: median, mean, average, standard deviation, or pure. This input must be a string in all lowercase.

The remaining parameters are floats representing their respective values.

The TweetyNET isolation_parameters dictionary is as follows:

isolation_parameters = {
    "model" : "tweetynet",
    "tweety_output": False,
    "technique" : "",
    "threshold_type" : "",
    "threshold_const" : 0.0,
    "threshold_min" : 0.0,
    "window_size" : 0.0,
    "chunk_size" : 0.0,
    "verbose" : True
}

The tweety_output parameter sets whether to use TweetyNET's original output or isolation techniques. If set to False, TweetyNET will use the specified technique parameter.

The Foreground-Background Separation technique isolation_parameters is as follows:

isolation_parameters = {
   "model" : "fg_bg_dsp_sep",
   "technique" : "",
   "threshold_type" : "",
   "threshold_const" : 0.0,
   "kernel_size" : 4,
   "power_threshold" : 0.0,
   "threshold_min" : 0.0,
   "verbose" : True
}

The kernel_size parameter is an integer n that specifies the size of the kernel used in the morphological opening process. For the opening of the binary mask, this will be an n by n kernel. For the processing of the indicator vector, this will be a 1 by n kernel.
The power_threshold parameter is a float that determines by how many times the power of a pixel must be larger than its row and column medians. For example, if this value is set to 3.0, each pixel will have to have a power of at least 3 times its row and column medians to be included in the binary mask.

The Template Matching isolation_parameters is as follows:

isolation_parameters = {
   "model" : "template_matching",
   "template_path" : "",
   "technique" : "",
   "window_size" : 0.0,
   "threshold_type" : "",
   "threshold_const" : 0.0,
   "cutoff_freq_low" : 0,
   "cutoff_freq_high" : 0,
   "verbose" : True,
   "write_confidence" : True
}

The template_path parameter should be set to the path to the template to use, stored as a .wav file.
The window_size parameter should be a float corresponding to the length (in seconds) of the template. This is so the Steinberg isolation can correctly convert the local score array into labels.
cutoff_freq_low and cutoff_freq_high should be integer values. If both are defined, both signal and template will be put through a butterworth bandpass filter set to those cutoff frequencies. This is recommended to ensure that the signal and template are the same shape on the frequency axis.
write_confidence determines whether or not the confidence of each label is written to the array, determined by the max score in the local score array for each label.

All files in the birdnet_lite directory are from a modified version of the BirdNET Lite repository, and their associated documentation can be found there.

All files in the microfaune_package directory are from the microfaune repository, and their associated documentation can be found there.

All files in the tweetynet directory are from the tweetynet repository, and their associated documentation can be found there.

All files in the tweetynet directory are from the tweetynet repository, and their associated documentation can be found there.

Examples

These examples were created on an Ubuntu 16.04 machine. Results may vary between different Operating Systems and Tensorflow versions.

Examples using Microfaune were created using the following dictionary for isolation_parameters:

isolation_parameters = {
     "model" : "microfaune",
     "technique" : "steinberg",
     "threshold_type" : "median",
     "threshold_const" : 2.0,
     "threshold_min" : 0.0,
     "window_size" : 2.0,
     "chunk_size" : 5.0
 }

To generate automated labels and get manual labels:

automated_df = generate_automated_labels(path,isolation_parameters,normalize_local_scores=True)
manual_df = pd.read_csv("ScreamingPiha_Manual_Labels.csv")

Function that gathers statistics about the duration of labels

annotation_duration_statistics(automated_df)

annotation_duration_statistics(manual_df)

Function that converts annotations into 3 second chunks

annotation_chunker(automated_df, 3)

Helper function to convert to kaleidoscope-compatible format

kaleidoscope_conversion(manual_df)

Baseline Graph without any annotations

clip_path = "./TEST/ScreamingPiha2.wav"
spectrogram_visualization(clip_path)

Baseline Graph with log scale

spectrogram_visualization(clip_path,log_scale = True)

Baseline graph with normalized local score values between [0,1]

spectrogram_visualization(clip_path, normalize_local_scores = True)

Graph with Automated Labeling

spectrogram_visualization(clip_path,automated_df = True, isolation_parameters = isolation_parameters)

Graph with Human Labelling

spectrogram_visualization(clip_path, premade_annotations_df = manual_df[manual_df["IN FILE"] == "ScreamingPiha2.wav"],premade_annotations_label = "Piha Human Labels")

Graph with Both Automated and Human Labels

Legend:

- Orange ==> True Positive
- Red ==> False Negative
- Yellow ==> False Positive
- White ==> True Negative

spectrogram_visualization(clip_path,automated_df = True,isolation_parameters=isolation_parameters,premade_annotations_df = manual_df[manual_df["IN FILE"] == "ScreamingPiha2.wav"])

Another Visualization of True Positives, False Positives, False Negatives, and True Negatives

automated_piha_df = automated_df[automated_df["IN FILE"] == "ScreamingPiha2.wav"]
manual_piha_df = manual_df[manual_df["IN FILE"] == "ScreamingPiha2.wav"]
piha_stats = binary_visualization(automated_piha_df,manual_piha_df)

Function that generates statistics to gauge efficacy of automated labeling compared to human labels

statistics_df = automated_labeling_statistics(automated_df,manual_df,stats_type = "general")

Function that takes the statistical output of all of the clips and gets the equivalent global scores

global_dataset_statistics(statistics_df)

Function that takes in the manual and automated labels for a clip and outputs human label-by-label IoU Scores. Used to derive statistics that measure how well a system is isolating desired segments of audio clips

Intersection_over_Union_Matrix = clip_IoU(automated_piha_df,manual_piha_df)

Function that turns the IoU Matrix of a clip into true positive and false positives values, as well as computing the precision, recall, and F1 statistics

matrix_IoU_Scores(Intersection_over_Union_Matrix,manual_piha_df,0.5)

Wrapper function that takes matrix_IoU_Scores across multiple clips. Allows user to modify the threshold that determines whether or not a label is a true positive.

stats_df = automated_labeling_statistics(automated_df,manual_df,stats_type = "IoU",threshold = 0.5)

Function that takes the output of dataset_IoU Statistics and outputs a global count of true positives and false positives, as well as computing common metrics across the dataset

global_stats_df = global_statistics(stats_df)

All relevant audio from the PyHa tutorial can be found within the "TEST" folder. In order to replicate the results displayed in the GitHub repository, make sure the audio clips are located in a folder called "TEST" in the same directory path as we had in the Jupyter Notebook tutorial.

All audio clips can be found on xeno-canto.org under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) [https://creativecommons.org/licenses/by-nc-sa/4.0/ license](https://creativecommons.org/licenses/by-nc-sa/4.0/ license).

The manual labels provided for this dataset are automatically downloaded as a .csv when the repository is cloned.

Testing

Tests require E4E NAS credentials. These must be provided as a JSON file, or as an environment variable.

If provided as a JSON file, this file must be placed at ${workspaceFolder}/credentials.json, and have the following structure:

{
    "username": "e4e_nas_user",
    "password": "e4e_nas_password"
}

If provided as an environment variable, the variable must be named NAS_CREDS and must have the following structure:

{"username":"e4e_nas_user","password":"e4e_nas_password"}

Any account used must have read access to the following share:

• //e4e-nas.ucsd.edu/temp

Execute pytest as follows:

python -m pytest pyha_tests