Potential GitHub Actions to add to repo

[JessyBarrette]

In an ideal world it would be good to implement an action to this repo that can be run on push to main which does the following tests:

• station.csv exists
  • latitude/longitude are within +-90/+-180
  • commissioned_time/decommissioned_time have a proper timestamp format
• the data subdirectories respects the convention: yyyy-mm-dd.*
• each data subdirectories contains a survey.csv file
• for each survey.csv files:
  • collection_time timestamp use the right format
  • station is matching an existing value within the station.csv file

Create an issue for each failed checks.

[timvdstap]

So proposed actions / checks, to run when merging survey-date-branch into main:

• [ ] Check whether the stations.csv file exists in the root of the repository
• [ ] Within stations.csv file, make sure that latitude and longitude are within +- 90/+-180
• [x] Each survey-date folder has three subfolders (Instruments, DR1900 and Aquafluor)
• [x] Each subdirectory has the appropriate file formats (i.e. an .xlsx file in the survey-date folder fails a check).
• [x] The data subdirectories respects the convention YYYY-MM-DD_* (e.g., 2024-09-24_survey_date)
• [x] Each data subdirectory contains a YYYY-MM-DD_survey_raw.jpeg and YYYY-MM-DD_survey_final.csv file.
• [x] Each instrument subdirectory (Instruments, DR1900, Aquafluor) has a _final.csv file.
• [ ] Each station_id in the _survey_final.csv file has an associated entry in the stations.csv file.
• [ ] Each Sampling Event Number in a _final.csv file has an associated Sampling Event Number in the _survey_final.csv file.
• [ ] Each survey_date in the _survey_final.csv file is in the correct format (YYYY-MM-DD)
• [ ] Within the Instrument, Nutrient and Aquafluor _final.csv data files, the Analysis Date column is in the correct format (YYYY-MM-DD).
• [ ] Each failed check creates an issue with me automatically assigned.

[timvdstap]

Tagging the Science Team to check whether there's any checks you would like to see implemented, or prioritize the checks listed above: @jdelbel @hakaidrew @CarrieWeekes @naomiboon7

[fostermh]

A minimal example to check that a date is in the correct format and survey station exists in stations.csv The global variables are poor coding practice.

import pandas as pd
import json
import unittest

# read in csv as pandas dataframes
survey_final_df = pd.read_csv(
    './data/2024-09-16_example_dataset/2024-09-16_survey_final.csv', sep=',')
stations_df = pd.read_csv('./stations.csv', sep=',')

class TestSurveyFinal(unittest.TestCase):
    def test_survey_date(self):
        # iterate over the rows and assert that the survey date is in the correct format. if not, throw exception.
        for index, row in survey_final_df.iterrows():
            self.assertRegex(
                row['Survey Date'],
                r'^\\d{4}\\-(0?[1-9]|1[012])\\-(0?[1-9]|[12][0-9]|3[01])$',
                f'Survey Date of {row['Survey Date']
                                  } does not match YYYY-MM-DD format'
            )
    def test_survey_stations(self):
        # find all survey stations that do not have a match in stations.csv
        joined_df = survey_final_df.merge(stations_df, 'left',
                              left_on='Station', right_on='station_id')
        missing_stations = list(
            set(
                joined_df[joined_df['station_id'].isnull()].Station
            )
        )
        count = len(missing_stations)
        self.assertEqual(count, 0, f"Survey stations missing from stations.csv: {', '.join(missing_stations)}")

unittest.main()