Create a RAG summary of the results of the scraping exercise

[Daniellappv]

Description: Create a RAG summary of the results of the scraping exercise.

RAG to be determined:

• Dark green: We scrapped and have all the metadata, we can determine if DP, collection or source
• Green: We scrapped and have all the metadata, we can't technically determine if DP, collection or source
• Light Amber: We have scrapped and have partial metadata
• Dark Amber: We have the title
• Red: Not scraped

Can we automate in some way? [Prob not in this sprint]

Acceptance criteria

• [x] We have a RAG summary of the results of the scraping exercise

Task-list

• [x] Agree the RAG status levels
• [x] create RAG per domain
• [x] Create RAG per publisher

Link to Jira card: https://open-data-ed.atlassian.net/browse/OD-503

[nightsh]

ETA 6h

[nightsh]

Some pseudocode to work as a silly tech spec (created in a meeting, so not very complex):

def transform(name=None, input_file=None):

    datasets = []

    if not input_file:
        # loop over directory structure
        if name:
            # loop over <name> scraper output e..g nces
            # datasets = list of all <name> files
        else:
            # loop over everything
            # datasets = list of all JSON files
    else:
        # read file, which is a list of files
        # datasets = list of all JSON files in the list

    for dataset in datasets:
        score = compute_score(dataset)
        # now write all the scores and some ID data in a pandas df
        # if S3 is configured, upload them to the bucket

def compute_score(json_obj):
    json_obj_score = "something"
    return json_obj_score

[osahon-okungbowa]

Tasks list updated

[nightsh]

We have:

• A RAG transformer that loops over collected data and measures metadata quality
• A XLSX dump generated by the above
• A dashboard page showing the RAG summary based on the dumped file: http://159.65.218.60:8050/quality