Mock data UI

[peterdudfield]

It would be great if we could mock the data for the UI. This will make testing and development easier.

• You can sign up to app.quartz.solar and create an account
• run the UI and save API calls to json
• create mocking option that loads json instead

[Jacqueline-J]

Hi Peter,

If I understand correctly, the goal is to modify this

import requests
url = "https://api.quartz.solar/v0/solar/GB/national/forecast?historic=true&"
r = requests.get(url=url,headers={"Authorization": "Bearer "+access_token})

data = r.json()

to also accommodate mock data? I've created two options for doing that.

1. Separate Function: I've added a function load_mock_data specifically for loading mock data from JSON files. Here's how it works:

import json
import requests

# load mock data
def load_mock_data(file_path=None):
    if not file_path:
        raise ValueError("A string representing the file path for mock data must be provided.")
    with open(file_path, 'r') as file:
        return json.load(file)

with example usage like this

# Example usage to load data from JSON file
mock_data_path = PATH_TO_DATA
mock_data = load_mock_data(file_path=mock_data_path)

print(mock_data)

2. Combined Function: Alternatively, the data could be loaded with a single function fetch_data that can handle both live API calls and loading mock data from JSON files. Here's the code:

import json
import requests

# Python example to get data from the API or Load Mock data.
def fetch_data(mock=False, file_path=None):
    if mock:
        # Load data from JSON file
        if not file_path:
          raise ValueError("A string representing the file path for mock data must be provided.")
        with open(file_path, 'r') as file:
          return json.load(file)
    else:
      # load data from API
        url = "https://api.quartz.solar/v0/solar/GB/national/forecast?historic=true&"
        r = requests.get(url=url, headers={"Authorization": "Bearer " + access_token})

        return r.json()

with example usage

# Example usage
mock_data_path = PATH_TO_DATA

# Set mock=True to load data from JSON file
data = fetch_data(mock=True, file_path=mock_data_path)
print(data)

For the mock data, I extracted these data from the delta and PV forecast tabs of the Quartz Solar app, saving it in JSON format. To ensure a comprehensive dataset, I randomly selected three sites for each of these tabs. Let me know if you need more mock data particularly for the 4-hour forecast or dashboard modes. Mine currently came from the default settings.

Just wanted to quickly share it here, but 'll be opening a pull request shortly to allow for a more thorough review of these changes.

[peterdudfield]

This is absolutely right, however this repo is in React, so it would be translating this to React and javascript. Can you do that?

[Jacqueline-J]

I'm not too familiar with React but I gave it a try and I've come up with this. It's a script consists of three main parts:

1.Data Fetching and Handling: Which imports two functions, loadMockData and fetchDataFromAPI, which are responsible for fetching data, either from a local mock file or from an external API. The user would need to update the file path to the mock data here and then set “useMockData” to true or false depending on the usage.

import loadMockData from './loadMockData.mjs'; // Import the loadMockData function from loadMockData.mjs
import fetchDataFromAPI from './fetchDataFromAPI.mjs'; // Import the fetchDataFromAPI function from fetchDataFromAPI.mjs

async function fetchData(filePath, useMockData) { // Accept filePath and useMockData as arguments
    if (!filePath) {
        throw new Error("A string representing the file path for data must be provided.");
    }

    if (useMockData) {
        console.log('Using mock data');
        try {
            const data = await loadMockData(filePath);
            console.log('Mock Data:', data); // Log the data
            return data;
        } catch (error) {
            console.error('Error fetching mock data:', error);
            throw new Error(`Error fetching mock data: ${error.message}`);
        }
    } else {
        console.log('Using data from API');
        try {
            const data = await fetchDataFromAPI(); // Call the fetchDataFromAPI function to fetch data from API
            console.log('API Data:', data); // Log the data
            return data;
        } catch (error) {
            console.error('Error fetching data from API:', error);
            throw new Error(`Error fetching data from API: ${error.message}`);
        }
    }
}

const filePath = "path"; // Define file path here
const useMockData = true; // Set to true to use mock data, false to fetch data from API

fetchData(filePath, useMockData); // Call the fetchData function with the file path and useMockData flag

3.fetchDataFromAPI function: sends a POST request to obtain an access token for authentication, then uses this token to fetch data from the API. So here the user will need to update the log in requirements to generate a token. This was based on the approach outlined here: https://openclimatefix.notion.site/API-Access-2d8d2f64215d4432be830cbcc9220012

async function fetchDataFromAPI() {
    // Authentication
    const client_id = "QLMdXCCHMS9Zl5W2tDG3otpdcY7GeEnJ";
    const username = "email"; //username
    const pwd = "password"; // password
    const domain = "nowcasting-pro.eu.auth0.com";
    const grant_type = "password";

    const authUrl = `https://${domain}/oauth/token`;
    const authHeader = { 'content-type': 'application/json' };
    const authData = JSON.stringify({
      "client_id": client_id,
      "username": username,
      "password": pwd,
      "grant_type": grant_type,
      "audience": "https://api.nowcasting.io/"
    });

    try {
      const authResponse = await fetch(authUrl, {
        method: 'POST',
        headers: authHeader,
        body: authData
      });
      const authDataJson = await authResponse.json();
      const accessToken = authDataJson.access_token;

      // Fetch data from API
      const apiUrl = "https://api.quartz.solar/v0/solar/GB/national/forecast?historic=true&";
      const apiHeader = {
        'Authorization': 'Bearer ' + accessToken
      };
      const apiResponse = await fetch(apiUrl, { headers: apiHeader });
      const apiData = await apiResponse.json();

      return apiData;
    } catch (error) {
      console.error('Error:', error);
      throw error;
    }
  }

  export default fetchDataFromAPI;

  async function run() {
    try {
      const data = await fetchDataFromAPI();
      console.log(data);
    } catch (error) {
      console.error('Error:', error);
    }
  }

  run();

4.loadMockData function: It parses the JSON data from the file and returns it

import { readFile } from 'fs/promises'; // Import the readFile function from fs/promises

/// load mock data

async function loadMockData(filePath) {
    if (!filePath) {
        throw new Error("A string representing the file path for mock data must be provided.");
    }
    try {
        const data = await readFile(filePath, 'utf-8'); // Read the file asynchronously
        return JSON.parse(data); // Parse the JSON data
    } catch (error) {
        throw new Error(`Error loading mock data from file: ${error.message}`); // Provide a more specific error message
    }
}

export default loadMockData; // Export the loadMockData function

[Jacqueline-J]

I can create a pull request for the files, should I put them in quartz-frontend/apps/ ?

Also, this is designed as a stand-alone application, is that ok or did you want something that can integrate directly with the existing quartz-frontend project?

[peterdudfield]

Hi @Jacqueline-J thanks for this work.

I would definetly trying to put them in the apps. I would perhaps start with nowcasting-app as this would be most useful there.

@braddf do you have any advise for this?