caching the data onto a json file and reading from there

[yemoski]

read all the data from the fpl api daily and write it into an excel sheet

[yemoski]

read from the excel sheet instead of from the api

[yemoski]

To reduce the frequency of API calls in your app and minimize costs or load on the API server, you can implement a caching mechanism. This involves making a single API call, storing the retrieved data, and then serving subsequent requests from this cache until it's time to refresh the data. Here’s how you can approach this:

1. Determine Data Freshness Needs First, decide how fresh the data needs to be. If once per day suffices, your cache can live for 24 hours. However, you might need different intervals for different types of data.

2. Choose a Storage Solution For temporary storage, you can choose between in-memory caching or a more persistent solution:

In-memory caching: Quick and easy, suitable if your app won’t restart frequently, and you don't need the data after a restart. Tools like Redis, Memcached, or even local variables could be used depending on your setup. Persistent storage: If you need the data to persist across app restarts or require more structured access to cached data, consider using a database or file system. Databases like MongoDB, PostgreSQL, or even a simple JSON or XML file could work based on your data complexity and volume.

3. Implement Caching Logic Initial API Call: Set up your application to make the API call when the cache is empty or expired. Save the Data: Store the retrieved data in your chosen storage solution with a timestamp. Set Expiry: For in-memory caches like Redis, you can set an expiry time directly. For databases or files, you’ll manage expiry manually by checking timestamps.
4. Serve Data from Cache Modify your application logic to:

Check if the cached data is available and not expired. Serve the data from the cache if available. If not, fetch from the API, update the cache, and serve the new data.

5. Scheduled Refresh (Optional) Instead of relying solely on data requests to refresh your cache, you can set up a scheduled task (using cron jobs in Linux, Scheduled Tasks in Windows, or cloud function triggers) to refresh the cache at a specific time each day.

6. Handling Failures Implement error handling for cases where the API is unavailable:

Retry mechanisms. Serving old cached data (with a warning) if newer data can’t be fetched. Notifications or logging for failed refresh attempts.

7. Security and Compliance Ensure that any cached data is secured appropriately, especially if personal or sensitive data is involved. Comply with any data storage and privacy regulations applicable to your use case.

8. Monitor and Optimize Finally, monitor the performance of your caching solution. Adjust the cache size, expiry times, and storage method based on actual app usage and performance metrics.

This approach will significantly reduce the number of API calls, leading to cost savings, improved performance, and a more scalable application.