Create api_data_retrieval

[Jacqueline-J]

Pull Request

This pull request addresses issue #3.

Description

I have added the "api_data_retrieval" script to the scripts folder. This code automatically generates Python functions for each endpoint of the Elexon Insights Solution API.

Overall, the code appears to be working well, although I have one more test that I would like to preform. But I have two issues that I'd like your input on.

1.) I tried to make the output for each endpoint as clean as possible, so the process of converting the JSON data to a DataFrame is handled with a function called json_to_df, which is defined at the beginning of the code. I'm not sure if this would be considered best practice or if it would be better to include it directly in the generate_function_from_openapi function.

2.) I'm unsure about the note_on_date_availability = 'Note: as of 2024-05-26, data only available from 2024-01-31 onwards' (see line 54). I took this line from the code you provided, but I'm not sure if it applies to one specific endpoint or all of them.

How Has This Been Tested?

I tested the script by randomly selecting 20 endpoints. I was able to draw data from all except three:

1. get_balancing_nonbm_stor_from_from_to_to
2. get_datasets_mdv_from_from_to_to
3. get_lolpdrm_forecast_evolution

In regards to the get_balancing_nonbm_stor_from_from_to_to and the get_datasets_mdv_from_from_to_to endpoints. When I run the code no data is returned, it might be a problem with the parameters I am passing, but I'm not sure.

Additionally, the endpoint get_lolpdrm_forecast_evolution also didn't work, but this endpoint has been moved as per their documentation.

As for the next steps, I am planning to test the data drawn from the API against data drawn manually. I should have this done in the next couple of days, but I wanted to share what I have just incase there are any changed I need to make.

Checklist:

• [ ] My code follows OCF's coding style guidelines
• [ ] I have performed a self-review of my own code
• [ ] I have checked my code and corrected any misspellings

[peterdudfield]

Thanks @Jacqueline-J this is amazing. Thank you so much for doing this.

1. json_to_df I would probably define this function once, and use it rather than have new bits of code in each call
2. I'm not sure either, i got that date just from one end point

[Jacqueline-J]

Hi Peter,

I've made some changes to the json_to_df code to allow users to retrieve metadata as well. Now, the function returns both the metadata and the data. Some endpoints provide metadata while others do not. In cases where metadata is not available, the function will return "no metadata provided."

I also removed the json_to_df code block from the main script and added it as a separate file in the scripts folder.

Additionally, I removed the line regarding date availability, as I couldn't find documentation confirming date availability for all endpoints.

Regarding the tests, everything looks good overall(I've added the jupyternotebook to a new test folder).

However, some URLs linking to the actual API work while others do not. I can't identify a pattern in the URL naming conventions, some need to have the parameters included in the URL some don't (it seems to be in cases where parameters are requiered), so I am unsure how to correct this aside from doing it manually.

[peterdudfield]

Hi Peter,

I've made some changes to the json_to_df code to allow users to retrieve metadata as well. Now, the function returns both the metadata and the data. Some endpoints provide metadata while others do not. In cases where metadata is not available, the function will return "no metadata provided."

I also removed the json_to_df code block from the main script and added it as a separate file in the scripts folder.

Additionally, I removed the line regarding date availability, as I couldn't find documentation confirming date availability for all endpoints.

Regarding the tests, everything looks good overall(I've added the jupyternotebook to a new test folder).

However, some URLs linking to the actual API work while others do not. I can't identify a pattern in the URL naming conventions, some need to have the parameters included in the URL some don't (it seems to be in cases where parameters are requiered), so I am unsure how to correct this aside from doing it manually.

Thank you so much for doing this. Yea I can understand that perhaps teh required ones go into the url, and the optional ones are extra.

I would suggest we merge this now. And we can then start using it to generate some functions, and see how it goes. Totally up to you how much you want to be involved. @14Richa should start work on this soonish

[Jacqueline-J]

ok, sounds good. I’m available and willing to assist with the project as needed

[peterdudfield]

@all-contributors please add @Jacqueline-J for code

[allcontributors[bot]]

@peterdudfield

I've put up a pull request to add @Jacqueline-J! :tada: