openclimatefix / uk-pv-national-gsp-api

API for hosting nowcasting solar predictions
MIT License
6 stars 10 forks source link

Fake data #354

Open peterdudfield opened 3 months ago

peterdudfield commented 3 months ago

Detailed Description

It would be great to run the api with fake data. This means users could run this locally without having to connect to a database. The FE users could then use this API with fake data.

Context

Possible Implementation

VikramsDataScience commented 2 months ago

Hello there @peterdudfield. This definitely looks to be a more complex and interesting change. If possible I'd like to attempt a contribution to this change?

If so, could I just clarify the following points, please (apologies if these questions are a bit dumb/obvious!):

What is meant by the route (again, sorry if these are obvious/dumb questions to ask)?

Need to match return types for each route

  • Would the return types be in reference to matching the dtypes that the DummyDBPredictedPowerProduction() class returns found in the _models.py module? Might want to add some noise on the forecast, so the actual is exactly the same
  • Looking at your implementation in the _basicSolarPowerProductionFunc() function in the client.py this seems to address that problem? Unless you're looking to improve this function further? In which case I could potentially look to conduct some research on how that might be improved (for instance, Meta did quite a cool implementation of a fourier series for their Prophet algorithm to address seasonality that might prove useful)? I can't promise I'll be successful, but if you'd like, I can certainly give it a try! Please do let me know how you'd like me to proceed, and I'll do my best.
peterdudfield commented 2 months ago

Thanks @VikramsDataScience for getting involved.

  1. I was thinking just adding FAKE as a new environmental variable. Which can be turn on and off
  2. I would try to only modify this repo, to keep it simple
  3. So, I would add something like this in each route
if os.enviorn['FAKE'].lower() = 'true':
     return make_fake_data(....)
  1. in the api, there are an number of different routes or urls. For example here

  2. Id try to keep the same return objects we give already. Not creating new ones

  3. Interesting idea. I would first use this one, and we can always update it later.

Hope this helps, and please do ask more questions

peterdudfield commented 2 months ago

This repo and code is how I was thinking it could be done

VikramsDataScience commented 2 months ago

Apologies @peterdudfield! I've been a little time poor, and distracted lately. I erroneously created a PR (https://github.com/openclimatefix/india-api/pull/76) in the india-api repo to address this issue, but I think I made the changes to the incorrect repo! I've since closed it, because I think we're looking to make similar changes but to this repo? If so, which module should I be looking to modify in this repo?

peterdudfield commented 2 months ago

No problem. See point 4 above, but we should try to fake all the 'routes' of the api

VikramsDataScience commented 2 months ago

Hey @peterdudfield. I've made the changes to what I think are the correct modules, but I'm running python 3.11.5 on my local machine and venv, but it looks like the .pre-commit-config.yaml requires python 3.9? Is there any way around this, as I'm really not too keen on downgrading my existing python version?

I've also tried to install 3.9 separately to avoid downgrading my system version. And, from there creating a venv that's built from 3.9, but I'm having a variety of challenges!

peterdudfield commented 2 months ago

yea, i would stick to python 3.11.

Does it stop you submitting code?

VikramsDataScience commented 2 months ago

Yeah, exactly! It prevents the commit from going through. Are you okay with me modifying the default_language_version in the .pre-commit-config.yaml?

peterdudfield commented 2 months ago

yea, feel free too

VikramsDataScience commented 2 months ago

Cool. Thank you.

Now that I've got the changes made to the .pre-commit-config.yaml file, it looks like my changes are failing some of the pre-commit checks. It's a bit late here, but if its cool with you, I'll take a gander at resolving these issues over tomorrow/day after, and report back if I'm having any additional challenges?

Otherwise, if all is good, I'll create a PR, and we can work together to get it right :).

peterdudfield commented 2 months ago

yea, of course, Thanks so much for helping out on this

VikramsDataScience commented 2 months ago

Sorry @peterdudfield about all the commits! That was me having fun and playing around with the pre-commit library 😂

VikramsDataScience commented 2 months ago

Hey @peterdudfield. I've created the PR. It was initially failing the CI checks, but I've modified the .pre-commit-config.yaml again to use a more flexible python3 build over the python3.11.5. It looks like using the 3.11.5 was too narrow a declaration in the CI workflow, and was raising errors when trying to build the environment.

I also quite foolishly left the modified relative imports that I used for testing in my push, not realising that the dockerfile already sets the PYTHONPATH.

The CI tests seem to be passing now. When you've got some time, could you review the changes in the PR, and let me know if I'm on the right track, please?