PV ForecastModel - alternate data source question

[Layoric]

Hi All!

Great to see a PV focused library with forecasting getting a lot of traction! Thanks and well done!

I'm currently working with a company that provides Irradiance and PV data (both forecasts and historical) derived (primarily) from satellite imagery and query-able from a simple HTTP API. We've had interest to use our data with this library.

After a look through your documentation and source code for ForecastModel, I wanted to get some feedback for an approach for integration.

The simplest might be just to build a separate library that follows the ForecastModel base class methods and provides the expected DataFrame return from get_processed_data. The other option would be to allow a more extensible data source for ForecastModel beyond just a thredds based server.

Would really appreciate some feedback if this is a good idea or something that doesn't suit the goals of this library. Solcast provides free access for researchers globally and want to make it easy to use with the tools researchers are already familiar with. More info on our site.

Thanks again.

[wholmgren]

I'm opposed to this for a couple of reasons.

I don't want pvlib to favor some forecast providers over others.

Your http api looks simple enough to use that people can get the data they want with just a couple of calls using the requests library or urllib.

Your backend has already cleaned up the raw model data, so there's no need for any of the processing code in the forecast.py module.

So, I don't see how it makes sense from a technical standpoint either.

[cwhanse]

I can see advantages and disadvantages to the proposal. What we have now is a forecast module tied to the UCAR Thredds server. In the spirit of pvlib, I could see adding reference implementations for access to other forecast sources, much as pvlib provides options for clearsky models, PV electrical performance models, etc.

But doing so means pvlib assumes the burden to maintain the interface to each forecast source. A more general and flexible path might be to provide an example showing how to download from SolCast and interface to pvlib. That puts a little distance between pvlib and SolCast should the API need to change.

OrangeButton initiative will soon release v1 of their PV modeling taxonomy which includes a data model for forecasts. I anticipate that we'll hear about adding methods to read/write forecasts using this data model.

[Layoric]

Really appreciate the feedback @wholmgren and @cwhanse, I wasn't trying to bring this up as a 'please do this' but more of a 'what do you think' of either of the two options I presented.

1. External libs just try to copy ForecastModel function signatures for non Thredds/NetCDF servers; or
2. pvlib provides a base class decoupled from Thredds/NetCDF to allow for different data sources and formats allowing external libraries to create implementations compatible with pvlib users.

I think 2. in the long run would allow pvlib maintainers then make choice on a case by case which implementations are worth pulling into pvlib.

@wholmgren I agree with what you are saying, but I think the simplicity of the current design to be able to just create a GFS model and class get_processed_data is a powerful one, plumbing of data sources and data structures is time consuming, tedious and error prone. It would be nice to continue that simplicity by enabling other developers to create additional data sources for something like ForecastModel. As maintainers, you can then pick and choose which ones should be baked into pvlib guided by whatever values/rules you've decided.

@cwhanse My thought was to do exactly that. I'll provide what I think is compatible with the current ForecastModel with maybe an example of how pvlib might look at abstracting datasource/data format using Solcast API as a usecase.

I'll update this issue once I have something going. Thanks again!

[uvchik]

I agree that we shouldn't overload the pvlib.

We had the same problem with combining a locally limited weather data set with the pvlib. We decided to open a new project where we combine the API of the weather data with the pvlib. Using pip or conda the pvlib will be installed automatically while people install our package. It's quite simple for the users and we have the freedom to add our own stuff, plots etc. without annoying the pvlib community 😉 and overloading the package. Anyhow we contributed to the pvlib if we found bugs, typos or things missing that might rather belong to the pvlib than to our library.

@Layoric So, what about opening a solcast project at github and provide a solcast package that combines your API with the pvlib? As @cwhanse suggested you can start with an example.

If the pvlib wants to support such packages, it could provide links to these projects.

[Layoric]

Closing issue has resolved with the suggestion to create pv example using PV lib and our own API + pandas. Coming up with a nice comparison of PV forecasts is still ongoing as with various models to choose from with PV lib, still learning how to build the equivalent request.

Comparison of radiation data has been added to an work in progress repo we've created to show use of both PV lib and and Solcast API when getting DNI/DHI/GHI data.

Thanks again all for the valuable feedback and this library 👍

https://github.com/Solcast/howto-pandas/