Integrate solar radiation

[l-kotzur]

Follow up on the discussion in #62

@clampr @adriesse I took the liberty of making a new issue for this.

[clampr]

Hi @l-kotzur & @adriesse,

Let's continue this conversation here.

As mentioned by @l-kotzur, solar radiation data has already been added to some of the import routines. So Meteostat is presently collecting this data. However, it is not redistributed through our interfaces atm.

The reason is mainly a planned re-structuring of Meteostat's architecture which will put this Python library in the center of the architecture. This is going to be a completely new version which requires most code to be rewritten.

Before any work on the new version of the library can begin, I'd like to get feedback on the idea. Therefore, I will soon open a new issue which describes the concept. I will also need to talk to a few people to create a stronger foundation for the project - so it doesn't rely as much on myself as a maintainer.

[l-kotzur]

Hey @clampr

Thank you for the information!

This sounds interesting but is also a little bit frustrating to wait for this feature until the new architecture is finished. Would this be that much additional effort, especially when the data is already queried?

Otherwise, I am happy to support this package. So let me know when you make the proposal.

[clampr]

I already thought about this.

The problem is that the current architecture isn't very flexible when it comes to adding new variables, changing the structure or data format. Meteostat Python relies entirely on the data provided through the bulk data interface. This uses CSV files with a static structure. Adding the solar radiation parameter would require a new version of these bulk data files. And every new version of the bulk data requires a temporal parallel export to ensure older versions don't stop working immediately.

I would vote for investing all capacity in the new architecture. This will not only add solar radiation, but also other parameters like cloud cover, visibility, etc.

[l-kotzur]

But isn't the bulk data interface working with NaN/sparse values anyway? So, would the addition of new columns destroy old interfaces? I understand it though for the modification of existing parameter names...

I hope you don't misunderstand my persistence. I just want to use this awesome package. :)

[adriesse]

I would perhaps add a comment that adding these parameters would be very attractive, but additional meta data would make them more valuable. I know solar radiation data, and it is certainly not all created equal. Of course the sky is the limit with meta data, but to take a German metaphor that seems somewhat similar but is not: "there is some air overhead" w.r.t. metadata in meteostat.

(To be taken as encouragement rather than criticism!)

[clampr]

Yes, simply appending additional columns could be feasible @l-kotzur. I will put this on the backlog and investigate if we can add it.

Also, @adriesse is right regarding the meta data. Meteostat is currently losing some valuable meta data for the sake of simplicity. It's something we'll need to change. But this can only be done as part of the v2 rewrite.