search

poblabs / weewx-belchertown

A clean and modern weewx skin with real time streaming updates, forecast data and interactive charts. View it in action at BelchertownWeather.com
https://belchertownweather.com
GNU General Public License v3.0
209 stars 111 forks source link

Aggregate of Wind Direction is not possible #824

Closed jkumeboshi closed 1 year ago

jkumeboshi commented 2 years ago

Describe the bug Wind Direction graph looks very different from one generated by WU using exactly same raw data.

Link to your website (IMPORTANT)

WeeWX: https://meteo.fedibox.com/

WU PWS: https://www.wunderground.com/dashboard/pws/ISAPRI3

Version of the skin you're using 1.3b

Screenshots WeeWx chart: wind direction is much more "unstable" and anyway looks veri different from WU one: immagine

WU chart: wind direction is somewhat "stable" immagine

Device Information (please complete the following information):

Additional context the graphs.conf to generte wind direction graph. setting aggregate_type = None generates event worse charts.

Increasing aggregate_interval = 1800 doesn't changes things very much, WeeWX and WU charts still very different.

    [[chart22]]
        title = Wind Direction
        yAxis_min = 0
        gapsize = 600
        aggregate_type = avg
        aggregate_interval = 600
        [[[windDir]]]
            zIndex = 1
            yAxis = 1
            yAxis_max = 360
            lineWidth = 0
            [[[[marker]]]]
                enabled = true
            [[[[states]]]]
                [[[[[hover]]]]]
                        lineWidthPlus = 0
areax99 commented 2 years ago

I think you have a wrong synchronization to produce data in web page and WU. How often do you generate web pages?

jkumeboshi commented 2 years ago

@areax99 data is exactly the same, the PWS sends it to WU and WeeWX at same rate.

The problem was that with wind direction you shall disable ANY kind of data aggregation, because the measure is "circular" and 360° is same as 0°. Setting aggregate_type = avg as I did will produce charts that tends to place points in in the middle... To average wind direction you should make tak ein account that 359° is much near to 1° than to 350°. So avg for 359° and 1° should be 360°, not 180°...

jkumeboshi commented 2 years ago

From weewx issues, they suggest to use wind.vecdir instead of windDir.avg:

weewx/weewx#798

James-76 commented 2 years ago

I'm slightly confused by this by this bug report. My graph from Belchertown, WU and the season's skin all match up, but there again I'm using a aggregate_interval of 5 mins and a gapsize = 300000, which should be your archive_interval from weewx.conf multiplied by 1000 to get milliseconds. Standard is 5 minutes (300 seconds)

image

image

image

So a couple of questions; Are you using the right 'gapsize' for your archive_interval which should be in milliseconds so probably 60000, if your archive is set to 60 seconds. WU is producing a graph of every 5 mins, but I'm guessing you are uploading every minute?

Are you saying that if you if you want a aggregate_interval of 5 minutes it is not producing the right value of the underlying 5 archive intervals?

James

jkumeboshi commented 2 years ago

@James-76 gapsize is now in seconds, since commit 11135767cc76115687dbca4346cbe1386dae7a40

My archive interval is 60 seconds, while default gapsize is 300 and I use that for most charts.

Since on low wind days the wind direction is very very variable, I tried to aggregate it in 600 or 900 seconds, but actually I can't, because aggregating wind direction using avg mode results in inaccurate values, all converging to values between 125° and 260°. As I can see, this skin is pulling data directly from the database. Moreover, I saw that it use SQL queries to extract aggregate values directly. And SQL can't just do following calculations to do a correct average of angles, I suppose:


from math import *
from statistics import *

WD = [355, 5, 15]

u_east = mean([sin(x * pi/180) for x in WD])
u_north = mean([cos(x * pi/180) for x in WD])
unit_WD = atan2(u_east, u_north) * 180/pi
unit_WD = (360 + unit_WD) % 360

print("should be 5 (correct):", unit_WD)
print("should be 125 (wrong):", mean(WD))
James-76 commented 2 years ago

Thanks for the bit about the gapsize - I missed that bit about the dropping the ms which makes sense. Just updated my graphs.conf!

Yes it does do some sql queries but this is for the likes of the Average Climatological Values for this year charts where you want to do xAsis_groupby.

For this graph it would be falling through and using the get_series to gather the data.

            try:
                x_domain = weeutil.weeutil.TimeSpan(start_ts, end_ts)
                (time_start_vt, time_stop_vt, obs_vt) = weewx.xtypes.get_series(
                    obs_lookup,
                    x_domain,
                    archive,
                    aggregate_type,
                    aggregate_interval,
                )
            except Exception as e:
                raise Warning(
                    "Error trying to use database binding %s to graph observation %s. Error was: %s."
                    % (binding, obs_lookup, e)
                )

I can see in xtypes how it does use atan2 under get_aggregate I'm just trying to work out in my head how this should be working

James-76 commented 2 years ago

I've been looking at the code briefly using my data from this morning. I have an interval of 300 seconds.

Setting up a graph to do the following; aggregate_type = avg aggregate_interval = 900

My graph output values on the new graph

04:52   175
05:07   154
05:22   168
05:37   164

My input data was as follows. Using your python script I quickly deduced that it in this case it is aggregating using the archive timestamps as shown 

04:47   175
04:52   175
04:57   175
----
05:02   163
05:07   152
05:12   147
---
05:17   172
05:22   164
05:27   168
---
05:32   160
05:37   167
05:42   165

This is showing the weewx is correctly calculating and aggregating the angle. Looking at the xtypes.py code briefly does try to calculate a windvec for windDir when using aggregate_type = avg

class WindVec(XType):
    """Extensions for calculating special observation types 'windvec' and 'windgustvec' from the
    main archive table. It provides functions for calculating series, and for calculating
    aggregates.
    """

    windvec_types = {
        'windvec': ('windSpeed', 'windDir'),
        'windgustvec': ('windGust', 'windGustDir')
    }

I therefore suspect there is an issue with the calculation as you have pointed out, but it is in the weewx code when the wind direction is coming from the North. We would have to dig further to find / create an exmaple before going back to Tom Keffer and team.

You are on the right track, but I think the issue is probably with the weewx code currently

James-76 commented 2 years ago

OK - I found an example as can be seen here where the second entry when aggregating 355.48, 0.0 and 0.0 should not be 118.5 degrees

        (time_start_vt, time_stop_vt, obs_vt) = weewx.xtypes.get_series(
            obs_lookup,
            x_domain,
            db_lookup(data_binding=binding),
            aggregate_type,
            aggregate_interval,
        )

obs_lookup is windDir
Start time is 1665440100 and end time is 1665442800
x_domain entries are 1665440100 and 1665442800
aggregate_type is avg
aggregate_interval is 300
([355.0, 355.0, 355.0, 355.480837630687, 0.0, 0.0, 0.0, 0.0, 4.249651034454402], 'degree_compass', 'group_direction')
aggregate_interval is 900
([355.0, 118.49361254356234, 1.416550344818134], 'degree_compass', 'group_direction')

So this does need to go back to Tom Keffer as I agree it is calculating the vector incorrectly.

James-76 commented 2 years ago

raised query on weewx-user forum

James-76 commented 2 years ago

In summary it is working as designed currently. Here is the response from Tom.

"Calculating the vector averaged direction requires a vector, so the observation type is 'wind', which is a vector, not 'windDir'. The aggregation that returns direction from a vector is 'vecdir', so, you want:

(start_ts, stop_ts, dirs) = weewx.xtypes.get_series('wind', x_domain, db_lookup(data_binding=binding), 'vecdir', aggregate_interval)

Unfortunately, 'wind' appears only in the daily summaries. This means aggregate_interval must be multiples of one day. This restriction could be relaxed should someone want to write the necessary xtypes extension."

James-76 commented 2 years ago

Since my last update, after somebody else discussed the 'seasons' skin as a result of the discussion on weewx-user, Tom improved the weewx code to allow an aggregation_interval of less than one day for wind. This has been packaged up and released in V4.9.1 of weewx. Please see weewx documentation for more detail about wind.

This does allow you to then use wind and vecdir where it will then create an aggregate wind vector over that period

Using my example from earlier it now gives the following correct output

obs_lookup is wind
Start time is 1665440100 and end time is 1665442800
x_domain entries are 1665440100 and 1665442800
archive is <weewx.manager.DaySummaryManager object at 0xb2d73220>
aggregate_type is vecdir
aggregate_interval is 900
([1665440100, 1665441000, 1665441900], 'unix_epoch', 'group_time')
([355.0, 357.7404188153435, 2.124825517227194], 'degree_compass', 'group_direction')

Within Belchertown graphs.conf you have to do something like the following to produce the graph you are looking after.

    [[dayWind2]]
        title = Wind Direction
        yAxis_min = 0
        aggregate_type = vecdir
        aggregate_interval = 900
        [[[wind]]]
            name = Wind Direction
            zIndex = 0
            yAxis = 0
            yAxis_max = 360
            lineWidth = 0
            [[[[marker]]]]
                enabled = true
            [[[[states]]]]
                [[[[[hover]]]]]
                        lineWidthPlus = 0

which gives the following output

image

This does need a minor change to belchertown.js.tmpl around 2395 to apply the ordinals correctly for wind as well as windDir.

                if (s.obsType == "windDir") {
                    options.yAxis[this_yAxis].tickInterval = 90;

to 

                if (s.obsType == "windDir" || s.obsType == "wind") {
                    options.yAxis[this_yAxis].tickInterval = 90;

Hope this helps you @jkumeboshi as you should be able to create graphs with a interval of 300 seconds even when your archive data is 60 seconds.

James

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

peted-davis commented 1 year ago

I have been trying to make this change to my graphs.conf to get proper averaging of wind direction, but I keep running into an error. I am running weewx version 4.10.2 and my graphs.conf looks like this:

[week]
    title = "Last 7 Days"
    show_button = true
    button_text = "Last 7 Days"
    tooltip_date_format = "LLLL"
    aggregate_type = avg
    time_length = 604800                                # 7 days in seconds
    aggregate_interval = 600                            # 10 minutes in seconds

    [[WindSpeedGraph]]
        connectNulls = true
        title = Wind Speed and Direction
        aggregate_type = vecdir
        yAxis_min = 0
        [[[windDir]]]
            zIndex = 1
            yAxis = 1
            yAxis_max = 360
            lineWidth = 0
            [[[[marker]]]]
                enabled = true
            [[[[states]]]]
                [[[[[hover]]]]]
                        lineWidthPlus = 0
        [[[windGust]]]
        [[[windSpeed]]]
            zIndex = 2
            type = area

but when weewx tries to run the reports each archive interval I get the following error

Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine: Caught unrecoverable exception in generator 'user.belchertown.HighchartsJsonGenerator'
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  Error trying to use database binding wx_binding to graph observation windDir. Error was: no such function: RADIANS.
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weedb/sqlite.py", line 39, in guarded_fn
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      return fn(*args, **kwargs)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weedb/sqlite.py", line 233, in execute
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      return sqlite3.Cursor.execute(self, *args, **kwargs)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  sqlite3.OperationalError: no such function: RADIANS
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/belchertown.py", line 3732, in get_observation_data
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      aggregate_interval,
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/xtypes.py", line 101, in get_series
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      aggregate_interval, **option_dict)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/xtypes.py", line 167, in get_series
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      agg_vt = get_aggregate(obs_type, stamp, do_aggregate, db_manager)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/xtypes.py", line 128, in get_aggregate
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      **option_dict)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/xtypes.py", line 327, in get_aggregate
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      row = db_manager.getSql(select_stmt)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/manager.py", line 579, in getSql
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      _cursor.execute(sql, sqlargs)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weedb/sqlite.py", line 53, in guarded_fn
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      raise weedb.OperationalError(e)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  weedb.OperationalError: no such function: RADIANS
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  During handling of the above exception, another exception occurred:
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  Traceback (most recent call last):
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 197, in run
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      obj.start()
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/weewx/reportengine.py", line 385, in start
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      self.run()
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/belchertown.py", line 2771, in run
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      special_target_unit
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****    File "/home/weewx/bin/user/belchertown.py", line 3737, in get_observation_data
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****      % (binding, obs_lookup, e)
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  Warning: Error trying to use database binding wx_binding to graph observation windDir. Error was: no such function: RADIANS.
Mar  6 09:00:31 Erlinde weewx[1726] ERROR weewx.reportengine:         ****  Generator terminated

It looks like it is related to this: https://groups.google.com/g/weewx-user/c/WQAUCUAKe18/m/fjsbwJSWBQAJ, but when checking the weewx code (specifically sqlite.py), it looks like all the changes suggested by Tom in that thread have already been made.

Any hints?

James-76 commented 1 year ago

@peted-davis, because you are now working with wind aggregation, you now needed to work with [[[wind]]] rather than [[[windDir]]], as wind is a vector of the wind speed and direction over the aggregation period.

James

peted-davis commented 1 year ago

@James-76 - perfect, I missed that small detail. Thanks for pointing me in the right direction!

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.