Invalid Date on Record page

[Stormtracker017]

Please don't delete these questions! They are asked almost with every issue - answering them as below saves us time from asking them!

Describe the bug A clear and concise description of what the bug is. On Record Page in the All Time Column, looks like if it is negative, any invalid date is stated.

Version of the skin you're using

WeeWX Version | 4.3.0 Belchertown Skin Version | 1.2

Screenshots If applicable, add screenshots to help explain your problem.

[gand07]

I have the same issue with negative numbers after upgrading to WeeWx 4.3.0. I also tried to update to the latest Belchertown release, same result. Gert

[ArendPWS]

I did an update from 4.1.1 to 4.3.0 yesterday and experience this error too. Because I skipped WeeWX 4.2.0, I don't know if this error was already present in that version. The error is caused by the "mintime" property, "maxtime" seems to be ok. I noticed that Tom Keffer has done some optimizing on the SQL strings in the last few months maybe that's a clue? The code for "mintime" can be found here: xtypes.py.

There is at least one exception regarding the error retreiving the "mintime", it is still working for the minimum pressure (barometer). So far haven't been able to find what might be the cause of this error. Is it a WeeWX problem or is something missing in the skin that wasn't required before?

Had another problem with XTypes after the update. But that was some special code and solved that one. Maybe the answer to this error could also lie in (new) XTypes code?

This line of code is still working:

jQuery("#at-barometer-min-ts").text( moment.unix($alltime.barometer.mintime.raw).utcOffset($moment_js_utc_offset).format( "$obs.label.time_records_page_full_date" ) );

This line fails:

jQuery("#at-outtemp-min-ts").text( moment.unix($alltime.outTemp.mintime.raw).utcOffset($moment_js_utc_offset).format( "$obs.label.time_records_page_full_date" ) );

Update:

I didn't notice before that the lowest barometer date in the screenshot provided by @Stormtracker017 is also showing "Invalid date". My record for lowest barometer is showing a valid date. This makes the behaviour even more random?

Update 2:

I checked out multiple sites running WeeWX 4.2.0 and none of them seem to have the mentioned error. Conclusion: it must have been introduced in WeeWX 4.3.0. But also found a site running 4.3.0 and Belchertown 1.2, this site displays all the record dates correctly :unamused::

Slate Valley Weather

Update 3:

I suspect that the $alltime tag needs attention. Looking at the relevant code in belchertown.py:

    search_list_extension = { 'belchertown_version': VERSION,
                              'belchertown_debug': belchertown_debug,
                              'moment_js_utc_offset': moment_js_utc_offset,
                              'highcharts_timezoneoffset': highcharts_timezoneoffset,
                              'system_locale': system_locale,
                              'system_locale_js': system_locale_js,
                              'locale_encoding': locale_encoding,
                              'highcharts_decimal': highcharts_decimal,
                              'highcharts_thousands': highcharts_thousands,
                              'radar_html': radar_html,
                              'archive_interval_ms': archive_interval_ms,
                              'ordinate_names': ordinate_names,
                              'charts': json.dumps(charts),
                              'graphpage_titles': json.dumps(graphpage_titles),
                              'graphpage_titles_dict': graphpage_titles,
                              'graphpage_content': json.dumps(graphpage_content),
                              'graph_page_buttons': graph_page_buttons,
                              'alltime' : all_stats,
                              'year_outTemp_range_max': year_outTemp_range_max,
                              'year_outTemp_range_min': year_outTemp_range_min,

And then the all_stats in belchertown.py :

    all_stats = TimespanBinder( timespan,
                                db_lookup,
                                context='year',
                                formatter=self.generator.formatter,
                                converter=self.generator.converter,
                                skin_dict=self.generator.skin_dict )

I have added the context='year' and before that tried the context='current', both made no difference.

Update 4:

Renamed alltime and all_stats in case of tagname conflicts, made no difference. Some things I try will make no sense, but could lead to a clue about the cause. Another tought: I am using a MySQL database instead of SQLite, I would like to know what kind of database @Stormtracker017 and @gand07 are using.

[gand07]

This site has worked fine so far. Suddenly I get Invalid Date on the Records page. Furthermore, Forecast is no longer displayed. Tried deleting forecast.json without success. WeeWx 4.3.0 Belchertown 1.2 /latest version

[41south]

Have a read of this Weewx thread, it may be related

https://groups.google.com/g/weewx-user/c/cPTRWVE2lBE

Cheers Colin

On Fri, 8 Jan 2021 at 20:31, gand07 notifications@github.com wrote:

This site http://billeder.de/ has worked fine so far. Suddenly I get Invalid Date on the Records page. Furthermore, Forecast is no longer displayed. Tried deleting forecast.json without success. WeeWx 4.3.0 Belchertown 1.2 /latest version [image: Invalid Date] https://user-images.githubusercontent.com/47734099/103986847-c0b01000-518b-11eb-8175-2ecee43969f7.jpg

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/poblabs/weewx-belchertown/issues/517#issuecomment-756599358, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANFESNP6SU7RQ6NW373JE3SY2YDLANCNFSM4VW567ZQ .

[ArendPWS]

Think I found the problem.

Made these changes (reverted back) for mintime in xtypes.py using the old SQL strings from WeeWX 4.1.1:

agg_sql_dict = {
    'diff': "SELECT (b.%(obs_type)s - a.%(obs_type)s) FROM archive a, archive b "
            "WHERE b.dateTime = (SELECT MAX(dateTime) FROM archive "
            "WHERE dateTime <= %(stop)s) "
            "AND a.dateTime = (SELECT MIN(dateTime) FROM archive "
            "WHERE dateTime >= %(start)s);",
    'first': "SELECT %(obs_type)s FROM %(table_name)s "
             "WHERE dateTime > %(start)s AND dateTime <= %(stop)s "
             "AND %(obs_type)s IS NOT NULL ORDER BY dateTime ASC LIMIT 1",
    'firsttime': "SELECT MIN(dateTime) FROM %(table_name)s "
                 "WHERE dateTime > %(start)s AND dateTime <= %(stop)s "
                 "AND %(obs_type)s IS NOT NULL",
    'last': "SELECT %(obs_type)s FROM %(table_name)s "
            "WHERE dateTime > %(start)s AND dateTime <= %(stop)s "
            "AND %(obs_type)s IS NOT NULL ORDER BY dateTime DESC LIMIT 1",
    'lasttime': "SELECT MAX(dateTime) FROM %(table_name)s "
                "WHERE dateTime > %(start)s AND dateTime <= %(stop)s "
                "AND %(obs_type)s IS NOT NULL",
    'maxtime': "SELECT dateTime FROM %(table_name)s "
               "WHERE dateTime > %(start)s AND dateTime <= %(stop)s "
               "AND %(obs_type)s IS NOT NULL ORDER BY %(obs_type)s DESC LIMIT 1",
    #'mintime': "SELECT dateTime FROM %(table_name)s "
               #"WHERE dateTime > %(start)s AND dateTime <= %(stop)s "
               #"AND %(obs_type)s IS NOT NULL ORDER BY %(obs_type)s ASC LIMIT 1",
    'mintime': "SELECT dateTime FROM %(table_name)s "
               "WHERE dateTime > %(start)s AND dateTime <= %(stop)s AND "
               "%(obs_type)s = (SELECT MIN(%(obs_type)s) FROM %(table_name)s "
               "WHERE dateTime > %(start)s and dateTime <= %(stop)s) "  
               "AND %(obs_type)s IS NOT NULL",
    'tderiv': "SELECT (b.%(obs_type)s - a.%(obs_type)s) / (b.dateTime-a.dateTime) "
              "FROM archive a, archive b "
              "WHERE b.dateTime = (SELECT MAX(dateTime) FROM archive "
              "WHERE dateTime <= %(stop)s) "
              "AND a.dateTime = (SELECT MIN(dateTime) FROM archive "
              "WHERE dateTime >= %(start)s);",
}

class DailySummaries(XType): """Calculate from the daily summaries."""

# Set of SQL statements to be used for calculating aggregates from the daily summaries.
daily_sql_dict = {
    'avg': "SELECT SUM(wsum),SUM(sumtime) FROM %(table_name)s_day_%(obs_key)s "
           "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'count': "SELECT SUM(count) FROM %(table_name)s_day_%(obs_key)s "
             "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'gustdir': "SELECT max_dir FROM %(table_name)s_day_%(obs_key)s  "
               "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
               "ORDER BY max DESC, maxtime ASC LIMIT 1",
    'max': "SELECT MAX(max) FROM %(table_name)s_day_%(obs_key)s "
           "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'max_ge': "SELECT SUM(max >= %(val)s) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'max_le': "SELECT SUM(max <= %(val)s) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'maxmin': "SELECT MAX(min) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'maxmintime': "SELECT mintime FROM %(table_name)s_day_%(obs_key)s  "
                  "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
                  "ORDER BY min DESC, mintime ASC LIMIT 1",
    'maxsum': "SELECT MAX(sum) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'maxsumtime': "SELECT maxtime FROM %(table_name)s_day_%(obs_key)s  "
                  "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
                  "ORDER BY sum DESC, maxtime ASC LIMIT 1",
    'maxtime': "SELECT maxtime FROM %(table_name)s_day_%(obs_key)s  "
               "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
               "ORDER BY max DESC, maxtime ASC LIMIT 1",
    'meanmax': "SELECT AVG(max) FROM %(table_name)s_day_%(obs_key)s "
               "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'meanmin': "SELECT AVG(min) FROM %(table_name)s_day_%(obs_key)s "
               "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'min': "SELECT MIN(min) FROM %(table_name)s_day_%(obs_key)s "
           "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'min_ge': "SELECT SUM(min >= %(val)s) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'min_le': "SELECT SUM(min <= %(val)s) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'minmax': "SELECT MIN(max) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'minmaxtime': "SELECT maxtime FROM %(table_name)s_day_%(obs_key)s  "
                  "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
                  "ORDER BY max ASC, maxtime ASC ",
    'minsum': "SELECT MIN(sum) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'minsumtime': "SELECT mintime FROM %(table_name)s_day_%(obs_key)s  "
                  "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
                  "ORDER BY sum ASC, mintime ASC LIMIT 1",
    #'mintime': "SELECT mintime FROM %(table_name)s_day_%(obs_key)s  "
               #"WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
               #"ORDER BY min ASC, mintime ASC LIMIT 1",
    'mintime': "SELECT mintime FROM %(table_name)s_day_%(obs_key)s  "
               "WHERE dateTime >= %(start)s AND dateTime < %(stop)s "
               "AND min = (SELECT MIN(min) FROM %(table_name)s_day_%(obs_key)s "
               "WHERE dateTime >= %(start)s AND dateTime <%(stop)s)",
    'rms': "SELECT SUM(wsquaresum),SUM(sumtime) FROM %(table_name)s_day_%(obs_key)s "
           "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'sum': "SELECT SUM(sum) FROM %(table_name)s_day_%(obs_key)s "
           "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'sum_ge': "SELECT SUM(sum >= %(val)s) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'sum_le': "SELECT SUM(sum <= %(val)s) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'vecavg': "SELECT SUM(xsum),SUM(ysum),SUM(sumtime)  FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
    'vecdir': "SELECT SUM(xsum),SUM(ysum) FROM %(table_name)s_day_%(obs_key)s "
              "WHERE dateTime >= %(start)s AND dateTime < %(stop)s",
}

It seems to me that the SQL optimizations created by Tom Keffer can cause problems. Maybe need to open an issue in WeeWX about this?

Result of putting back in 4.1.1 SQL strings for mintime, no Invalid dates anymore:

[michaelundwd]

I took a look at your website at 12.20 hrs your time and the forecast was working correctly However, it does have a number of snow indicators in the forecast that do cause a problem sometime (see #512). I am in the process of fixing this and will issue a PR when done. If it happens again, could you take a look at the console log and let me know if the error is related to an undefined precip value please. I need to know if that is not the problem. Thanks Michael

[gand07]

Hi Michael Thanks for looking. The forecast is more or less on/off during the day. The forecast was just off and I have attached the log. Gert

[michaelundwd]

Thanks @gand07 - that's just what I needed and confirms it is the precip error I was hoping for. That has been corrected in #519 I hope. Forgot to say thanks to @ArendPWS for his helpful suggestions. Michael

[poblabs]

519 is now merged

[ArendPWS]

The issue regarding invalid dates should be solved now.

Please check: Some queries in xtypes do not work for dates with no values and update xtypes.py

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.