Open epag opened 3 weeks ago
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T08:33:01Z
Creating this for the debugging/development in connection w/ #90958, but I also don't mind if we do that work (edit: in whole or in part) in one of the existing tickets (edit: probably #65903, which I will connect momentarily).
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T08:40:22Z
We could reduce the priority if there's a workaround for the immediate situation that makes these evaluations succeed (and that does not increase the timeout period). Some speculative possibilities in #90958 are to add more data (may seem counter-intuitive, but the statistics should stabilize as the db grows and we have not seen this issue on a "full" db instance that I can recall, whereas we have seen it on an empty instance, #84959) and/or to refresh the db statistics.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T08:42:04Z
In the first instance, I'd appreciate if someone could check the db logs for the evaluations in #90958 and confirm that the queries did timeout and then post some example query plans here (although it may be the same as the existing example in #89872-238).
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T08:44:17Z
After that, a snapshot of the -prod db to restore in -ti, but #89872-240 may have achieved that already.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T08:47:51Z
( Grammar. )
epag
commented
3 weeks ago Original Redmine Comment Author Name: Hank (Hank) Original Date: 2021-04-19T12:57:14Z
James,
I have a couple more things to do this morning, but should be able to help out in a bit.
IIRC, I had trouble looking at the db logs before due to permissions, but I think ITSG resolved that. I guess we'll find out.
Hank
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:15:02Z
I'll take another backup shortly, then mitigate with a clean.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:17:05Z
Running this from -ti01: @[~]$ wres-admin-scripts-20210304-5b7ed93/pg_dump_wres.sh -f /wres_share/dbdumps/prod_wres8_5.8_20210419 -h prod-db-fqdn -d wres8 -U wres_user8 -p /usr/pgsql-10/bin/pg_dump@
epag
commented
3 weeks ago Original Redmine Comment Author Name: Hank (Hank) Original Date: 2021-04-19T13:28:06Z
Log attached. I'm not sure what to look for, but can say that there is no mention of "timeout" or "time out". I did find this:
2021-04-17 03:40:57.574 UTC 10.3.10.102 wres8 wres_user8 607a0e3c.1fc1 ERROR: canceling statement due to user request
2021-04-17 03:40:57.574 UTC 10.3.10.102 wres8 wres_user8 607a0e3c.1fc1 STATEMENT: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
and
2021-04-17 09:47:27.995 UTC 10.3.10.104 wres8 wres_user8 607a66ba.14294 ERROR: canceling statement due to user request
2021-04-17 09:47:27.995 UTC 10.3.10.104 wres8 wres_user8 607a66ba.14294 STATEMENT: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
Hank
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:29:55Z
First pass of logs showing probable query timeouts:
postgresql-0416.log:2021-04-16 15:35:20.529 UTC 10.w.v.u wres8 wres_user8 60796333.1eecc ERROR: canceling statement due to user request
postgresql-0417.log:2021-04-17 03:40:57.574 UTC 10.x.y.z wres8 wres_user8 607a0e3c.1fc1 ERROR: canceling statement due to user request
postgresql-0417.log:2021-04-17 09:47:27.995 UTC 10.w.v.u wres8 wres_user8 607a66ba.14294 ERROR: canceling statement due to user request
...
postgresql-0417.log:2021-04-17 19:30:49.530 UTC 10.w.v.u wres8 wres_user8 607aecfa.e59c ERROR: canceling statement due to user request
postgresql-0418.log:2021-04-18 03:11:15.751 UTC 10.x.y.z wres8 wres_user8 607b59a4.3ee1 ERROR: canceling statement due to user request
And first log lines indicating that first session:
2021-04-16 10:13:07.011 UTC 10.w.v.u [unknown] [unknown] 60796333.1eecc LOG: connection received: host=10.w.v.u port=49224
2021-04-16 10:13:07.029 UTC 10.w.v.u wres8 wres_user8 60796333.1eecc LOG: connection authorized: user=wres_user8 database=wres8 SSL enabled (protocol=TLSv1.2, cipher=ECDHE-RSA-AES128-GCM-SHA256, compression=off)
2021-04-16 15:35:20.529 UTC 10.w.v.u wres8 wres_user8 60796333.1eecc ERROR: canceling statement due to user request
2021-04-16 15:35:20.529 UTC 10.w.v.u wres8 wres_user8 60796333.1eecc STATEMENT: SELECT
2021-04-16 15:35:20.750 UTC 10.w.v.u wres8 wres_user8 60796333.1eecc LOG: disconnection: session time: 5:22:13.739 user=wres_user8 database=wres8 host=10.w.v.u port=49224Yes, it's a query timeout.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:30:48Z
The reason the server will not show "timeout" is it is a timeout on the client side, not the server side.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Hank (Hank) Original Date: 2021-04-19T13:30:51Z
Yep, I looked for the wrong phrase; needed to add a 'd' to timed out:
2021-04-17 22:00:43.766 UTC 10.3.10.104 wres8 wres_user8 607a66c5.142a3 LOG: could not send data to client: Connection timed out
2021-04-17 22:00:43.766 UTC 10.3.10.104 wres8 wres_user8 607a66c5.142a3 STATEMENT: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
2021-04-17 22:00:43.766 UTC 10.3.10.104 wres8 wres_user8 607a66c5.142a3 FATAL: connection to client lost
2021-04-17 22:00:43.766 UTC 10.3.10.104 wres8 wres_user8 607a66c5.142a3 STATEMENT: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
2021-04-17 22:00:43.830 UTC 10.3.10.104 wres8 wres_user8 607a66c5.142a3 LOG: disconnection: session time: 17:20:06.828 user=wres_user8 database=wres8 host=10.3.10.104 port=56236
2021-04-17 22:00:45.727 UTC 10.3.10.104 wres8 wres_user8 607a66c2.1429e LOG: could not send data to client: Connection timed out
2021-04-17 22:00:45.727 UTC 10.3.10.104 wres8 wres_user8 607a66c2.1429e STATEMENT: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
2021-04-17 22:00:45.727 UTC 10.3.10.104 wres8 wres_user8 607a66c2.1429e FATAL: connection to client lost
2021-04-17 22:00:45.727 UTC 10.3.10.104 wres8 wres_user8 607a66c2.1429e STATEMENT: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
Hank
epag
commented
3 weeks ago Original Redmine Comment Author Name: Hank (Hank) Original Date: 2021-04-19T13:31:18Z
Looks like Jesse is on it, so I'll step aside for more meeting preparation. Woohoo!
Hank
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:35:00Z
Hank wrote:
Yep, I looked for the wrong phrase; needed to add a 'd' to timed out:
[...]
These look like connection (e.g. socket) timeouts (as opposed to query timeouts).
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:37:18Z
Outline of the db as of a few minutes ago:
wres8=# \dti+ wres.*
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+-------------------------------------------------------+-------+------------+----------------------------------------+------------+-------------------------------------------------------------------------------------------------------------------------------
wres | ensemble | table | wres | | 48 kB |
wres | ensemble_ensemble_name_unique_index | index | wres | ensemble | 16 kB |
wres | ensemble_pk | index | wres | ensemble | 16 kB |
wres | executionlog | table | wres | | 83 MB |
wres | feature | table | wres_user8 | | 904 kB |
wres | feature_name_description_srid_wkt_key | index | wres_user8 | feature | 968 kB |
wres | feature_pkey | index | wres_user8 | feature | 208 kB |
wres | gridprojection | table | wres | | 8192 bytes |
wres | gridprojection_pk | index | wres | gridprojection | 8192 bytes |
wres | measurementunit | table | wres | | 48 kB |
wres | measurementunit_pkey | index | wres | measurementunit | 16 kB |
wres | measurementunit_unit_name_key | index | wres | measurementunit | 16 kB |
wres | netcdfcoordinate | table | wres | | 0 bytes |
wres | netcdfcoordinate_coordinate_idx | index | wres | netcdfcoordinate | 8192 bytes |
wres | project | table | wres_user8 | | 16 kB | Each row represents a full project dataset. Each WRES execution uses exactly one. One dataset may be used by many executions.
wres | project_hash_key | index | wres_user8 | project | 16 kB |
wres | project_pkey | index | wres_user8 | project | 16 kB |
wres | projectsource | table | wres_user8 | | 199 MB | Each row indicates an instance of a source/timeseries to be included in the project dataset for the given member.
wres | projectsource_project_index | index | wres_user8 | projectsource | 148 MB |
wres | projectsource_source_index | index | wres_user8 | projectsource | 112 MB |
wres | source | table | wres_user8 | | 902 MB | For vector data, each row represents a collection of timeseries traces.
wres | source_hash_key | index | wres_user8 | source | 104 MB |
wres | source_pkey | index | wres_user8 | source | 31 MB |
wres | sourcecompleted | table | wres | | 51 MB |
wres | sourcecompleted_source_idx | index | wres | sourcecompleted | 30 MB |
wres | timeseries | table | wres_user8 | | 1025 MB |
wres | timeseries_feature_index | index | wres_user8 | timeseries | 296 MB |
wres | timeseries_pk | index | wres_user8 | timeseries | 232 MB |
wres | timeseries_source_index | index | wres_user8 | timeseries | 277 MB |
wres | timeseriesvalue | table | wres | | 0 bytes |
wres | timeseriesvalue_lead_0 | table | wres_user8 | | 2069 MB |
wres | timeseriesvalue_lead_0_lead_idx | index | wres_user8 | timeseriesvalue_lead_0 | 1333 MB |
wres | timeseriesvalue_lead_0_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_0 | 1114 MB |
wres | timeseriesvalue_lead_1 | table | wres_user8 | | 1843 MB |
wres | timeseriesvalue_lead_10 | table | wres_user8 | | 1380 MB |
wres | timeseriesvalue_lead_100 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_100_lead_idx | index | wres_user8 | timeseriesvalue_lead_100 | 291 MB |
wres | timeseriesvalue_lead_100_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_100 | 229 MB |
wres | timeseriesvalue_lead_101 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_101_lead_idx | index | wres_user8 | timeseriesvalue_lead_101 | 290 MB |
wres | timeseriesvalue_lead_101_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_101 | 229 MB |
wres | timeseriesvalue_lead_102 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_102_lead_idx | index | wres_user8 | timeseriesvalue_lead_102 | 387 MB |
wres | timeseriesvalue_lead_102_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_102 | 305 MB |
wres | timeseriesvalue_lead_103 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_103_lead_idx | index | wres_user8 | timeseriesvalue_lead_103 | 291 MB |
wres | timeseriesvalue_lead_103_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_103 | 229 MB |
wres | timeseriesvalue_lead_104 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_104_lead_idx | index | wres_user8 | timeseriesvalue_lead_104 | 291 MB |
wres | timeseriesvalue_lead_104_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_104 | 229 MB |
wres | timeseriesvalue_lead_105 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_105_lead_idx | index | wres_user8 | timeseriesvalue_lead_105 | 387 MB |
wres | timeseriesvalue_lead_105_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_105 | 305 MB |
wres | timeseriesvalue_lead_106 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_106_lead_idx | index | wres_user8 | timeseriesvalue_lead_106 | 291 MB |
wres | timeseriesvalue_lead_106_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_106 | 229 MB |
wres | timeseriesvalue_lead_107 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_107_lead_idx | index | wres_user8 | timeseriesvalue_lead_107 | 290 MB |
wres | timeseriesvalue_lead_107_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_107 | 229 MB |
wres | timeseriesvalue_lead_108 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_108_lead_idx | index | wres_user8 | timeseriesvalue_lead_108 | 387 MB |
wres | timeseriesvalue_lead_108_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_108 | 305 MB |
wres | timeseriesvalue_lead_109 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_109_lead_idx | index | wres_user8 | timeseriesvalue_lead_109 | 290 MB |
wres | timeseriesvalue_lead_109_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_109 | 229 MB |
wres | timeseriesvalue_lead_10_lead_idx | index | wres_user8 | timeseriesvalue_lead_10 | 886 MB |
wres | timeseriesvalue_lead_10_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_10 | 726 MB |
wres | timeseriesvalue_lead_11 | table | wres_user8 | | 1190 MB |
wres | timeseriesvalue_lead_110 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_110_lead_idx | index | wres_user8 | timeseriesvalue_lead_110 | 291 MB |
wres | timeseriesvalue_lead_110_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_110 | 229 MB |
wres | timeseriesvalue_lead_111 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_111_lead_idx | index | wres_user8 | timeseriesvalue_lead_111 | 388 MB |
wres | timeseriesvalue_lead_111_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_111 | 305 MB |
wres | timeseriesvalue_lead_112 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_112_lead_idx | index | wres_user8 | timeseriesvalue_lead_112 | 291 MB |
wres | timeseriesvalue_lead_112_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_112 | 229 MB |
wres | timeseriesvalue_lead_113 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_113_lead_idx | index | wres_user8 | timeseriesvalue_lead_113 | 291 MB |
wres | timeseriesvalue_lead_113_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_113 | 229 MB |
wres | timeseriesvalue_lead_114 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_114_lead_idx | index | wres_user8 | timeseriesvalue_lead_114 | 388 MB |
wres | timeseriesvalue_lead_114_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_114 | 305 MB |
wres | timeseriesvalue_lead_115 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_115_lead_idx | index | wres_user8 | timeseriesvalue_lead_115 | 291 MB |
wres | timeseriesvalue_lead_115_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_115 | 229 MB |
wres | timeseriesvalue_lead_116 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_116_lead_idx | index | wres_user8 | timeseriesvalue_lead_116 | 290 MB |
wres | timeseriesvalue_lead_116_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_116 | 229 MB |
wres | timeseriesvalue_lead_117 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_117_lead_idx | index | wres_user8 | timeseriesvalue_lead_117 | 388 MB |
wres | timeseriesvalue_lead_117_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_117 | 305 MB |
wres | timeseriesvalue_lead_118 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_118_lead_idx | index | wres_user8 | timeseriesvalue_lead_118 | 290 MB |
wres | timeseriesvalue_lead_118_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_118 | 229 MB |
wres | timeseriesvalue_lead_119 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_119_lead_idx | index | wres_user8 | timeseriesvalue_lead_119 | 290 MB |
wres | timeseriesvalue_lead_119_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_119 | 229 MB |
wres | timeseriesvalue_lead_11_lead_idx | index | wres_user8 | timeseriesvalue_lead_11 | 765 MB |
wres | timeseriesvalue_lead_11_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_11 | 622 MB |
wres | timeseriesvalue_lead_12 | table | wres_user8 | | 1410 MB |
wres | timeseriesvalue_lead_120 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_120_lead_idx | index | wres_user8 | timeseriesvalue_lead_120 | 387 MB |
wres | timeseriesvalue_lead_120_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_120 | 305 MB |
wres | timeseriesvalue_lead_121 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_121_lead_idx | index | wres_user8 | timeseriesvalue_lead_121 | 290 MB |
wres | timeseriesvalue_lead_121_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_121 | 229 MB |
wres | timeseriesvalue_lead_122 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_122_lead_idx | index | wres_user8 | timeseriesvalue_lead_122 | 291 MB |
wres | timeseriesvalue_lead_122_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_122 | 229 MB |
wres | timeseriesvalue_lead_123 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_123_lead_idx | index | wres_user8 | timeseriesvalue_lead_123 | 387 MB |
wres | timeseriesvalue_lead_123_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_123 | 305 MB |
wres | timeseriesvalue_lead_124 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_124_lead_idx | index | wres_user8 | timeseriesvalue_lead_124 | 291 MB |
wres | timeseriesvalue_lead_124_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_124 | 229 MB |
wres | timeseriesvalue_lead_125 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_125_lead_idx | index | wres_user8 | timeseriesvalue_lead_125 | 291 MB |
wres | timeseriesvalue_lead_125_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_125 | 229 MB |
wres | timeseriesvalue_lead_126 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_126_lead_idx | index | wres_user8 | timeseriesvalue_lead_126 | 387 MB |
wres | timeseriesvalue_lead_126_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_126 | 305 MB |
wres | timeseriesvalue_lead_127 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_127_lead_idx | index | wres_user8 | timeseriesvalue_lead_127 | 291 MB |
wres | timeseriesvalue_lead_127_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_127 | 229 MB |
wres | timeseriesvalue_lead_128 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_128_lead_idx | index | wres_user8 | timeseriesvalue_lead_128 | 291 MB |
wres | timeseriesvalue_lead_128_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_128 | 229 MB |
wres | timeseriesvalue_lead_129 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_129_lead_idx | index | wres_user8 | timeseriesvalue_lead_129 | 387 MB |
wres | timeseriesvalue_lead_129_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_129 | 305 MB |
wres | timeseriesvalue_lead_12_lead_idx | index | wres_user8 | timeseriesvalue_lead_12 | 906 MB |
wres | timeseriesvalue_lead_12_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_12 | 725 MB |
wres | timeseriesvalue_lead_13 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_130 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_130_lead_idx | index | wres_user8 | timeseriesvalue_lead_130 | 289 MB |
wres | timeseriesvalue_lead_130_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_130 | 229 MB |
wres | timeseriesvalue_lead_131 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_131_lead_idx | index | wres_user8 | timeseriesvalue_lead_131 | 291 MB |
wres | timeseriesvalue_lead_131_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_131 | 229 MB |
wres | timeseriesvalue_lead_132 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_132_lead_idx | index | wres_user8 | timeseriesvalue_lead_132 | 387 MB |
wres | timeseriesvalue_lead_132_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_132 | 305 MB |
wres | timeseriesvalue_lead_133 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_133_lead_idx | index | wres_user8 | timeseriesvalue_lead_133 | 290 MB |
wres | timeseriesvalue_lead_133_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_133 | 229 MB |
wres | timeseriesvalue_lead_134 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_134_lead_idx | index | wres_user8 | timeseriesvalue_lead_134 | 290 MB |
wres | timeseriesvalue_lead_134_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_134 | 229 MB |
wres | timeseriesvalue_lead_135 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_135_lead_idx | index | wres_user8 | timeseriesvalue_lead_135 | 387 MB |
wres | timeseriesvalue_lead_135_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_135 | 305 MB |
wres | timeseriesvalue_lead_136 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_136_lead_idx | index | wres_user8 | timeseriesvalue_lead_136 | 291 MB |
wres | timeseriesvalue_lead_136_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_136 | 229 MB |
wres | timeseriesvalue_lead_137 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_137_lead_idx | index | wres_user8 | timeseriesvalue_lead_137 | 291 MB |
wres | timeseriesvalue_lead_137_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_137 | 229 MB |
wres | timeseriesvalue_lead_138 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_138_lead_idx | index | wres_user8 | timeseriesvalue_lead_138 | 387 MB |
wres | timeseriesvalue_lead_138_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_138 | 305 MB |
wres | timeseriesvalue_lead_139 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_139_lead_idx | index | wres_user8 | timeseriesvalue_lead_139 | 291 MB |
wres | timeseriesvalue_lead_139_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_139 | 229 MB |
wres | timeseriesvalue_lead_13_lead_idx | index | wres_user8 | timeseriesvalue_lead_13 | 669 MB |
wres | timeseriesvalue_lead_13_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_13 | 531 MB |
wres | timeseriesvalue_lead_14 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_140 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_140_lead_idx | index | wres_user8 | timeseriesvalue_lead_140 | 291 MB |
wres | timeseriesvalue_lead_140_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_140 | 229 MB |
wres | timeseriesvalue_lead_141 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_141_lead_idx | index | wres_user8 | timeseriesvalue_lead_141 | 386 MB |
wres | timeseriesvalue_lead_141_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_141 | 305 MB |
wres | timeseriesvalue_lead_142 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_142_lead_idx | index | wres_user8 | timeseriesvalue_lead_142 | 291 MB |
wres | timeseriesvalue_lead_142_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_142 | 229 MB |
wres | timeseriesvalue_lead_143 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_143_lead_idx | index | wres_user8 | timeseriesvalue_lead_143 | 291 MB |
wres | timeseriesvalue_lead_143_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_143 | 229 MB |
wres | timeseriesvalue_lead_144 | table | wres_user8 | | 602 MB |
wres | timeseriesvalue_lead_144_lead_idx | index | wres_user8 | timeseriesvalue_lead_144 | 386 MB |
wres | timeseriesvalue_lead_144_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_144 | 304 MB |
wres | timeseriesvalue_lead_145 | table | wres_user8 | | 451 MB |
wres | timeseriesvalue_lead_145_lead_idx | index | wres_user8 | timeseriesvalue_lead_145 | 290 MB |
wres | timeseriesvalue_lead_145_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_145 | 228 MB |
wres | timeseriesvalue_lead_146 | table | wres_user8 | | 451 MB |
wres | timeseriesvalue_lead_146_lead_idx | index | wres_user8 | timeseriesvalue_lead_146 | 290 MB |
wres | timeseriesvalue_lead_146_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_146 | 228 MB |
wres | timeseriesvalue_lead_147 | table | wres_user8 | | 602 MB |
wres | timeseriesvalue_lead_147_lead_idx | index | wres_user8 | timeseriesvalue_lead_147 | 387 MB |
wres | timeseriesvalue_lead_147_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_147 | 304 MB |
wres | timeseriesvalue_lead_148 | table | wres_user8 | | 452 MB |
wres | timeseriesvalue_lead_148_lead_idx | index | wres_user8 | timeseriesvalue_lead_148 | 290 MB |
wres | timeseriesvalue_lead_148_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_148 | 228 MB |
wres | timeseriesvalue_lead_149 | table | wres_user8 | | 452 MB |
wres | timeseriesvalue_lead_149_lead_idx | index | wres_user8 | timeseriesvalue_lead_149 | 289 MB |
wres | timeseriesvalue_lead_149_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_149 | 228 MB |
wres | timeseriesvalue_lead_14_lead_idx | index | wres_user8 | timeseriesvalue_lead_14 | 670 MB |
wres | timeseriesvalue_lead_14_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_14 | 531 MB |
wres | timeseriesvalue_lead_15 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_150 | table | wres_user8 | | 602 MB |
wres | timeseriesvalue_lead_150_lead_idx | index | wres_user8 | timeseriesvalue_lead_150 | 386 MB |
wres | timeseriesvalue_lead_150_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_150 | 304 MB |
wres | timeseriesvalue_lead_15_lead_idx | index | wres_user8 | timeseriesvalue_lead_15 | 890 MB |
wres | timeseriesvalue_lead_15_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_15 | 707 MB |
wres | timeseriesvalue_lead_16 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_16_lead_idx | index | wres_user8 | timeseriesvalue_lead_16 | 670 MB |
wres | timeseriesvalue_lead_16_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_16 | 531 MB |
wres | timeseriesvalue_lead_17 | table | wres_user8 | | 1042 MB |
wres | timeseriesvalue_lead_17_lead_idx | index | wres_user8 | timeseriesvalue_lead_17 | 669 MB |
wres | timeseriesvalue_lead_17_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_17 | 531 MB |
wres | timeseriesvalue_lead_18 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_18_lead_idx | index | wres_user8 | timeseriesvalue_lead_18 | 890 MB |
wres | timeseriesvalue_lead_18_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_18 | 707 MB |
wres | timeseriesvalue_lead_19 | table | wres_user8 | | 1042 MB |
wres | timeseriesvalue_lead_19_lead_idx | index | wres_user8 | timeseriesvalue_lead_19 | 669 MB |
wres | timeseriesvalue_lead_19_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_19 | 531 MB |
wres | timeseriesvalue_lead_1_lead_idx | index | wres_user8 | timeseriesvalue_lead_1 | 1185 MB |
wres | timeseriesvalue_lead_1_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_1 | 977 MB |
wres | timeseriesvalue_lead_2 | table | wres_user8 | | 1734 MB |
wres | timeseriesvalue_lead_20 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_20_lead_idx | index | wres_user8 | timeseriesvalue_lead_20 | 670 MB |
wres | timeseriesvalue_lead_20_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_20 | 531 MB |
wres | timeseriesvalue_lead_21 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_21_lead_idx | index | wres_user8 | timeseriesvalue_lead_21 | 890 MB |
wres | timeseriesvalue_lead_21_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_21 | 707 MB |
wres | timeseriesvalue_lead_22 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_22_lead_idx | index | wres_user8 | timeseriesvalue_lead_22 | 668 MB |
wres | timeseriesvalue_lead_22_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_22 | 531 MB |
wres | timeseriesvalue_lead_23 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_23_lead_idx | index | wres_user8 | timeseriesvalue_lead_23 | 670 MB |
wres | timeseriesvalue_lead_23_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_23 | 531 MB |
wres | timeseriesvalue_lead_24 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_24_lead_idx | index | wres_user8 | timeseriesvalue_lead_24 | 890 MB |
wres | timeseriesvalue_lead_24_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_24 | 707 MB |
wres | timeseriesvalue_lead_25 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_25_lead_idx | index | wres_user8 | timeseriesvalue_lead_25 | 669 MB |
wres | timeseriesvalue_lead_25_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_25 | 531 MB |
wres | timeseriesvalue_lead_26 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_26_lead_idx | index | wres_user8 | timeseriesvalue_lead_26 | 670 MB |
wres | timeseriesvalue_lead_26_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_26 | 531 MB |
wres | timeseriesvalue_lead_27 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_27_lead_idx | index | wres_user8 | timeseriesvalue_lead_27 | 891 MB |
wres | timeseriesvalue_lead_27_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_27 | 707 MB |
wres | timeseriesvalue_lead_28 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_28_lead_idx | index | wres_user8 | timeseriesvalue_lead_28 | 670 MB |
wres | timeseriesvalue_lead_28_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_28 | 531 MB |
wres | timeseriesvalue_lead_29 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_29_lead_idx | index | wres_user8 | timeseriesvalue_lead_29 | 669 MB |
wres | timeseriesvalue_lead_29_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_29 | 531 MB |
wres | timeseriesvalue_lead_2_lead_idx | index | wres_user8 | timeseriesvalue_lead_2 | 1116 MB |
wres | timeseriesvalue_lead_2_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_2 | 919 MB |
wres | timeseriesvalue_lead_3 | table | wres_user8 | | 2179 MB |
wres | timeseriesvalue_lead_30 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_30_lead_idx | index | wres_user8 | timeseriesvalue_lead_30 | 891 MB |
wres | timeseriesvalue_lead_30_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_30 | 707 MB |
wres | timeseriesvalue_lead_31 | table | wres_user8 | | 1043 MB |
wres | timeseriesvalue_lead_31_lead_idx | index | wres_user8 | timeseriesvalue_lead_31 | 670 MB |
wres | timeseriesvalue_lead_31_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_31 | 531 MB |
wres | timeseriesvalue_lead_32 | table | wres_user8 | | 1042 MB |
wres | timeseriesvalue_lead_32_lead_idx | index | wres_user8 | timeseriesvalue_lead_32 | 669 MB |
wres | timeseriesvalue_lead_32_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_32 | 531 MB |
wres | timeseriesvalue_lead_33 | table | wres_user8 | | 1385 MB |
wres | timeseriesvalue_lead_33_lead_idx | index | wres_user8 | timeseriesvalue_lead_33 | 890 MB |
wres | timeseriesvalue_lead_33_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_33 | 707 MB |
wres | timeseriesvalue_lead_34 | table | wres_user8 | | 1042 MB |
wres | timeseriesvalue_lead_34_lead_idx | index | wres_user8 | timeseriesvalue_lead_34 | 669 MB |
wres | timeseriesvalue_lead_34_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_34 | 531 MB |
wres | timeseriesvalue_lead_35 | table | wres_user8 | | 1042 MB |
wres | timeseriesvalue_lead_35_lead_idx | index | wres_user8 | timeseriesvalue_lead_35 | 669 MB |
wres | timeseriesvalue_lead_35_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_35 | 531 MB |
wres | timeseriesvalue_lead_36 | table | wres_user8 | | 808 MB |
wres | timeseriesvalue_lead_36_lead_idx | index | wres_user8 | timeseriesvalue_lead_36 | 519 MB |
wres | timeseriesvalue_lead_36_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_36 | 409 MB |
wres | timeseriesvalue_lead_37 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_37_lead_idx | index | wres_user8 | timeseriesvalue_lead_37 | 298 MB |
wres | timeseriesvalue_lead_37_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_37 | 235 MB |
wres | timeseriesvalue_lead_38 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_38_lead_idx | index | wres_user8 | timeseriesvalue_lead_38 | 298 MB |
wres | timeseriesvalue_lead_38_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_38 | 235 MB |
wres | timeseriesvalue_lead_39 | table | wres_user8 | | 615 MB |
wres | timeseriesvalue_lead_39_lead_idx | index | wres_user8 | timeseriesvalue_lead_39 | 395 MB |
wres | timeseriesvalue_lead_39_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_39 | 312 MB |
wres | timeseriesvalue_lead_3_lead_idx | index | wres_user8 | timeseriesvalue_lead_3 | 1402 MB |
wres | timeseriesvalue_lead_3_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_3 | 1159 MB |
wres | timeseriesvalue_lead_4 | table | wres_user8 | | 1833 MB |
wres | timeseriesvalue_lead_40 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_40_lead_idx | index | wres_user8 | timeseriesvalue_lead_40 | 298 MB |
wres | timeseriesvalue_lead_40_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_40 | 235 MB |
wres | timeseriesvalue_lead_41 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_41_lead_idx | index | wres_user8 | timeseriesvalue_lead_41 | 298 MB |
wres | timeseriesvalue_lead_41_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_41 | 235 MB |
wres | timeseriesvalue_lead_42 | table | wres_user8 | | 614 MB |
wres | timeseriesvalue_lead_42_lead_idx | index | wres_user8 | timeseriesvalue_lead_42 | 395 MB |
wres | timeseriesvalue_lead_42_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_42 | 311 MB |
wres | timeseriesvalue_lead_43 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_43_lead_idx | index | wres_user8 | timeseriesvalue_lead_43 | 298 MB |
wres | timeseriesvalue_lead_43_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_43 | 235 MB |
wres | timeseriesvalue_lead_44 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_44_lead_idx | index | wres_user8 | timeseriesvalue_lead_44 | 298 MB |
wres | timeseriesvalue_lead_44_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_44 | 235 MB |
wres | timeseriesvalue_lead_45 | table | wres_user8 | | 615 MB |
wres | timeseriesvalue_lead_45_lead_idx | index | wres_user8 | timeseriesvalue_lead_45 | 395 MB |
wres | timeseriesvalue_lead_45_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_45 | 312 MB |
wres | timeseriesvalue_lead_46 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_46_lead_idx | index | wres_user8 | timeseriesvalue_lead_46 | 298 MB |
wres | timeseriesvalue_lead_46_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_46 | 235 MB |
wres | timeseriesvalue_lead_47 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_47_lead_idx | index | wres_user8 | timeseriesvalue_lead_47 | 298 MB |
wres | timeseriesvalue_lead_47_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_47 | 235 MB |
wres | timeseriesvalue_lead_48 | table | wres_user8 | | 615 MB |
wres | timeseriesvalue_lead_48_lead_idx | index | wres_user8 | timeseriesvalue_lead_48 | 394 MB |
wres | timeseriesvalue_lead_48_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_48 | 312 MB |
wres | timeseriesvalue_lead_49 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_49_lead_idx | index | wres_user8 | timeseriesvalue_lead_49 | 298 MB |
wres | timeseriesvalue_lead_49_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_49 | 235 MB |
wres | timeseriesvalue_lead_4_lead_idx | index | wres_user8 | timeseriesvalue_lead_4 | 1179 MB |
wres | timeseriesvalue_lead_4_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_4 | 974 MB |
wres | timeseriesvalue_lead_5 | table | wres_user8 | | 1723 MB |
wres | timeseriesvalue_lead_50 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_50_lead_idx | index | wres_user8 | timeseriesvalue_lead_50 | 298 MB |
wres | timeseriesvalue_lead_50_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_50 | 235 MB |
wres | timeseriesvalue_lead_51 | table | wres_user8 | | 615 MB |
wres | timeseriesvalue_lead_51_lead_idx | index | wres_user8 | timeseriesvalue_lead_51 | 394 MB |
wres | timeseriesvalue_lead_51_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_51 | 311 MB |
wres | timeseriesvalue_lead_52 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_52_lead_idx | index | wres_user8 | timeseriesvalue_lead_52 | 299 MB |
wres | timeseriesvalue_lead_52_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_52 | 235 MB |
wres | timeseriesvalue_lead_53 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_53_lead_idx | index | wres_user8 | timeseriesvalue_lead_53 | 298 MB |
wres | timeseriesvalue_lead_53_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_53 | 235 MB |
wres | timeseriesvalue_lead_54 | table | wres_user8 | | 614 MB |
wres | timeseriesvalue_lead_54_lead_idx | index | wres_user8 | timeseriesvalue_lead_54 | 395 MB |
wres | timeseriesvalue_lead_54_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_54 | 311 MB |
wres | timeseriesvalue_lead_55 | table | wres_user8 | | 465 MB |
wres | timeseriesvalue_lead_55_lead_idx | index | wres_user8 | timeseriesvalue_lead_55 | 298 MB |
wres | timeseriesvalue_lead_55_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_55 | 235 MB |
wres | timeseriesvalue_lead_56 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_56_lead_idx | index | wres_user8 | timeseriesvalue_lead_56 | 298 MB |
wres | timeseriesvalue_lead_56_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_56 | 235 MB |
wres | timeseriesvalue_lead_57 | table | wres_user8 | | 615 MB |
wres | timeseriesvalue_lead_57_lead_idx | index | wres_user8 | timeseriesvalue_lead_57 | 395 MB |
wres | timeseriesvalue_lead_57_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_57 | 312 MB |
wres | timeseriesvalue_lead_58 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_58_lead_idx | index | wres_user8 | timeseriesvalue_lead_58 | 298 MB |
wres | timeseriesvalue_lead_58_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_58 | 235 MB |
wres | timeseriesvalue_lead_59 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_59_lead_idx | index | wres_user8 | timeseriesvalue_lead_59 | 298 MB |
wres | timeseriesvalue_lead_59_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_59 | 235 MB |
wres | timeseriesvalue_lead_5_lead_idx | index | wres_user8 | timeseriesvalue_lead_5 | 1108 MB |
wres | timeseriesvalue_lead_5_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_5 | 914 MB |
wres | timeseriesvalue_lead_6 | table | wres_user8 | | 2147 MB |
wres | timeseriesvalue_lead_60 | table | wres_user8 | | 614 MB |
wres | timeseriesvalue_lead_60_lead_idx | index | wres_user8 | timeseriesvalue_lead_60 | 394 MB |
wres | timeseriesvalue_lead_60_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_60 | 312 MB |
wres | timeseriesvalue_lead_61 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_61_lead_idx | index | wres_user8 | timeseriesvalue_lead_61 | 298 MB |
wres | timeseriesvalue_lead_61_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_61 | 235 MB |
wres | timeseriesvalue_lead_62 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_62_lead_idx | index | wres_user8 | timeseriesvalue_lead_62 | 298 MB |
wres | timeseriesvalue_lead_62_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_62 | 235 MB |
wres | timeseriesvalue_lead_63 | table | wres_user8 | | 614 MB |
wres | timeseriesvalue_lead_63_lead_idx | index | wres_user8 | timeseriesvalue_lead_63 | 395 MB |
wres | timeseriesvalue_lead_63_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_63 | 311 MB |
wres | timeseriesvalue_lead_64 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_64_lead_idx | index | wres_user8 | timeseriesvalue_lead_64 | 298 MB |
wres | timeseriesvalue_lead_64_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_64 | 235 MB |
wres | timeseriesvalue_lead_65 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_65_lead_idx | index | wres_user8 | timeseriesvalue_lead_65 | 298 MB |
wres | timeseriesvalue_lead_65_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_65 | 235 MB |
wres | timeseriesvalue_lead_66 | table | wres_user8 | | 614 MB |
wres | timeseriesvalue_lead_66_lead_idx | index | wres_user8 | timeseriesvalue_lead_66 | 395 MB |
wres | timeseriesvalue_lead_66_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_66 | 311 MB |
wres | timeseriesvalue_lead_67 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_67_lead_idx | index | wres_user8 | timeseriesvalue_lead_67 | 298 MB |
wres | timeseriesvalue_lead_67_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_67 | 235 MB |
wres | timeseriesvalue_lead_68 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_68_lead_idx | index | wres_user8 | timeseriesvalue_lead_68 | 299 MB |
wres | timeseriesvalue_lead_68_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_68 | 235 MB |
wres | timeseriesvalue_lead_69 | table | wres_user8 | | 614 MB |
wres | timeseriesvalue_lead_69_lead_idx | index | wres_user8 | timeseriesvalue_lead_69 | 395 MB |
wres | timeseriesvalue_lead_69_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_69 | 312 MB |
wres | timeseriesvalue_lead_6_lead_idx | index | wres_user8 | timeseriesvalue_lead_6 | 1381 MB |
wres | timeseriesvalue_lead_6_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_6 | 1141 MB |
wres | timeseriesvalue_lead_7 | table | wres_user8 | | 1803 MB |
wres | timeseriesvalue_lead_70 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_70_lead_idx | index | wres_user8 | timeseriesvalue_lead_70 | 298 MB |
wres | timeseriesvalue_lead_70_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_70 | 235 MB |
wres | timeseriesvalue_lead_71 | table | wres_user8 | | 464 MB |
wres | timeseriesvalue_lead_71_lead_idx | index | wres_user8 | timeseriesvalue_lead_71 | 298 MB |
wres | timeseriesvalue_lead_71_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_71 | 235 MB |
wres | timeseriesvalue_lead_72 | table | wres_user8 | | 606 MB |
wres | timeseriesvalue_lead_72_lead_idx | index | wres_user8 | timeseriesvalue_lead_72 | 389 MB |
wres | timeseriesvalue_lead_72_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_72 | 306 MB |
wres | timeseriesvalue_lead_73 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_73_lead_idx | index | wres_user8 | timeseriesvalue_lead_73 | 291 MB |
wres | timeseriesvalue_lead_73_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_73 | 229 MB |
wres | timeseriesvalue_lead_74 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_74_lead_idx | index | wres_user8 | timeseriesvalue_lead_74 | 291 MB |
wres | timeseriesvalue_lead_74_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_74 | 229 MB |
wres | timeseriesvalue_lead_75 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_75_lead_idx | index | wres_user8 | timeseriesvalue_lead_75 | 388 MB |
wres | timeseriesvalue_lead_75_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_75 | 305 MB |
wres | timeseriesvalue_lead_76 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_76_lead_idx | index | wres_user8 | timeseriesvalue_lead_76 | 291 MB |
wres | timeseriesvalue_lead_76_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_76 | 229 MB |
wres | timeseriesvalue_lead_77 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_77_lead_idx | index | wres_user8 | timeseriesvalue_lead_77 | 291 MB |
wres | timeseriesvalue_lead_77_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_77 | 229 MB |
wres | timeseriesvalue_lead_78 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_78_lead_idx | index | wres_user8 | timeseriesvalue_lead_78 | 387 MB |
wres | timeseriesvalue_lead_78_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_78 | 305 MB |
wres | timeseriesvalue_lead_79 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_79_lead_idx | index | wres_user8 | timeseriesvalue_lead_79 | 291 MB |
wres | timeseriesvalue_lead_79_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_79 | 229 MB |
wres | timeseriesvalue_lead_7_lead_idx | index | wres_user8 | timeseriesvalue_lead_7 | 1158 MB |
wres | timeseriesvalue_lead_7_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_7 | 958 MB |
wres | timeseriesvalue_lead_8 | table | wres_user8 | | 1693 MB |
wres | timeseriesvalue_lead_80 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_80_lead_idx | index | wres_user8 | timeseriesvalue_lead_80 | 291 MB |
wres | timeseriesvalue_lead_80_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_80 | 229 MB |
wres | timeseriesvalue_lead_81 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_81_lead_idx | index | wres_user8 | timeseriesvalue_lead_81 | 388 MB |
wres | timeseriesvalue_lead_81_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_81 | 305 MB |
wres | timeseriesvalue_lead_82 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_82_lead_idx | index | wres_user8 | timeseriesvalue_lead_82 | 291 MB |
wres | timeseriesvalue_lead_82_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_82 | 229 MB |
wres | timeseriesvalue_lead_83 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_83_lead_idx | index | wres_user8 | timeseriesvalue_lead_83 | 291 MB |
wres | timeseriesvalue_lead_83_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_83 | 229 MB |
wres | timeseriesvalue_lead_84 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_84_lead_idx | index | wres_user8 | timeseriesvalue_lead_84 | 388 MB |
wres | timeseriesvalue_lead_84_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_84 | 305 MB |
wres | timeseriesvalue_lead_85 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_85_lead_idx | index | wres_user8 | timeseriesvalue_lead_85 | 291 MB |
wres | timeseriesvalue_lead_85_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_85 | 229 MB |
wres | timeseriesvalue_lead_86 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_86_lead_idx | index | wres_user8 | timeseriesvalue_lead_86 | 291 MB |
wres | timeseriesvalue_lead_86_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_86 | 229 MB |
wres | timeseriesvalue_lead_87 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_87_lead_idx | index | wres_user8 | timeseriesvalue_lead_87 | 388 MB |
wres | timeseriesvalue_lead_87_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_87 | 305 MB |
wres | timeseriesvalue_lead_88 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_88_lead_idx | index | wres_user8 | timeseriesvalue_lead_88 | 290 MB |
wres | timeseriesvalue_lead_88_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_88 | 229 MB |
wres | timeseriesvalue_lead_89 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_89_lead_idx | index | wres_user8 | timeseriesvalue_lead_89 | 291 MB |
wres | timeseriesvalue_lead_89_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_89 | 229 MB |
wres | timeseriesvalue_lead_8_lead_idx | index | wres_user8 | timeseriesvalue_lead_8 | 1087 MB |
wres | timeseriesvalue_lead_8_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_8 | 897 MB |
wres | timeseriesvalue_lead_9 | table | wres_user8 | | 2137 MB |
wres | timeseriesvalue_lead_90 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_90_lead_idx | index | wres_user8 | timeseriesvalue_lead_90 | 388 MB |
wres | timeseriesvalue_lead_90_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_90 | 305 MB |
wres | timeseriesvalue_lead_91 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_91_lead_idx | index | wres_user8 | timeseriesvalue_lead_91 | 291 MB |
wres | timeseriesvalue_lead_91_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_91 | 229 MB |
wres | timeseriesvalue_lead_92 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_92_lead_idx | index | wres_user8 | timeseriesvalue_lead_92 | 291 MB |
wres | timeseriesvalue_lead_92_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_92 | 229 MB |
wres | timeseriesvalue_lead_93 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_93_lead_idx | index | wres_user8 | timeseriesvalue_lead_93 | 387 MB |
wres | timeseriesvalue_lead_93_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_93 | 305 MB |
wres | timeseriesvalue_lead_94 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_94_lead_idx | index | wres_user8 | timeseriesvalue_lead_94 | 291 MB |
wres | timeseriesvalue_lead_94_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_94 | 229 MB |
wres | timeseriesvalue_lead_95 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_95_lead_idx | index | wres_user8 | timeseriesvalue_lead_95 | 291 MB |
wres | timeseriesvalue_lead_95_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_95 | 229 MB |
wres | timeseriesvalue_lead_96 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_96_lead_idx | index | wres_user8 | timeseriesvalue_lead_96 | 387 MB |
wres | timeseriesvalue_lead_96_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_96 | 305 MB |
wres | timeseriesvalue_lead_97 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_97_lead_idx | index | wres_user8 | timeseriesvalue_lead_97 | 290 MB |
wres | timeseriesvalue_lead_97_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_97 | 229 MB |
wres | timeseriesvalue_lead_98 | table | wres_user8 | | 453 MB |
wres | timeseriesvalue_lead_98_lead_idx | index | wres_user8 | timeseriesvalue_lead_98 | 291 MB |
wres | timeseriesvalue_lead_98_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_98 | 229 MB |
wres | timeseriesvalue_lead_99 | table | wres_user8 | | 603 MB |
wres | timeseriesvalue_lead_99_lead_idx | index | wres_user8 | timeseriesvalue_lead_99 | 388 MB |
wres | timeseriesvalue_lead_99_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_99 | 305 MB |
wres | timeseriesvalue_lead_9_lead_idx | index | wres_user8 | timeseriesvalue_lead_9 | 1373 MB |
wres | timeseriesvalue_lead_9_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_9 | 1138 MB |
wres | timeseriesvalue_lead_above_150 | table | wres_user8 | | 140 GB |
wres | timeseriesvalue_lead_above_150_lead_idx | index | wres_user8 | timeseriesvalue_lead_above_150 | 90 GB |
wres | timeseriesvalue_lead_above_150_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_above_150 | 96 GB |
wres | timeseriesvalue_lead_below_negative_10 | table | wres_user8 | | 9162 MB |
wres | timeseriesvalue_lead_below_negative_10_lead_idx | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 6052 MB |
wres | timeseriesvalue_lead_below_negative_10_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 5989 MB |
wres | timeseriesvalue_lead_negative_1 | table | wres_user8 | | 186 MB |
wres | timeseriesvalue_lead_negative_10 | table | wres_user8 | | 185 MB |
wres | timeseriesvalue_lead_negative_10_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_10 | 119 MB |
wres | timeseriesvalue_lead_negative_10_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_10 | 97 MB |
wres | timeseriesvalue_lead_negative_1_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_1 | 120 MB |
wres | timeseriesvalue_lead_negative_1_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_1 | 99 MB |
wres | timeseriesvalue_lead_negative_2 | table | wres_user8 | | 185 MB |
wres | timeseriesvalue_lead_negative_2_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_2 | 120 MB |
wres | timeseriesvalue_lead_negative_2_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_2 | 98 MB |
wres | timeseriesvalue_lead_negative_3 | table | wres_user8 | | 185 MB |
wres | timeseriesvalue_lead_negative_3_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_3 | 119 MB |
wres | timeseriesvalue_lead_negative_3_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_3 | 98 MB |
wres | timeseriesvalue_lead_negative_4 | table | wres_user8 | | 185 MB |
wres | timeseriesvalue_lead_negative_4_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_4 | 119 MB |
wres | timeseriesvalue_lead_negative_4_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_4 | 97 MB |
wres | timeseriesvalue_lead_negative_5 | table | wres_user8 | | 184 MB |
wres | timeseriesvalue_lead_negative_5_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_5 | 119 MB |
wres | timeseriesvalue_lead_negative_5_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_5 | 97 MB |
wres | timeseriesvalue_lead_negative_6 | table | wres_user8 | | 184 MB |
wres | timeseriesvalue_lead_negative_6_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_6 | 119 MB |
wres | timeseriesvalue_lead_negative_6_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_6 | 97 MB |
wres | timeseriesvalue_lead_negative_7 | table | wres_user8 | | 184 MB |
wres | timeseriesvalue_lead_negative_7_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_7 | 119 MB |
wres | timeseriesvalue_lead_negative_7_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_7 | 97 MB |
wres | timeseriesvalue_lead_negative_8 | table | wres_user8 | | 185 MB |
wres | timeseriesvalue_lead_negative_8_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_8 | 119 MB |
wres | timeseriesvalue_lead_negative_8_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_8 | 97 MB |
wres | timeseriesvalue_lead_negative_9 | table | wres_user8 | | 185 MB |
wres | timeseriesvalue_lead_negative_9_lead_idx | index | wres_user8 | timeseriesvalue_lead_negative_9 | 119 MB |
wres | timeseriesvalue_lead_negative_9_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_negative_9 | 97 MB |
wres | timeseriesvalue_timeseries_idx | index | wres | timeseriesvalue | 8192 bytes |
wres | unitconversion | table | wres | | 72 kB |
wres | unitconversion_measurement_uix | index | wres | unitconversion | 40 kB |
(522 rows)
wres8=# \l+
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
-----------+------------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 7855 kB | pg_default | default administrative connection database
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +| 7863 kB | pg_default | unmodifiable empty database
| | | | | postgres=CTc/postgres | | |
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +| 7863 kB | pg_default | default template for new databases
| | | | | postgres=CTc/postgres | | |
wres8 | wres_user8 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 582 GB | pg_default |
(4 rows)
Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T13:42:00Z
Thanks, both.
One point of detail. As Jesse notes, this is triggered on the client side, which defines the timeout period (@DatabaseSettings@, iirc). However, the actual exception is triggered by the server when the client cancels, which in turn disrupts the client retrieval and throws the client exception (@DataAccessException@).
I guess I would expect to see a @SQLTimeoutException@ somewhere in the client logs, but I don't recall seeing this as being a trigger, i.e., the cancellation exception propagates via the server back to the client before any client timeout triggers an @SQLTimeoutException@? Probably not important, but it would be nice to see clear evidence of a client timeout in the client logs.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:43:49Z
Queries that finished between the "I am going to log the plan server-side" and "I am going to time out client-side" limits:
-bash-4.2$ grep -A 5 duration postgresql-041[1-9].log
postgresql-0415.log:2021-04-15 19:21:46.085 UTC ti01 wres8 wres_user8 607851ba.6587 LOG: duration: 8382154.496 ms statement: COPY wres.timeseriesvalue_lead_above_150 (timeseries_id, lead, series_value) TO stdout;
postgresql-0415.log-2021-04-15 19:25:55.590 UTC 10.w.v.u wres8 wres_user8 60788db5.1286a LOG: disconnection: session time: 0:23:42.553 user=wres_user8 database=wres8 host=10.w.v.u port=35766
postgresql-0415.log-2021-04-15 19:25:55.591 UTC 10.w.v.u wres8 wres_user8 60788db5.1286b LOG: disconnection: session time: 0:23:42.525 user=wres_user8 database=wres8 host=10.w.v.u port=35768
postgresql-0415.log-2021-04-15 19:25:56.674 UTC 10.w.v.u wres8 wres_user8 60788db3.1285b LOG: disconnection: session time: 0:23:45.548 user=wres_user8 database=wres8 host=10.w.v.u port=35736
postgresql-0415.log-2021-04-15 19:25:56.675 UTC 10.w.v.u wres8 wres_user8 60788db3.1285d LOG: disconnection: session time: 0:23:45.388 user=wres_user8 database=wres8 host=10.w.v.u port=35740
postgresql-0415.log-2021-04-15 19:25:56.675 UTC 10.w.v.u wres8 wres_user8 60788db3.1285f LOG: disconnection: session time: 0:23:45.295 user=wres_user8 database=wres8 host=10.w.v.u port=35744
--
postgresql-0416.log:2021-04-16 09:04:55.366 UTC 10.w.v.u wres8 wres_user8 60794470.18c82 LOG: duration: 3725464.856 ms execute <unnamed>/C_1: SELECT
postgresql-0416.log- MIN(TS.timeseries_id) AS series_id,
postgresql-0416.log- TS.initialization_date AS reference_time,
postgresql-0416.log- TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
postgresql-0416.log- ARRAY_AGG(
postgresql-0416.log- TSV.series_value
--
postgresql-0416.log:2021-04-16 09:05:02.795 UTC 10.w.v.u wres8 wres_user8 60794470.18c82 LOG: duration: 3728024.525 ms plan:
postgresql-0416.log- Query Text: SELECT
postgresql-0416.log- MIN(TS.timeseries_id) AS series_id,
postgresql-0416.log- TS.initialization_date AS reference_time,
postgresql-0416.log- TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
postgresql-0416.log- ARRAY_AGG(
--
postgresql-0416.log:2021-04-16 09:13:25.838 UTC 10.w.v.u wres8 wres_user8 60794466.18c77 LOG: duration: 4236559.530 ms execute <unnamed>/C_1: SELECT
postgresql-0416.log- MIN(TS.timeseries_id) AS series_id,
postgresql-0416.log- TS.initialization_date AS reference_time,
postgresql-0416.log- TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
postgresql-0416.log- ARRAY_AGG(
postgresql-0416.log- TSV.series_value
--
postgresql-0416.log:2021-04-16 09:13:32.903 UTC 10.w.v.u wres8 wres_user8 60794466.18c77 LOG: duration: 4238947.782 ms plan:
postgresql-0416.log- Query Text: SELECT
postgresql-0416.log- MIN(TS.timeseries_id) AS series_id,
postgresql-0416.log- TS.initialization_date AS reference_time,
postgresql-0416.log- TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
postgresql-0416.log- ARRAY_AGG(
--
postgresql-0416.log:2021-04-16 10:05:57.456 UTC 10.w.v.u wres8 wres_user8 6079508f.1b31b LOG: duration: 4049322.766 ms execute <unnamed>/C_1: SELECT
postgresql-0416.log- MIN(TS.timeseries_id) AS series_id,
postgresql-0416.log- TS.initialization_date AS reference_time,
postgresql-0416.log- TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
postgresql-0416.log- ARRAY_AGG(
postgresql-0416.log- TSV.series_value
--
postgresql-0416.log:2021-04-16 10:06:04.107 UTC 10.w.v.u wres8 wres_user8 6079508f.1b31b LOG: duration: 4051409.541 ms plan:
postgresql-0416.log- Query Text: SELECT
postgresql-0416.log- MIN(TS.timeseries_id) AS series_id,
postgresql-0416.log- TS.initialization_date AS reference_time,
postgresql-0416.log- TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
postgresql-0416.log- ARRAY_AGG(
--
postgresql-0416.log:2021-04-16 17:18:26.427 UTC ti01 wres8 wres_user8 6079aa8b.da86 LOG: duration: 4223142.245 ms statement: COPY wres.timeseriesvalue_lead_above_150 (timeseries_id, lead, series_value) TO stdout;
I was surprised by the COPY showing up until I realized these were the database dumps from ti01, not from the WRES application.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:47:47Z
Choices as I see them:
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T13:49:17Z
It seems clear to me that we do some combination of those because we don't want days-to-months of downtime.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T13:57:46Z
Jesse wrote:
- Mitigate by rolling back to 5.7 (could take around 4h-32h)
I'd be surprised if that were mitigation at all and we'd want to analyze the cause further while considering that. We'd need to look at what changed that could impact these queries within the 5.8 cycle and I think the answer is "nothing material", unless I am not recalling some significant change in that area, so we'd definitely want to test 5.7 first and, if that is not slow, the question surrounding why would arise. The only things I recall are the addition of the labels (edit: and I did some limited performance testing of that) and the use of prepared statements.
Separately, I don't think a straight swap back to 5.7 is an option anyway because the filtering of observations (for which the labels are needed) is a deal breaker for HEFS use, so rolling back would involve significantly more time/effort/whatevers than "32" in my estimate.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:00:17Z
Jesse wrote:
- Mitigate by cleaning after every handful of HEFS BV runs (takes around 0h-1h a day)
Also not clear to me that this is mitigation. If I'm reading #90967-20 correctly, there's very little there and the context is that such timeouts have happened twice early in a release (albeit with different queries) and never later in a release.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:04:59Z
Jesse wrote:
- Mitigate with a better query (Chris provided one, something in the 8h-32h range I guess)
We can look at that. However, I've tried something very similar to that variation before and I don't recall it helping, but have not tried with these exact circumstances, of course.
What about adding back the explicit @ANALYZE@ as another option? I think we need to reconsider that too. It may ultimately mitigate a poor query but, if it mitigates, then it's mitigation.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:07:36Z
Vacuums and analyzes up through noon UTC the same day:
postgresql-0415.log:2021-04-15 19:39:57.502 UTC 10.3.10.102 wres8 wres_user8 60789671.14521 WARNING: skipping "pg_shseclabel" --- only superuser can vacuum it
postgresql-0415.log:2021-04-15 23:25:08.052 UTC 6078c907.1e864 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_33" system usage: CPU: user: 0.37 s, system: 0.64 s, elapsed: 352.73 s
postgresql-0416.log:2021-04-16 00:11:13.189 UTC 6078cdeb.1f857 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_above_150" system usage: CPU: user: 1.35 s, system: 2.58 s, elapsed: 2050.96 s
postgresql-0416.log:2021-04-16 00:23:52.420 UTC 6078c907.1e864 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_15" system usage: CPU: user: 0.45 s, system: 0.75 s, elapsed: 330.43 s
postgresql-0416.log:2021-04-16 00:30:16.021 UTC 6078c907.1e864 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_18" system usage: CPU: user: 0.51 s, system: 0.86 s, elapsed: 383.58 s
postgresql-0416.log:2021-04-16 01:14:15.931 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_21" system usage: CPU: user: 0.58 s, system: 1.02 s, elapsed: 547.23 s
postgresql-0416.log:2021-04-16 01:18:59.028 UTC 6078cdeb.1f857 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_28" system usage: CPU: user: 0.47 s, system: 0.71 s, elapsed: 303.28 s
postgresql-0416.log:2021-04-16 01:24:01.892 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_30" system usage: CPU: user: 0.52 s, system: 1.02 s, elapsed: 585.95 s
postgresql-0416.log:2021-04-16 01:28:03.369 UTC 6078cdeb.1f857 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_24" system usage: CPU: user: 0.49 s, system: 0.97 s, elapsed: 544.33 s
postgresql-0416.log:2021-04-16 01:30:47.537 UTC 6078db2a.1f9b LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_above_150" system usage: CPU: user: 1.41 s, system: 3.28 s, elapsed: 3480.71 s
postgresql-0416.log:2021-04-16 01:48:22.946 UTC 6078db2a.1f9b LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_below_negative_10" system usage: CPU: user: 0.82 s, system: 1.34 s, elapsed: 777.99 s
postgresql-0416.log:2021-04-16 02:08:15.822 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_9" system usage: CPU: user: 0.53 s, system: 0.75 s, elapsed: 476.48 s
postgresql-0416.log:2021-04-16 02:13:58.841 UTC 6078cdeb.1f857 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_3" system usage: CPU: user: 0.52 s, system: 0.84 s, elapsed: 453.70 s
postgresql-0416.log:2021-04-16 02:23:08.342 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_5" system usage: CPU: user: 0.50 s, system: 0.65 s, elapsed: 323.06 s
postgresql-0416.log:2021-04-16 02:29:19.730 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_6" system usage: CPU: user: 0.60 s, system: 0.80 s, elapsed: 371.37 s
postgresql-0416.log:2021-04-16 02:34:27.538 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_7" system usage: CPU: user: 0.60 s, system: 0.70 s, elapsed: 307.79 s
postgresql-0416.log:2021-04-16 02:40:14.603 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_0" system usage: CPU: user: 0.68 s, system: 0.83 s, elapsed: 347.05 s
postgresql-0416.log:2021-04-16 02:45:27.840 UTC 6078dadf.1ee9 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_1" system usage: CPU: user: 0.68 s, system: 0.80 s, elapsed: 313.22 s
postgresql-0416.log:2021-04-16 03:22:48.207 UTC 6078db2a.1f9b LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_9" system usage: CPU: user: 0.68 s, system: 0.86 s, elapsed: 481.74 s
postgresql-0416.log:2021-04-16 03:28:35.836 UTC 6078db2a.1f9b LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_4" system usage: CPU: user: 0.64 s, system: 0.71 s, elapsed: 347.61 s
postgresql-0416.log:2021-04-16 03:50:52.675 UTC 6078db2a.1f9b LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_6" system usage: CPU: user: 0.59 s, system: 1.11 s, elapsed: 803.35 s
postgresql-0416.log:2021-04-16 03:51:53.974 UTC 6078cdeb.1f857 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_15" system usage: CPU: user: 0.40 s, system: 0.63 s, elapsed: 419.23 s
postgresql-0416.log:2021-04-16 03:55:52.566 UTC 6079098f.b674 LOG: automatic analyze of table "wres8.wres.source" system usage: CPU: user: 0.99 s, system: 0.73 s, elapsed: 300.93 s
postgresql-0416.log:2021-04-16 03:59:08.577 UTC 6078cdeb.1f857 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_18" system usage: CPU: user: 0.48 s, system: 0.66 s, elapsed: 434.59 s
postgresql-0416.log:2021-04-16 04:13:09.696 UTC 60790a9f.b9e6 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_33" system usage: CPU: user: 0.49 s, system: 0.75 s, elapsed: 537.60 s
postgresql-0416.log:2021-04-16 04:46:02.104 UTC 60790b8d.bcef LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_below_negative_10" system usage: CPU: user: 0.88 s, system: 1.62 s, elapsed: 2297.84 s
postgresql-0416.log:2021-04-16 05:14:23.665 UTC 6079098f.b674 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_above_150" system usage: CPU: user: 1.26 s, system: 2.42 s, elapsed: 4711.08 s
postgresql-0416.log:2021-04-16 05:18:02.624 UTC 60790b8d.bcef LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_22" system usage: CPU: user: 0.41 s, system: 0.65 s, elapsed: 451.21 s
postgresql-0416.log:2021-04-16 05:18:23.904 UTC 60790a9f.b9e6 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_25" system usage: CPU: user: 0.42 s, system: 0.60 s, elapsed: 416.72 s
postgresql-0416.log:2021-04-16 05:26:22.979 UTC 60790b8d.bcef LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_20" system usage: CPU: user: 0.40 s, system: 0.58 s, elapsed: 500.34 s
postgresql-0416.log:2021-04-16 05:34:31.901 UTC 60790a9f.b9e6 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_35" system usage: CPU: user: 0.39 s, system: 0.62 s, elapsed: 510.87 s
postgresql-0416.log:2021-04-16 05:46:11.639 UTC 60790b8d.bcef LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_27" system usage: CPU: user: 0.53 s, system: 0.87 s, elapsed: 1188.61 s
postgresql-0416.log:2021-04-16 06:26:45.230 UTC 6079098f.b674 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_2" system usage: CPU: user: 0.75 s, system: 1.18 s, elapsed: 1701.12 s
postgresql-0416.log:2021-04-16 06:38:47.936 UTC 60790a9f.b9e6 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_7" system usage: CPU: user: 0.74 s, system: 1.18 s, elapsed: 1621.03 s
postgresql-0416.log:2021-04-16 06:46:58.537 UTC 6079098f.b674 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_15" system usage: CPU: user: 0.49 s, system: 0.89 s, elapsed: 965.12 s
postgresql-0416.log:2021-04-16 06:51:44.239 UTC 60790b8d.bcef LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_3" system usage: CPU: user: 0.81 s, system: 1.51 s, elapsed: 2243.61 s
postgresql-0416.log:2021-04-16 07:04:38.446 UTC 6079098f.b674 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_18" system usage: CPU: user: 0.56 s, system: 0.87 s, elapsed: 1059.89 s
postgresql-0416.log:2021-04-16 07:18:30.889 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.source" system usage: CPU: user: 0.91 s, system: 0.93 s, elapsed: 558.81 s
postgresql-0416.log:2021-04-16 07:59:05.288 UTC 60794227.1858c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_31" system usage: CPU: user: 0.50 s, system: 0.79 s, elapsed: 409.17 s
postgresql-0416.log:2021-04-16 08:10:11.125 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_33" system usage: CPU: user: 0.57 s, system: 0.80 s, elapsed: 521.18 s
postgresql-0416.log:2021-04-16 08:40:08.619 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_above_150" system usage: CPU: user: 1.39 s, system: 2.50 s, elapsed: 4700.71 s
postgresql-0416.log:2021-04-16 08:52:40.992 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_29" system usage: CPU: user: 0.46 s, system: 0.55 s, elapsed: 312.78 s
postgresql-0416.log:2021-04-16 08:58:30.829 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_24" system usage: CPU: user: 0.48 s, system: 0.64 s, elapsed: 473.67 s
postgresql-0416.log:2021-04-16 09:01:26.530 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_21" system usage: CPU: user: 0.47 s, system: 0.68 s, elapsed: 525.52 s
postgresql-0416.log:2021-04-16 09:03:11.571 UTC 60794227.1858c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_30" system usage: CPU: user: 0.45 s, system: 0.68 s, elapsed: 588.04 s
postgresql-0416.log:2021-04-16 09:03:40.634 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_32" system usage: CPU: user: 0.41 s, system: 0.47 s, elapsed: 309.79 s
postgresql-0416.log:2021-04-16 09:38:55.309 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_4" system usage: CPU: user: 0.51 s, system: 0.77 s, elapsed: 654.26 s
postgresql-0416.log:2021-04-16 09:40:31.150 UTC 60794227.1858c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_9" system usage: CPU: user: 0.53 s, system: 0.87 s, elapsed: 763.75 s
postgresql-0416.log:2021-04-16 09:41:33.058 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_8" system usage: CPU: user: 0.49 s, system: 0.75 s, elapsed: 571.91 s
postgresql-0416.log:2021-04-16 09:48:57.642 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_5" system usage: CPU: user: 0.47 s, system: 0.81 s, elapsed: 602.31 s
postgresql-0416.log:2021-04-16 09:52:03.930 UTC 60794227.1858c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_6" system usage: CPU: user: 0.57 s, system: 0.88 s, elapsed: 692.74 s
postgresql-0416.log:2021-04-16 09:52:30.755 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_0" system usage: CPU: user: 0.55 s, system: 0.87 s, elapsed: 657.68 s
postgresql-0416.log:2021-04-16 09:57:42.683 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_19" system usage: CPU: user: 0.32 s, system: 0.49 s, elapsed: 311.91 s
postgresql-0416.log:2021-04-16 09:58:34.288 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_1" system usage: CPU: user: 0.49 s, system: 0.80 s, elapsed: 576.63 s
postgresql-0416.log:2021-04-16 10:04:00.925 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_16" system usage: CPU: user: 0.35 s, system: 0.54 s, elapsed: 378.22 s
postgresql-0416.log:2021-04-16 10:09:55.385 UTC 60794227.1858c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_12" system usage: CPU: user: 0.47 s, system: 0.74 s, elapsed: 616.34 s
postgresql-0416.log:2021-04-16 10:10:00.603 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_14" system usage: CPU: user: 0.39 s, system: 0.59 s, elapsed: 359.59 s
postgresql-0416.log:2021-04-16 10:10:27.251 UTC 6079431a.18898 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_11" system usage: CPU: user: 0.43 s, system: 0.61 s, elapsed: 412.60 s
postgresql-0416.log:2021-04-16 10:16:16.742 UTC 60794227.1858c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_13" system usage: CPU: user: 0.45 s, system: 0.59 s, elapsed: 381.34 s
postgresql-0416.log:2021-04-16 10:17:51.912 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_10" system usage: CPU: user: 0.44 s, system: 0.79 s, elapsed: 471.23 s
postgresql-0416.log:2021-04-16 10:26:26.719 UTC 60793809.1654d LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_17" system usage: CPU: user: 0.46 s, system: 0.60 s, elapsed: 303.77 s
postgresql-0417.log:2021-04-17 08:01:11.477 UTC 607a9443.1ce8c LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_below_negative_10" system usage: CPU: user: 0.71 s, system: 2.41 s, elapsed: 387.70 s
postgresql-0417.log:2021-04-17 10:19:38.989 UTC 607ab2df.2f85 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_27" system usage: CPU: user: 0.43 s, system: 0.98 s, elapsed: 859.42 s
postgresql-0417.log:2021-04-17 11:35:25.642 UTC 607ac560.6930 LOG: automatic analyze of table "wres8.wres.source" system usage: CPU: user: 0.80 s, system: 0.94 s, elapsed: 664.86 s
postgresql-0417.log:2021-04-17 11:35:38.009 UTC 607ab768.3d57 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_above_150" system usage: CPU: user: 0.86 s, system: 1.99 s, elapsed: 4256.27 s
postgresql-0417.log:2021-04-17 12:41:43.926 UTC 607ac560.6930 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_7" system usage: CPU: user: 0.70 s, system: 1.56 s, elapsed: 1955.12 s
Looking at this one, longest one is an hour and 11 minutes, so this started well after that query above, around T10:24Z or so: @postgresql-0417.log:2021-04-17 11:35:38.009 UTC 607ab768.3d57 LOG: automatic analyze of table "wres8.wres.timeseriesvalue_lead_above_150" system usage: CPU: user: 0.86 s, system: 1.99 s, elapsed: 4256.27 s@
What about the plan?
2021-04-16 09:04:55.366 UTC 10.w.v.u wres8 wres_user8 60794470.18c82 LOG: duration: 3725464.856 ms execute <unnamed>/C_1: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
2021-04-16 09:04:55.366 UTC 10.w.v.u wres8 wres_user8 60794470.18c82 DETAIL: parameters: $1 = '28', $2 = 'MAT', $3 = '11073', $4 = 'right', $5 = '27360', $6 = '30240', $7 = '1', $8 = '1', $9 = '12', $10 = '31'
2021-04-16 09:05:02.795 UTC 10.w.v.u wres8 wres_user8 60794470.18c82 LOG: duration: 3728024.525 ms plan:
Query Text: SELECT
MIN(TS.timeseries_id) AS series_id,
TS.initialization_date AS reference_time,
TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
ARRAY_AGG(
TSV.series_value
ORDER BY E.ensemble_name
) AS ensemble_members,
ARRAY_AGG(
E.ensemble_name
ORDER BY E.ensemble_name
) AS ensemble_names,
TS.scale_period,
TS.scale_function,
TS.measurementunit_id
FROM wres.TimeSeries TS
INNER JOIN wres.Ensemble E
ON E.ensemble_id = TS.ensemble_id
INNER JOIN wres.TimeSeriesValue TSV
ON TSV.timeseries_id = TS.timeseries_id
INNER JOIN wres.ProjectSource PS
ON PS.source_id = TS.source_id
WHERE PS.project_id = $1
AND TS.variable_name = $2
AND TS.feature_id = $3
AND PS.member = $4
AND TSV.lead > $5
AND TSV.lead <= $6
AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
GROUP BY TS.initialization_date, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id, TS.source_id
ORDER BY TS.initialization_date, valid_time, series_id
Sort (cost=54.29..54.30 rows=1 width=110)
Sort Key: ts.initialization_date, ((ts.initialization_date + ('00:01:00'::interval minute * (tsv.lead)::double precision))), (min(ts.timeseries_id))
-> GroupAggregate (cost=54.23..54.28 rows=1 width=110)
Group Key: ts.initialization_date, tsv.lead, ts.scale_period, ts.scale_function, ts.measurementunit_id, ts.source_id
-> Sort (cost=54.23..54.24 rows=1 width=51)
Sort Key: ts.initialization_date, tsv.lead, ts.scale_period, ts.scale_function, ts.measurementunit_id, ts.source_id
-> Nested Loop (cost=0.86..54.22 rows=1 width=51)
Join Filter: (ts.ensemble_id = e.ensemble_id)
-> Nested Loop (cost=0.86..52.10 rows=1 width=50)
-> Nested Loop (cost=0.86..17.16 rows=1 width=38)
-> Index Scan using projectsource_project_index on projectsource ps (cost=0.42..6.19 rows=1 width=8)
Index Cond: ((project_id = 28) AND ((member)::text = 'right'::text))
-> Index Scan using timeseries_source_index on timeseries ts (cost=0.43..10.95 rows=1 width=38)
Index Cond: (source_id = ps.source_id)
Filter: (((variable_name)::text = 'MAT'::text) AND (feature_id = '11073'::bigint) AND ((initialization_date)::date >= make_date((date_part('year'::text, initialization_date))::integer, 1, 1)) AND ((initialization_date)::date <= make_date((date_part('year'::text, initialization_date))::integer, 12, 31)))
-> Append (cost=0.00..34.65 rows=29 width=16)
-> Seq Scan on timeseriesvalue tsv (cost=0.00..0.00 rows=1 width=16)
Filter: ((lead > '27360'::bigint) AND (lead <= '30240'::bigint) AND (ts.timeseries_id = timeseries_id))
-> Index Scan using timeseriesvalue_lead_22_timeseries_idx on timeseriesvalue_lead_22 tsv_1 (cost=0.44..8.62 rows=1 width=16)
Index Cond: (timeseries_id = ts.timeseries_id)
Filter: ((lead > '27360'::bigint) AND (lead <= '30240'::bigint))
-> Index Scan using timeseriesvalue_lead_23_timeseries_idx on timeseriesvalue_lead_23 tsv_2 (cost=0.44..8.62 rows=8 width=16)
Index Cond: (timeseries_id = ts.timeseries_id)
Filter: ((lead > '27360'::bigint) AND (lead <= '30240'::bigint))
-> Index Scan using timeseriesvalue_lead_24_timeseries_idx on timeseriesvalue_lead_24 tsv_3 (cost=0.44..8.80 rows=16 width=16)
Index Cond: (timeseries_id = ts.timeseries_id)
Filter: ((lead > '27360'::bigint) AND (lead <= '30240'::bigint))
-> Index Scan using timeseriesvalue_lead_25_timeseries_idx on timeseriesvalue_lead_25 tsv_4 (cost=0.44..8.62 rows=3 width=16)
Index Cond: (timeseries_id = ts.timeseries_id)
Filter: ((lead > '27360'::bigint) AND (lead <= '30240'::bigint))
-> Seq Scan on ensemble e (cost=0.00..1.50 rows=50 width=9)
This query doesn't really touch the "above 150" table, so it is could be due to something else, such as some other instance doing simultaneous ingest (maybe to that partition), rather than the size of that partition involved in this query meaning a slow query.
(Db backup still running)
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:13:52Z
James wrote:
Jesse wrote:
- Mitigate by cleaning after every handful of HEFS BV runs (takes around 0h-1h a day)
Also not clear to me that this is mitigation. If I'm reading #90967-20 correctly, there's very little there and the context is that such timeouts have happened twice early in a release (albeit with different queries) and never later in a release.
It depends on what you mean by "very little" I suppose.
Through WRES 2.0 I don't think WRES ever saw any databases beyond 100GiB, but I might be mis-remembering.
@ wres8 | wres_user8 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 582 GB | pg_default |@
This is 582GiB. It's not a direct comparison because the efficiency of storage in this database has dropped, but not in the area with the bulk of that data:
wres | timeseriesvalue_lead_above_150 | table | wres_user8 | | 140 GB |
wres | timeseriesvalue_lead_above_150_lead_idx | index | wres_user8 | timeseriesvalue_lead_above_150 | 90 GB |
wres | timeseriesvalue_lead_above_150_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_above_150 | 96 GB |
Again, until the partition count was capped, evaluations like these wouldn't even finish ingest before blowing up (edit: not just blowing up that evaluation but the whole system). The irony is that we need the data more evenly spread through partitions, but we cannot have too many partitions because we already have so many.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:16:43Z
Since it's above 500GiB, we cannot even reproduce this issue in the test environment because ITSG didn't feel like giving it to us. I guess we need to pester them again.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:26:10Z
Request made: https://noaa.samanage.com/incidents/70314246-make-wres-database-volume-size-in-ti-match-that-of-prod-1tib?report_id=9003344
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:29:22Z
Jesse wrote:
It depends on what you mean by "very little" I suppose.
Yes, fair point. By comparison, what was the picture at the end of 5.7 before we cleaned? I think we were running similar queries on that instance and those were not timing out, but perhaps 5.8 was filled with more HEFS data in that short period than 5.7 ever had or that the distribution (and statistics) were different in a way that was very material to the query plans.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:31:33Z
(Db backup ongoing)
James wrote:
Jesse wrote:
- Mitigate with a better query (Chris provided one, something in the 8h-32h range I guess)
We can look at that. However, I've tried something very similar to that variation before and I don't recall it helping, but have not tried with these exact circumstances, of course.
What about adding back the explicit @ANALYZE@ as another option? I think we need to reconsider that too. It may ultimately mitigate a poor query but, if it mitigates, then it's mitigation.
In our testing to date, even with fully analyzed data, we saw the queries taking too long.
If the analyze is put back in (in the manner it previously was there), you'll see evaluations that used to take 30 seconds going up to several hours. Yeah, you might also see evaluations that were going to take 10 hours take 5 hours, too, in some limited circumstances. But I don't think there's evidence that the explicit analyze is helpful on the whole. These are datasets WRES has never seen (at least at this scale).
In all it's a deficiency of our testing. Hank's (and my) hunch that HEFS BV could reveal further issues was correct : )
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:31:57Z
James wrote:
Jesse wrote:
It depends on what you mean by "very little" I suppose.
Yes, fair point. By comparison, what was the picture at the end of 5.7 before we cleaned? I think we were running similar queries on that instance and those were not timing out, but perhaps 5.8 was filled with more HEFS data in that short period than 5.7 ever had or that the distribution (and statistics) were different in a way that was very material to the query plans.
Over 900GiB.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:34:05Z
I don't deny that the issue pre-dates 5.8, I'm suggesting that it pre-dates it so much that it will be time-consuming to resolve.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:38:00Z
Jesse wrote:
If the analyze is put back in (in the manner it previously was there), you'll see evaluations that used to take 30 seconds going up to several hours. Yeah, you might also see evaluations that were going to take 10 hours take 5 hours, too, in some limited circumstances. But I don't think there's evidence that the explicit analyze is helpful on the whole. These are datasets WRES has never seen (at least at this scale).
Right, but I think the first order of business is that evaluations complete (and, again, increasing the timeout isn't a good option). I think we should add it to the list of potential mitigations and evaluate it for this specific shape of evaluation, bearing in mind that it does have downsides (which is why we removed it).
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:39:54Z
Jesse wrote:
I don't deny that the issue pre-dates 5.8, I'm suggesting that it pre-dates it so much that it will be time-consuming to resolve.
Oh, I think it will be time-consuming to resolve too. I was just trying to understand the thought process whereby you thought that rolling back to v5.7 was mitigation (it sounded like you had something specific in mind).
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:42:55Z
There might be a handful of things in 5.8 that pushed it over the edge:
There isn't evidence for the last change having any negative effect on ingest performance. Also the increase of those fields from 32 to 64 bits shouldn't have -any- much impact (don't appear to grow the relevant tables much, they are tiny compared to this large partition for example).
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:45:39Z
I also agree that the HEFS was likely to expose things.
As an aside, perhaps 95%+ of the current HEFS evaluations are "one-and-done", so cleaning regularly might be a viable short-term solution for HEFS, if ugly. But we don't have a HEFS WRES instance, rather all users on one instance, so cleaning would disrupt other users with a different pattern. But perhaps this pattern is common to a majority of our users right now.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:47:54Z
Jesse wrote:
There might be a handful of things in 5.8 that pushed it over the edge:
- Increasing some fields from 32-bit to 64-bit integers
- Addition of another @ARRAY_AGG@
- Change to COPY
There isn't evidence for the last change having any negative effect on ingest performance. Also the increase of those fields from 32 to 64 bits shouldn't have -any- much impact (don't appear to grow the relevant tables much, they are tiny compared to this large partition for example).
And, again, I explored (and reported on) the impacts of the additional @ARRAY_AGG@ and saw none, so there would have to be a significant db-size dependency of that addition and, in any case, removing it isn't an option because it's absence is a deal-breaker for HEFS use. But let's include it in the mix of things to investigate so that we can be more specific about the best set of potential mitigations.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:52:09Z
I agree that the changes are minimal, that we can't reasonably expect them to cause a dramatic change, and that in testing we didn't see a dramatic change. We have only these production issues as evidence that something from 5.7 to 5.8 caused the issue.
So it still comes down to our testing approach, one way or another. If something in 5.7 to 5.8 caused it, we would like to see it in our testing. If something to do with the HEFS BV jobs is different, we would like to have and use those jobs in our testing.
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:53:00Z
Gotcha. Yup.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T14:53:10Z
(DB backup still running...)
epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T14:55:04Z
That is to say, I agree completely with the testing deficiency. I also think that "average usage" could've changed dramatically between 5.7 and the early part of 5.8, so in the big picture there are two variables to investigate here (software and usage patterns).
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T15:59:58Z
Database backup still running.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:22:29Z
The very act of taking a db backup impacts production runtimes as well (to what degree is not clear, but one would think it's insignificant).
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:29:43Z
This feels slower than usual (then again it's larger than usual) but for comparison #87437-59 has a 2-hour db dump
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:31:24Z
2 hour db dump resulted in @ 19G Feb 11 21:25 prod_wres8_5.4_20210211.pgdump@ We are currently at twice that @40G Apr 19 16:30 prod_wres8_5.8_20210419.pgdump@ so I guess that is enough of an explanation.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:37:04Z
Done:
[~]$ wres-admin-scripts-20210304-5b7ed93/pg_dump_wres.sh -f /wres_share/dbdumps/prod_wres8_5.8_20210419 -h db-fqdn -d wres8 -U wres_user8 -p /usr/pgsql-10/bin/pg_dump
Creating dump_file /wres_share/dbdumps/prod_wres8_5.8_20210419.pgdump (does not yet exist)
Creating changelog_dump_file /wres_share/dbdumps/prod_wres8_5.8_20210419_changelog.pgdump (does not yet exist)
Using pg_dump executable /usr/pgsql-10/bin/pg_dump (exists) (executable)
Using database host db-fqdn (resolves)
Using database name wres8
Using database username wres_user8
Please ctrl-c if that is not correct, any key otherwise...
2021-04-19T13:16:24,288239999+0000
++ /usr/pgsql-10/bin/pg_dump -h db-fqdn -d wres8 -Fc -U wres_user8 -n wres
++ /usr/pgsql-10/bin/pg_dump -h db-fqdn -d wres8 -Fc -U wres_user8 -n public --table databasechangelog --table databasechangeloglock
++ set +x
2021-04-19T16:32:26,905280368+0000
Dump took around 11762 seconds
-rw-rw-r--. 1 user.name wres 113K Apr 19 16:32 /wres_share/dbdumps/prod_wres8_5.8_20210419_changelog.pgdump
-rw-rw-r--. 1 user.name wres 41G Apr 19 16:32 /wres_share/dbdumps/prod_wres8_5.8_20210419.pgdump
Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:42:20Z
@CLEAN FAILED FOR https://nwcal-wres-prod.[host]/job/8802387268524421152@ As expected when an evaluation is running.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:44:16Z
Two ongoing evaluations.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T16:46:46Z
I would try to restore it locally, but my disk is only 461GiB.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T17:00:53Z
I guess we also need the datasets, will try to get them.
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T17:23:15Z
Got at least one set of data into /home/ISED/wres/wresTestData/issue90967
Clean done:
[redmine_66829]$ ./wres_http_cleandatabase.sh
We are using the -prod environment in this program.
Please ctrl-c if that is not correct, any key otherwise...
Found the WRES CA file at ../cacerts/dod_root_ca_3_expires_2029-12.pem
The last status code in the response was 201
The response code was successful: 201
The location of the resource created by server was https://nwcal-wres-prod.[host]/job/4585948276269780732
Clean succeeded for https://nwcal-wres-prod.[host]/job/4585948276269780732
[redmine_66829]$ date --iso-8601=ns
2021-04-19T17:22:05,346617382+0000
That should give some breathing room until tomorrow at least : )
epag
commented
3 weeks ago Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T17:24:29Z
@ wres8 | wres_user8 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 101 MB | pg_default |@
Author Name: James (James) Original Redmine Issue: 90967, https://vlab.noaa.gov/redmine/issues/90967 Original Date: 2021-04-19
Given an HEFS evaluation in production When that evaluation issues retrieval queries Then the queries should succeed within a reasonable period and should not timeout (as in #90958)
( To be clear, increasing the timeout period is not a solution to this, as it's already very long - PT5H. )
Redmine related issue(s): 93723, 98571