NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a user, I want to avoid bookend database partitions that become overly full leading to slow retrieval #257

Open epag opened 2 months ago

epag commented 2 months ago

Author Name: James (James) Original Redmine Issue: 90967, https://vlab.noaa.gov/redmine/issues/90967 Original Date: 2021-04-19


Given an HEFS evaluation in production When that evaluation issues retrieval queries Then the queries should succeed within a reasonable period and should not timeout (as in #90958)

( To be clear, increasing the timeout period is not a solution to this, as it's already very long - PT5H. )


Redmine related issue(s): 93723, 98571


epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T17:25:38Z


Thanks. Can we ask Jim to resubmit his two jobs or, rather, one of the two and, after it succeeds, the other one?

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T18:27:24Z


Sure

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T18:28:35Z


Thanks, doing that...

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T19:09:31Z


I guess the other issue is 1 TiB in -prod (and shortly in -ti, all being well), right? In other words, we either need a lot more disk in -prod soon or we'll have to start cleaning -prod regularly, regardless of whether it is mitigation for this issue. That is, if our users amassed 500+ GiB in a small number of days. edit: I mean, the HEFS folks are barely getting started. Kinda surprised at how they got to 500+ GiB in that amount of time, based on what I know about these evaluations.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T19:09:50Z


Copied more data:

[issue90967]$ ls -1
MRNN6HUD_MAP.xml
MRNN6HUD_Precipitation_ESP_Baseline_Validation.tgz
MRNN6HUD_Precipitation_HEFS_Baseline_Validation.tgz
NILM4_MAP.xml
NILM4_Precipitation_ESP_Baseline_Validation.tgz
NILM4_Precipitation_HEFS_Baseline_Validation.tgz
[issue90967]$ pwd
/home/ISED/wres/wresTestData/issue90967
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T19:10:39Z


I'm about to look in check_mk, but I expect a kind of logarithmic growth, because a lot of data are repeated/reused. Will post graphs in a moment.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T19:15:14Z


OK, not really logarithmic but there's an early spike followed by linear growth with a handful of bursts upward, that kind of thing. This is a year:

!db_disk_size_20210419.png!

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T19:23:03Z


FYI, one good thing that came out of a HEFS meeting with Ed and Tom earlier today was their being strongly behind making WRES work for HEFS, which means understanding the resources needed and then providing them (instructing ITSG to provide them). Resourcing WRES was their preferred approach, rather than the HEFS folks being delayed or having to reconsider WRES. But, also, if it didn't work, then this would lead to questions about why the WRES cannot scale. So I think that is both a motivation and a consideration w/r to our next priorities (e.g., h2 or similar; edit: and related scaling and performance investigation/work). Right, Hank?

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T19:37:16Z


I think we can just run those evaluations again in -prod to see, before Jim does.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T19:38:31Z


Yeah, if they haven't been submitted, good idea.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T19:42:27Z


One of em as run by me (for our sake, not for Jim's): https://nwcal-wres-prod.[host]/job/2113233765380721901

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T20:00:04Z


With a spot check in the dataset being ingested (untarring, unzipping), I am at a loss to see why so many values are being put in the bucket of "lead duration greater than 125 days" even though I see some will come from @MRNN6HUD_MAP.xml@.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:02:40Z


You mean it doesn't stack up with the shape of the pi sources? Each of these pi sources should have a very similar shape, with varying T0. The forecast horizon for each one should be 30 days, I think, so a majority will be > PT125H. edit: oh wait, you said days, not hours. I thought our last bucket was > PT150H...? Checking.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T20:05:46Z


wres8=> select * from wres.timeseriesvalue_lead_above_150 limit 10;
 timeseries_id |  lead  | series_value
---------------+--------+--------------
      40515957 | 181440 |        0.052
      40515957 | 181800 |            0
      40515957 | 182160 |        0.142
      40515957 | 182520 |         0.03
      40515957 | 182880 |        0.024
      40515957 | 183240 |        0.002
      40515957 | 183600 |            0
      40515957 | 183960 |            0
      40515957 | 184320 |            0
      40515957 | 184680 |            0
(10 rows)

wres8=> select distinct( timeseries_id)  from wres.timeseriesvalue_lead_above_150 limit 10;
Cancel request sent
ERROR:  canceling statement due to user request
wres8=> select * from wres.timeseries where timeseries_id = 40515957;
 timeseries_id | ensemble_id | measurementunit_id | initialization_date | scale_period | scale_function | source_id | variable_name | feature_id
---------------+-------------+--------------------+---------------------+--------------+----------------+-----------+---------------+------------
      40515957 |         145 |                  7 | 1989-01-02 12:00:00 |          360 | UNKNOWN        |   1424262 | MAP           |      19835
(1 row)

wres8=> select * from wres.source where source_id = 1424262;
 source_id |               hash               |        output_time        | lead | is_point_data |                                                   path     
-----------+----------------------------------+---------------------------+------+---------------+-----------------------------------------------------------------------------------------------------------
   1424262 | 754497D945D1BF4240F401E7FE32F9D8 | 2021-04-19 19:40:42.09527 |      | t             | file:/home/ISED/wres/wresTestData/issue90967/clone_nerfc_1989/EVSdata/198901021200_MRNN6HUD_MEFP_FMAP.xml
(1 row)
epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:10:16Z


Hmmm, I guess it is days. Not immediately obvious because we're not using durations in that part of the code base, rather @int@ with unspecified units, but it's minutes and there are 1200 of them per partition, it looks like.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:10:59Z


So perhaps these are not 30-day forecasts, perhaps they are from NYC, which are much, much longer, perhaps 9 months or even longer.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:12:53Z


Jesse wrote:

Copied more data: [...]

Ah, yes, at least some of them. MRNN6HUD is Mount Marion, NY, an NYCDEP location. The NYCDEP forecasts were much, much longer than typical. edit: and we probably did zero testing of that length of forecast. Starting to make more sense...

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T20:13:07Z


Looking in @clone_nerfc_1989/EVSdata/198901021200_MRNN6HUD_MEFP_FMAP.xml@...

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-19T20:14:40Z


Ok, good, that checks out then.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:18:13Z


The good news is that such a shape is extremely atypical for HEFS and the evaluation of them is done. The NYCDEP hindcasts were really a one-off. For the rest of the baseline validation, it will be 30 days. On the other hand, there are probably some other atypical shapes to contend with, like forecasts every PT1H (rather than PT6H) for some RFCs. Still, a forecast horizon of 9 months or more is not a base case and I guess this makes sense if Jim is testing a few things.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:20:04Z


None of that to say that WRES shouldn't handle it, but it takes some pressure off, I think.

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2021-04-19T20:25:47Z


James wrote:

FYI, one good thing that came out of a HEFS meeting with Ed and Tom earlier today was their being strongly behind making WRES work for HEFS, which means understanding the resources needed and then providing them (instructing ITSG to provide them). Resourcing WRES was their preferred approach, rather than the HEFS folks being delayed or having to reconsider WRES. But, also, if it didn't work, then this would lead to questions about why the WRES cannot scale. So I think that is both a motivation and a consideration w/r to our next priorities (e.g., h2 or similar; edit: and related scaling and performance investigation/work). Right, Hank?

Yes, that sounds right to me.

I've made clear to Russ that the issue of scaling up is more than just throwing resources at the WRES. There are questions of scaling up on the database side that would need to be addressed, which is where H2 could come in.

In my mind, this HEFS work has to shoot to the top of the queue until something else even more important comes up (NextGen???). That means figuring out how to stream data and implement it; how to scale up the WRES to allow for more workers/"lanes" (including database side); and how to allow the HEFS team to queue up many jobs at once perhaps using prioritization to allow routine jobs to jump the queue (lower priority since throttling on their end can mitigate it).

Sorry for the slow reply,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:33:39Z


Thanks, Hank. Of those, I think the scaling question is the biggest one and also answers the question of more lanes so that there are fewer queued evaluations (and less time for other users to wait). I think the use of @home@ is sub-optimal, but workable.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-19T20:37:18Z


Regarding the situation with this ticket, I would be tempted to reduce the priority from immediate to urgent. I think the partitioning strategy will need to be rethought (minimally, existing partitions extended or refactored) before we can performantly retrieve forecasts with such long lead durations. I think we can advise the HEFS folks that we're not quite ready for this (NYCDEP) style of forecast and it will take longer to get there. Meanwhile, tickets like #89746 are closer to the base case for HEFS and other use (edit: in the foreseeable future) and will have some similar origins and potential mitigations to this ticket, but also some differences.

edit: actually forget about the partitioning speculation above as it isn't central to the point about getting the priority right for this one.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-20T08:33:38Z


90958-15.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-20T11:29:50Z


Demoting. See #90958-16 - #90958-18. Cleaning works as a mitigation.

Demoting doesn't necessarily mean that we stop work on this issue, but it does reflect the true priority, given that the style of evaluation that caused it is atypical and not anticipated for near-term HEFS evaluations.

If we are going to place this one "on hold" in favor of other priorities (e.g., scaling/h2/other), then we should do the same for #90958, which is currently marked "in progress" too.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-20T13:18:47Z


I'm going to give it at least another day of experimentation:

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-20T13:20:32Z


(Didn't mean to change from Urgent to Immediate: I don't care what you call it)

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-20T13:25:17Z


My run with same data off to the side: https://nwcal-wres-prod.[host]/job/4554626750549746051 @2021-04-19T22:36:51.725+0000 INFO Main The function 'execute' took PT2H56M14.654895S@ Jim's run: https://nwcal-wres-prod.[host]/job/6630411095231087901 @2021-04-19T22:58:11.752+0000 INFO Main The function 'execute' took PT55M33.771894S@

It looks like the prod db is small enough now to restore to -ti, and has similar characteristics to the larger one where the one partition is much larger than shared_buffers and much larger than RAM:

wres8=> \dti+ wres.timeseriesvalue_lead_above_150
                                     List of relations
 Schema |              Name              | Type  |   Owner    | Table | Size  | Description
--------+--------------------------------+-------+------------+-------+-------+-------------
 wres   | timeseriesvalue_lead_above_150 | table | wres_user8 |       | 94 GB |
(1 row)

wres8=> \l+
                                                                     List of databases
   Name    |   Owner    | Encoding |   Collate   |    Ctype    |   Access privileges   |  Size   | Tablespace |                Description
-----------+------------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
 postgres  | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 7855 kB | pg_default | default administrative connection database
 template0 | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 7863 kB | pg_default | unmodifiable empty database
           |            |          |             |             | postgres=CTc/postgres |         |            |
 template1 | postgres   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +| 7863 kB | pg_default | default template for new databases
           |            |          |             |             | postgres=CTc/postgres |         |            |
 wres8     | wres_user8 | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       | 364 GB  | pg_default |
(4 rows)

We don't have to wait on ITSG for this. And we can probably use the off-to-the-side datasets, also can use dump/restore.

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-04-20T13:31:31Z


Sounds good to me.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-20T13:51:07Z


In -ti: https://nwcal-wres-ti.[host]/job/2717372054925793640 Also trying to clean up my local to have enough room to work with pg12.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-20T15:36:42Z


Dump finished, clean finished: https://nwcal-wres-prod.[host]/job/2543748734154572928


2021-04-20T15:34:14.562+0000 INFO Main Process: 9112
2021-04-20T15:34:22.726+0000 INFO Database Analyzing data for efficient execution...
2021-04-20T15:34:24.271+0000 INFO Database Database statistical analysis is now complete.
epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-21T18:38:33Z


See the progress in #61488 on what should help with the datasets-expanding-a-partition-indefinitely issue, as well as the partitions2 branch. It is about the same speed or faster as the current WRES when using postgresql 12. But we need to upgrade in order to proceed because we have 10 in -dev and -ti. The ticket for that is https://noaa.samanage.com/incidents/70473332-install-newer-postgresql-on-wres-db-hosts?report_id=9003344

I'm not seeing the over-90GiB partition in production right now. I am satisfied with the progress having proven that the sky won't fall when using declarative partitioning in later versions of postgresql.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-05-26T20:02:52Z


Saw this in a recent HEFS job:

2021-05-26T19:49:04.348+0000 ERROR Main Operation 'execute' completed unsuccessfully
wres.control.InternalWresException: Could not complete project execution
    at wres.control.Control.accept(Control.java:337)
    at wres.control.Control.apply(Control.java:181)
    at wres.MainFunctions.execute(MainFunctions.java:131)
    at wres.MainFunctions.call(MainFunctions.java:94)
    at wres.Main.main(Main.java:99)
Caused by: wres.control.WresProcessingException: Encountered an error while processing evaluation '_5E-Cvc6FjFQra0MdGaidtWtYhU': 
    at wres.control.ProcessorHelper.processEvaluation(ProcessorHelper.java:234)
    at wres.control.Control.accept(Control.java:320)
    ... 4 common frames omitted
Caused by: wres.control.WresProcessingException: Project failed to complete with the following error: 
    at wres.control.ProcessorHelper.processProjectConfig(ProcessorHelper.java:470)
    at wres.control.ProcessorHelper.processEvaluation(ProcessorHelper.java:173)
    ... 5 common frames omitted
Caused by: java.util.concurrent.CompletionException: wres.control.WresProcessingException: While processing feature FeatureTuple[left=FeatureKey[name=ARTN6GRL,description=,srid=0,wkt=POINT ( -75.92 43.98 0.0 )],right=FeatureKey[name=ARTN6GRL,description=,srid=0,wkt=],baseline=FeatureKey[name=ARTN6GRL,description=,srid=0,wkt=]]
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: wres.control.WresProcessingException: While processing feature FeatureTuple[left=FeatureKey[name=ARTN6GRL,description=,srid=0,wkt=POINT ( -75.92 43.98 0.0 )],right=FeatureKey[name=ARTN6GRL,description=,srid=0,wkt=],baseline=FeatureKey[name=ARTN6GRL,description=,srid=0,wkt=]]
    at wres.control.FeatureProcessor.processFeature(FeatureProcessor.java:341)
    at wres.control.FeatureProcessor.get(FeatureProcessor.java:193)
    at wres.control.FeatureProcessor.get(FeatureProcessor.java:50)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    ... 3 common frames omitted
Caused by: java.util.concurrent.CompletionException: wres.io.retrieval.DataAccessException: Failed to access the time-series data.
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
    ... 3 common frames omitted
Caused by: wres.io.retrieval.DataAccessException: Failed to access the time-series data.
    at wres.io.retrieval.TimeSeriesRetriever.getTimeSeriesFromScript(TimeSeriesRetriever.java:297)
    at wres.io.retrieval.EnsembleForecastRetriever.get(EnsembleForecastRetriever.java:231)
    at wres.io.retrieval.EnsembleForecastRetriever.get(EnsembleForecastRetriever.java:30)
    at wres.io.pooling.PoolSupplier.createPool(PoolSupplier.java:274)
    at wres.io.pooling.PoolSupplier.get(PoolSupplier.java:244)
    at wres.io.pooling.PoolSupplier.get(PoolSupplier.java:80)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
    ... 3 common frames omitted
Caused by: java.sql.SQLException: The Query: 
wres.io.utilities.Query@5ca691cb[script=SELECT 
    TS.source_id AS series_id,
    TS.initialization_date AS reference_time,
    TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
    ARRAY_AGG(TSV.series_value ORDER BY TS.ensemble_id) AS ensemble_members,
    ARRAY_AGG(TS.ensemble_id ORDER BY TS.ensemble_id) AS ensemble_ids,
    TS.scale_period,
    TS.scale_function,
    TS.measurementunit_id,
    COUNT(TS.ensemble_id) / COUNT(DISTINCT TS.ensemble_id) AS occurrences
FROM wres.TimeSeries TS
    INNER JOIN wres.TimeSeriesValue TSV
        ON TSV.timeseries_id = TS.timeseries_id
    INNER JOIN wres.ProjectSource PS
        ON PS.source_id = TS.source_id
WHERE PS.project_id = ?
    AND TS.variable_name = ?
    AND TS.feature_id = ?
    AND PS.member = ?
    AND TSV.lead > ?
    AND TSV.lead <= ?
    AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), ?, ?)
    AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), ?, ?)
GROUP BY reference_time, series_id, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id;
,forceTransaction=false,useCursor=true,parameters={776,MAP,47167,right,37080,39960,1,1,12,31},batchParameters=<null>,sqlStatesToRetry=[],insertedIds=[]]

Failed.
    at wres.io.utilities.Query.call(Query.java:457)
    at wres.io.utilities.Database.buffer(Database.java:810)
    at wres.io.utilities.DataScripter.buffer(DataScripter.java:223)
    at wres.io.retrieval.TimeSeriesRetriever.getTimeSeriesFromScript(TimeSeriesRetriever.java:225)
    ... 9 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: canceling statement due to user request
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
    at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
    at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
    at wres.io.utilities.Query.callWithParameters(Query.java:724)
    at wres.io.utilities.Query.call(Query.java:413)
    ... 12 common frames omitted
2021-05-26T19:49:04.371+0000 INFO MainFunctions 
2021-05-26T19:49:04.371+0000 INFO MainFunctions Forcefully shutting down the application (you may see some errors)...
2021-05-26T19:49:04.371+0000 INFO Operations Forcefully shutting down the IO module...
2021-05-26T19:49:04.372+0000 INFO HikariDataSource HikariPool-1 - Shutdown initiated...
2021-05-26T19:49:04.379+0000 INFO HikariDataSource HikariPool-1 - Shutdown completed.
2021-05-26T19:49:04.380+0000 INFO HikariDataSource HikariPool-2 - Shutdown initiated...
2021-05-26T19:49:04.383+0000 INFO HikariDataSource HikariPool-2 - Shutdown completed.
2021-05-26T19:49:04.383+0000 INFO Operations IO module was forcefully shut down. Abandoned around 0 executor tasks and around 0 database tasks.
Log messages have been written to the file /container_home/wres_logs/wres.log (unless otherwise configured in lib/conf/logback.xml).
2021-05-26T19:49:04.387+0000 INFO Main The function 'execute' took PT6H49M19.491624S
epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2021-05-26T20:11:03Z


An evaluation from #92087, I wonder. I see #92087-97 and nearby, for example, but possibly unrelated.

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-05-26T20:23:22Z


From earlier today:

2021-05-26 18:38:28.493 UTC 10.x.y.z wres8 wres_user8 6xx.xx0 LOG:  duration: 17883340.891 ms  plan:
        Query Text: SELECT
            TS.source_id AS series_id,
            TS.initialization_date AS reference_time,
            TS.initialization_date + INTERVAL '1' MINUTE * TSV.lead AS valid_time,
            ARRAY_AGG(TSV.series_value ORDER BY TS.ensemble_id) AS ensemble_members,
            ARRAY_AGG(TS.ensemble_id ORDER BY TS.ensemble_id) AS ensemble_ids,
            TS.scale_period,
            TS.scale_function,
            TS.measurementunit_id,
            COUNT(TS.ensemble_id) / COUNT(DISTINCT TS.ensemble_id) AS occurrences
        FROM wres.TimeSeries TS
            INNER JOIN wres.TimeSeriesValue TSV
                ON TSV.timeseries_id = TS.timeseries_id
            INNER JOIN wres.ProjectSource PS
                ON PS.source_id = TS.source_id
        WHERE PS.project_id = $1
            AND TS.variable_name = $2
            AND TS.feature_id = $3
            AND PS.member = $4
            AND TSV.lead > $5
            AND TSV.lead <= $6
            AND CAST (TS.initialization_date AS DATE ) >= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $7, $8)
            AND CAST (TS.initialization_date AS DATE ) <= MAKE_DATE( CAST( EXTRACT( YEAR FROM TS.initialization_date ) AS INTEGER ), $9, $10)
        GROUP BY reference_time, series_id, TSV.lead, TS.scale_period, TS.scale_function, TS.measurementunit_id
        GroupAggregate  (cost=33702.24..33702.35 rows=2 width=114)
          Group Key: ts.initialization_date, ts.source_id, tsv.lead, ts.scale_period, ts.scale_function, ts.measurementunit_id
          ->  Sort  (cost=33702.24..33702.25 rows=2 width=46)
                Sort Key: ts.initialization_date, ts.source_id, tsv.lead, ts.scale_period, ts.scale_function, ts.measurementunit_id
                ->  Gather  (cost=1001.00..33702.23 rows=2 width=46)
                      Workers Planned: 1
                      ->  Nested Loop  (cost=1.00..32702.03 rows=1 width=46)
                            ->  Nested Loop  (cost=1.00..13256.81 rows=2 width=38)
                                  ->  Parallel Index Scan using timeseries_feature_index on timeseries ts  (cost=0.56..7721.13 rows=226 width=38)
                                        Index Cond: (feature_id = $3)
                                        Filter: (((variable_name)::text = ($2)::text) AND ((initialization_date)::date >= make_date((date_part('year'::text, initialization_date))::integer, $7, $8)) AND ((initialization_date)::date <= make_date((date_part('year'::text, initialization_date))::integer, $9, $10)))
                                  ->  Index Scan using projectsource_source_index on projectsource ps  (cost=0.43..24.48 rows=1 width=8)
                                        Index Cond: (source_id = ts.source_id)
                                        Filter: ((project_id = $1) AND ((member)::text = ($4)::text))
                            ->  Append  (cost=0.00..9713.78 rows=883 width=16)
                                  ->  Seq Scan on timeseriesvalue tsv  (cost=0.00..0.00 rows=1 width=16)
                                        Filter: ((lead > $5) AND (lead <= $6) AND (ts.timeseries_id = timeseries_id))
                                  ->  Index Scan using timeseriesvalue_lead_1_timeseries_idx on timeseriesvalue_lead_1 tsv_1  (cost=0.57..10.19 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_2_timeseries_idx on timeseriesvalue_lead_2 tsv_2  (cost=0.57..10.19 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_3_timeseries_idx on timeseriesvalue_lead_3 tsv_3  (cost=0.57..11.45 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_4_timeseries_idx on timeseriesvalue_lead_4 tsv_4  (cost=0.57..10.23 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_5_timeseries_idx on timeseriesvalue_lead_5 tsv_5  (cost=0.57..10.06 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_6_timeseries_idx on timeseriesvalue_lead_6 tsv_6  (cost=0.57..11.54 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_7_timeseries_idx on timeseriesvalue_lead_7 tsv_7  (cost=0.57..10.17 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_8_timeseries_idx on timeseriesvalue_lead_8 tsv_8  (cost=0.57..10.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_9_timeseries_idx on timeseriesvalue_lead_9 tsv_9  (cost=0.57..11.14 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_10_timeseries_idx on timeseriesvalue_lead_10 tsv_10  (cost=0.57..9.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_11_timeseries_idx on timeseriesvalue_lead_11 tsv_11  (cost=0.57..9.99 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_12_timeseries_idx on timeseriesvalue_lead_12 tsv_12  (cost=0.57..10.80 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_13_timeseries_idx on timeseriesvalue_lead_13 tsv_13  (cost=0.57..9.74 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_14_timeseries_idx on timeseriesvalue_lead_14 tsv_14  (cost=0.57..9.76 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_15_timeseries_idx on timeseriesvalue_lead_15 tsv_15  (cost=0.57..10.71 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_16_timeseries_idx on timeseriesvalue_lead_16 tsv_16  (cost=0.57..9.60 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_17_timeseries_idx on timeseriesvalue_lead_17 tsv_17  (cost=0.57..9.63 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_18_timeseries_idx on timeseriesvalue_lead_18 tsv_18  (cost=0.57..11.18 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_19_timeseries_idx on timeseriesvalue_lead_19 tsv_19  (cost=0.57..9.67 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_20_timeseries_idx on timeseriesvalue_lead_20 tsv_20  (cost=0.57..9.70 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_21_timeseries_idx on timeseriesvalue_lead_21 tsv_21  (cost=0.57..10.93 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_22_timeseries_idx on timeseriesvalue_lead_22 tsv_22  (cost=0.57..9.70 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_23_timeseries_idx on timeseriesvalue_lead_23 tsv_23  (cost=0.57..9.60 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_24_timeseries_idx on timeseriesvalue_lead_24 tsv_24  (cost=0.57..10.93 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_25_timeseries_idx on timeseriesvalue_lead_25 tsv_25  (cost=0.57..9.56 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_26_timeseries_idx on timeseriesvalue_lead_26 tsv_26  (cost=0.57..9.83 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_27_timeseries_idx on timeseriesvalue_lead_27 tsv_27  (cost=0.57..11.11 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_28_timeseries_idx on timeseriesvalue_lead_28 tsv_28  (cost=0.57..9.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_29_timeseries_idx on timeseriesvalue_lead_29 tsv_29  (cost=0.57..9.92 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_30_timeseries_idx on timeseriesvalue_lead_30 tsv_30  (cost=0.57..10.84 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_31_timeseries_idx on timeseriesvalue_lead_31 tsv_31  (cost=0.57..9.70 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_32_timeseries_idx on timeseriesvalue_lead_32 tsv_32  (cost=0.57..9.67 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_33_timeseries_idx on timeseriesvalue_lead_33 tsv_33  (cost=0.57..10.69 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_34_timeseries_idx on timeseriesvalue_lead_34 tsv_34  (cost=0.57..9.72 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_35_timeseries_idx on timeseriesvalue_lead_35 tsv_35  (cost=0.57..9.72 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_36_timeseries_idx on timeseriesvalue_lead_36 tsv_36  (cost=0.56..8.59 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_37_timeseries_idx on timeseriesvalue_lead_37 tsv_37  (cost=0.29..9.38 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_38_timeseries_idx on timeseriesvalue_lead_38 tsv_38  (cost=0.29..8.99 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_39_timeseries_idx on timeseriesvalue_lead_39 tsv_39  (cost=0.29..9.49 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_40_timeseries_idx on timeseriesvalue_lead_40 tsv_40  (cost=0.29..9.07 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_41_timeseries_idx on timeseriesvalue_lead_41 tsv_41  (cost=0.29..9.16 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_42_timeseries_idx on timeseriesvalue_lead_42 tsv_42  (cost=0.29..9.55 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_43_timeseries_idx on timeseriesvalue_lead_43 tsv_43  (cost=0.29..8.99 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_44_timeseries_idx on timeseriesvalue_lead_44 tsv_44  (cost=0.29..9.45 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_45_timeseries_idx on timeseriesvalue_lead_45 tsv_45  (cost=0.29..9.28 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_46_timeseries_idx on timeseriesvalue_lead_46 tsv_46  (cost=0.29..9.13 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_47_timeseries_idx on timeseriesvalue_lead_47 tsv_47  (cost=0.29..8.99 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_48_timeseries_idx on timeseriesvalue_lead_48 tsv_48  (cost=0.29..9.37 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_49_timeseries_idx on timeseriesvalue_lead_49 tsv_49  (cost=0.29..8.78 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_50_timeseries_idx on timeseriesvalue_lead_50 tsv_50  (cost=0.29..9.15 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_51_timeseries_idx on timeseriesvalue_lead_51 tsv_51  (cost=0.29..9.70 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_52_timeseries_idx on timeseriesvalue_lead_52 tsv_52  (cost=0.29..9.24 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_53_timeseries_idx on timeseriesvalue_lead_53 tsv_53  (cost=0.29..9.18 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_54_timeseries_idx on timeseriesvalue_lead_54 tsv_54  (cost=0.29..9.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_55_timeseries_idx on timeseriesvalue_lead_55 tsv_55  (cost=0.29..9.59 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_56_timeseries_idx on timeseriesvalue_lead_56 tsv_56  (cost=0.29..9.89 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_57_timeseries_idx on timeseriesvalue_lead_57 tsv_57  (cost=0.29..9.36 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_58_timeseries_idx on timeseriesvalue_lead_58 tsv_58  (cost=0.29..9.20 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_59_timeseries_idx on timeseriesvalue_lead_59 tsv_59  (cost=0.29..9.19 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_60_timeseries_idx on timeseriesvalue_lead_60 tsv_60  (cost=0.29..9.35 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_61_timeseries_idx on timeseriesvalue_lead_61 tsv_61  (cost=0.29..9.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_62_timeseries_idx on timeseriesvalue_lead_62 tsv_62  (cost=0.29..8.76 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_63_timeseries_idx on timeseriesvalue_lead_63 tsv_63  (cost=0.29..9.04 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_64_timeseries_idx on timeseriesvalue_lead_64 tsv_64  (cost=0.29..8.73 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_65_timeseries_idx on timeseriesvalue_lead_65 tsv_65  (cost=0.29..9.29 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_66_timeseries_idx on timeseriesvalue_lead_66 tsv_66  (cost=0.29..9.39 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_67_timeseries_idx on timeseriesvalue_lead_67 tsv_67  (cost=0.29..9.27 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_68_timeseries_idx on timeseriesvalue_lead_68 tsv_68  (cost=0.29..9.29 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_69_timeseries_idx on timeseriesvalue_lead_69 tsv_69  (cost=0.29..9.14 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_70_timeseries_idx on timeseriesvalue_lead_70 tsv_70  (cost=0.29..9.85 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_71_timeseries_idx on timeseriesvalue_lead_71 tsv_71  (cost=0.29..10.26 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_72_timeseries_idx on timeseriesvalue_lead_72 tsv_72  (cost=0.29..9.63 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_73_timeseries_idx on timeseriesvalue_lead_73 tsv_73  (cost=0.29..9.06 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_74_timeseries_idx on timeseriesvalue_lead_74 tsv_74  (cost=0.29..9.71 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_75_timeseries_idx on timeseriesvalue_lead_75 tsv_75  (cost=0.29..9.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_76_timeseries_idx on timeseriesvalue_lead_76 tsv_76  (cost=0.29..9.21 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_77_timeseries_idx on timeseriesvalue_lead_77 tsv_77  (cost=0.29..8.98 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_78_timeseries_idx on timeseriesvalue_lead_78 tsv_78  (cost=0.29..9.19 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_79_timeseries_idx on timeseriesvalue_lead_79 tsv_79  (cost=0.29..8.97 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_80_timeseries_idx on timeseriesvalue_lead_80 tsv_80  (cost=0.29..8.75 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_81_timeseries_idx on timeseriesvalue_lead_81 tsv_81  (cost=0.29..11.38 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_82_timeseries_idx on timeseriesvalue_lead_82 tsv_82  (cost=0.29..9.12 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_83_timeseries_idx on timeseriesvalue_lead_83 tsv_83  (cost=0.29..8.84 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_84_timeseries_idx on timeseriesvalue_lead_84 tsv_84  (cost=0.29..9.33 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_85_timeseries_idx on timeseriesvalue_lead_85 tsv_85  (cost=0.29..8.88 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_86_timeseries_idx on timeseriesvalue_lead_86 tsv_86  (cost=0.29..9.43 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_87_timeseries_idx on timeseriesvalue_lead_87 tsv_87  (cost=0.29..9.14 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_88_timeseries_idx on timeseriesvalue_lead_88 tsv_88  (cost=0.29..9.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_89_timeseries_idx on timeseriesvalue_lead_89 tsv_89  (cost=0.29..8.92 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_90_timeseries_idx on timeseriesvalue_lead_90 tsv_90  (cost=0.29..8.83 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_91_timeseries_idx on timeseriesvalue_lead_91 tsv_91  (cost=0.29..9.60 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_92_timeseries_idx on timeseriesvalue_lead_92 tsv_92  (cost=0.29..9.47 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_93_timeseries_idx on timeseriesvalue_lead_93 tsv_93  (cost=0.29..9.37 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_94_timeseries_idx on timeseriesvalue_lead_94 tsv_94  (cost=0.29..10.15 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_95_timeseries_idx on timeseriesvalue_lead_95 tsv_95  (cost=0.29..9.17 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_96_timeseries_idx on timeseriesvalue_lead_96 tsv_96  (cost=0.29..9.62 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_97_timeseries_idx on timeseriesvalue_lead_97 tsv_97  (cost=0.29..9.25 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_98_timeseries_idx on timeseriesvalue_lead_98 tsv_98  (cost=0.29..9.22 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_99_timeseries_idx on timeseriesvalue_lead_99 tsv_99  (cost=0.29..9.99 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_100_timeseries_idx on timeseriesvalue_lead_100 tsv_100  (cost=0.29..9.47 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_101_timeseries_idx on timeseriesvalue_lead_101 tsv_101  (cost=0.29..10.10 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_102_timeseries_idx on timeseriesvalue_lead_102 tsv_102  (cost=0.29..10.84 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_103_timeseries_idx on timeseriesvalue_lead_103 tsv_103  (cost=0.29..9.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_104_timeseries_idx on timeseriesvalue_lead_104 tsv_104  (cost=0.29..9.53 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_105_timeseries_idx on timeseriesvalue_lead_105 tsv_105  (cost=0.29..9.22 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_106_timeseries_idx on timeseriesvalue_lead_106 tsv_106  (cost=0.29..10.63 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_107_timeseries_idx on timeseriesvalue_lead_107 tsv_107  (cost=0.29..9.49 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_108_timeseries_idx on timeseriesvalue_lead_108 tsv_108  (cost=0.29..9.22 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_109_timeseries_idx on timeseriesvalue_lead_109 tsv_109  (cost=0.29..10.72 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_110_timeseries_idx on timeseriesvalue_lead_110 tsv_110  (cost=0.29..8.96 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_111_timeseries_idx on timeseriesvalue_lead_111 tsv_111  (cost=0.29..9.22 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_112_timeseries_idx on timeseriesvalue_lead_112 tsv_112  (cost=0.29..8.84 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_113_timeseries_idx on timeseriesvalue_lead_113 tsv_113  (cost=0.29..8.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_114_timeseries_idx on timeseriesvalue_lead_114 tsv_114  (cost=0.29..8.86 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_115_timeseries_idx on timeseriesvalue_lead_115 tsv_115  (cost=0.29..9.67 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_116_timeseries_idx on timeseriesvalue_lead_116 tsv_116  (cost=0.29..10.28 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_117_timeseries_idx on timeseriesvalue_lead_117 tsv_117  (cost=0.29..9.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_118_timeseries_idx on timeseriesvalue_lead_118 tsv_118  (cost=0.29..9.48 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_119_timeseries_idx on timeseriesvalue_lead_119 tsv_119  (cost=0.29..9.67 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_120_timeseries_idx on timeseriesvalue_lead_120 tsv_120  (cost=0.29..8.75 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_121_timeseries_idx on timeseriesvalue_lead_121 tsv_121  (cost=0.29..9.66 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_122_timeseries_idx on timeseriesvalue_lead_122 tsv_122  (cost=0.29..8.98 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_123_timeseries_idx on timeseriesvalue_lead_123 tsv_123  (cost=0.29..9.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_124_timeseries_idx on timeseriesvalue_lead_124 tsv_124  (cost=0.29..8.89 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_125_timeseries_idx on timeseriesvalue_lead_125 tsv_125  (cost=0.29..9.46 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_126_timeseries_idx on timeseriesvalue_lead_126 tsv_126  (cost=0.29..9.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_127_timeseries_idx on timeseriesvalue_lead_127 tsv_127  (cost=0.29..9.47 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_128_timeseries_idx on timeseriesvalue_lead_128 tsv_128  (cost=0.29..10.15 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_129_timeseries_idx on timeseriesvalue_lead_129 tsv_129  (cost=0.29..9.74 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_130_timeseries_idx on timeseriesvalue_lead_130 tsv_130  (cost=0.29..9.15 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_131_timeseries_idx on timeseriesvalue_lead_131 tsv_131  (cost=0.29..9.13 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_132_timeseries_idx on timeseriesvalue_lead_132 tsv_132  (cost=0.29..9.75 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_133_timeseries_idx on timeseriesvalue_lead_133 tsv_133  (cost=0.29..9.00 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_134_timeseries_idx on timeseriesvalue_lead_134 tsv_134  (cost=0.29..9.99 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_135_timeseries_idx on timeseriesvalue_lead_135 tsv_135  (cost=0.29..9.66 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_136_timeseries_idx on timeseriesvalue_lead_136 tsv_136  (cost=0.29..9.19 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_137_timeseries_idx on timeseriesvalue_lead_137 tsv_137  (cost=0.29..9.07 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_138_timeseries_idx on timeseriesvalue_lead_138 tsv_138  (cost=0.29..9.27 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_139_timeseries_idx on timeseriesvalue_lead_139 tsv_139  (cost=0.29..10.00 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_140_timeseries_idx on timeseriesvalue_lead_140 tsv_140  (cost=0.29..8.98 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_141_timeseries_idx on timeseriesvalue_lead_141 tsv_141  (cost=0.29..9.24 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_142_timeseries_idx on timeseriesvalue_lead_142 tsv_142  (cost=0.29..11.03 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_143_timeseries_idx on timeseriesvalue_lead_143 tsv_143  (cost=0.29..9.08 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_144_timeseries_idx on timeseriesvalue_lead_144 tsv_144  (cost=0.28..7.53 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_145_timeseries_idx on timeseriesvalue_lead_145 tsv_145  (cost=0.15..4.91 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_146_timeseries_idx on timeseriesvalue_lead_146 tsv_146  (cost=0.15..4.91 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_147_timeseries_idx on timeseriesvalue_lead_147 tsv_147  (cost=0.27..6.05 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_148_timeseries_idx on timeseriesvalue_lead_148 tsv_148  (cost=0.15..4.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_149_timeseries_idx on timeseriesvalue_lead_149 tsv_149  (cost=0.15..4.90 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_150_timeseries_idx on timeseriesvalue_lead_150 tsv_150  (cost=0.27..6.05 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_1_timeseries_idx on timeseriesvalue_lead_negative_1 tsv_151  (cost=0.43..10.51 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_2_timeseries_idx on timeseriesvalue_lead_negative_2 tsv_152  (cost=0.43..10.46 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_3_timeseries_idx on timeseriesvalue_lead_negative_3 tsv_153  (cost=0.43..10.46 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_4_timeseries_idx on timeseriesvalue_lead_negative_4 tsv_154  (cost=0.43..10.51 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_5_timeseries_idx on timeseriesvalue_lead_negative_5 tsv_155  (cost=0.43..10.49 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_6_timeseries_idx on timeseriesvalue_lead_negative_6 tsv_156  (cost=0.43..10.51 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_7_timeseries_idx on timeseriesvalue_lead_negative_7 tsv_157  (cost=0.43..10.49 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_8_timeseries_idx on timeseriesvalue_lead_negative_8 tsv_158  (cost=0.43..10.49 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_9_timeseries_idx on timeseriesvalue_lead_negative_9 tsv_159  (cost=0.43..10.51 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_negative_10_timeseries_idx on timeseriesvalue_lead_negative_10 tsv_160  (cost=0.43..10.46 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_0_timeseries_idx on timeseriesvalue_lead_0 tsv_161  (cost=0.57..12.55 rows=1 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_below_negative_10_timeseries_idx on timeseriesvalue_lead_below_negative_10 tsv_162  (cost=0.57..1645.56 rows=143 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))
                                  ->  Index Scan using timeseriesvalue_lead_above_150_timeseries_idx on timeseriesvalue_lead_above_150 tsv_163  (cost=0.43..6540.19 rows=578 width=16)
                                        Index Cond: (timeseries_id = ts.timeseries_id)
                                        Filter: ((lead > $5) AND (lead <= $6))

Note the cost of those last two partitions (not to mention the fact that it's reaching into every dadgum partition).

epag commented 2 months ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-05-26T20:26:37Z


Some more info:

wres8=> \dti+ wres.*
                                                                                                                             List of relations
 Schema |                         Name                          | Type  |   Owner    |                 Table                  |    Size    |                                                          Description
--------+-------------------------------------------------------+-------+------------+----------------------------------------+------------+-------------------------------------------------------------------------------------------------------------------------------
 wres   | ensemble                                              | table | wres       |                                        | 48 kB      |
 wres   | ensemble_ensemble_name_unique_index                   | index | wres       | ensemble                               | 16 kB      |
 wres   | ensemble_pk                                           | index | wres       | ensemble                               | 16 kB      |
 wres   | executionlog                                          | table | wres       |                                        | 89 MB      |
 wres   | feature                                               | table | wres_user8 |                                        | 1008 kB    |
 wres   | feature_name_description_srid_wkt_key                 | index | wres_user8 | feature                                | 968 kB     |
 wres   | feature_pkey                                          | index | wres_user8 | feature                                | 208 kB     |
 wres   | gridprojection                                        | table | wres       |                                        | 8192 bytes |
 wres   | gridprojection_pk                                     | index | wres       | gridprojection                         | 8192 bytes |
 wres   | measurementunit                                       | table | wres       |                                        | 48 kB      |
 wres   | measurementunit_pkey                                  | index | wres       | measurementunit                        | 16 kB      |
 wres   | measurementunit_unit_name_key                         | index | wres       | measurementunit                        | 16 kB      |
 wres   | netcdfcoordinate                                      | table | wres       |                                        | 0 bytes    |
 wres   | netcdfcoordinate_coordinate_idx                       | index | wres       | netcdfcoordinate                       | 8192 bytes |
 wres   | project                                               | table | wres_user8 |                                        | 72 kB      | Each row represents a full project dataset. Each WRES execution uses exactly one. One dataset may be used by many executions.
 wres   | project_hash_key                                      | index | wres_user8 | project                                | 48 kB      |
 wres   | project_pkey                                          | index | wres_user8 | project                                | 16 kB      |
 wres   | projectsource                                         | table | wres_user8 |                                        | 414 MB     | Each row indicates an instance of a source/timeseries to be included in the project dataset for the given member.
 wres   | projectsource_project_index                           | index | wres_user8 | projectsource                          | 309 MB     |
 wres   | projectsource_source_index                            | index | wres_user8 | projectsource                          | 228 MB     |
 wres   | source                                                | table | wres_user8 |                                        | 1194 MB    | For vector data, each row represents a collection of timeseries traces.
 wres   | source_hash_key                                       | index | wres_user8 | source                                 | 267 MB     |
 wres   | source_pkey                                           | index | wres_user8 | source                                 | 78 MB      |
 wres   | sourcecompleted                                       | table | wres       |                                        | 131 MB     |
 wres   | sourcecompleted_source_idx                            | index | wres       | sourcecompleted                        | 78 MB      |
 wres   | timeseries                                            | table | wres_user8 |                                        | 5638 MB    |
 wres   | timeseries_feature_index                              | index | wres_user8 | timeseries                             | 1645 MB    |
 wres   | timeseries_pk                                         | index | wres_user8 | timeseries                             | 1295 MB    |
 wres   | timeseries_source_index                               | index | wres_user8 | timeseries                             | 1812 MB    |
 wres   | timeseriesvalue                                       | table | wres       |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_0                                | table | wres_user8 |                                        | 9807 MB    |
 wres   | timeseriesvalue_lead_0_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_0                 | 6308 MB    |
 wres   | timeseriesvalue_lead_0_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_0                 | 5161 MB    |
 wres   | timeseriesvalue_lead_1                                | table | wres_user8 |                                        | 8009 MB    |
 wres   | timeseriesvalue_lead_10                               | table | wres_user8 |                                        | 7563 MB    |
 wres   | timeseriesvalue_lead_100                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_100_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_100               | 872 kB     |
 wres   | timeseriesvalue_lead_100_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_100               | 672 kB     |
 wres   | timeseriesvalue_lead_101                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_101_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_101               | 880 kB     |
 wres   | timeseriesvalue_lead_101_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_101               | 672 kB     |
 wres   | timeseriesvalue_lead_102                              | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_102_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_102               | 864 kB     |
 wres   | timeseriesvalue_lead_102_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_102               | 672 kB     |
 wres   | timeseriesvalue_lead_103                              | table | wres_user8 |                                        | 1504 kB    |
 wres   | timeseriesvalue_lead_103_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_103               | 888 kB     |
 wres   | timeseriesvalue_lead_103_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_103               | 672 kB     |
 wres   | timeseriesvalue_lead_104                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_104_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_104               | 872 kB     |
 wres   | timeseriesvalue_lead_104_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_104               | 672 kB     |
 wres   | timeseriesvalue_lead_105                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_105_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_105               | 872 kB     |
 wres   | timeseriesvalue_lead_105_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_105               | 672 kB     |
 wres   | timeseriesvalue_lead_106                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_106_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_106               | 848 kB     |
 wres   | timeseriesvalue_lead_106_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_106               | 672 kB     |
 wres   | timeseriesvalue_lead_107                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_107_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_107               | 856 kB     |
 wres   | timeseriesvalue_lead_107_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_107               | 784 kB     |
 wres   | timeseriesvalue_lead_108                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_108_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_108               | 864 kB     |
 wres   | timeseriesvalue_lead_108_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_108               | 672 kB     |
 wres   | timeseriesvalue_lead_109                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_109_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_109               | 848 kB     |
 wres   | timeseriesvalue_lead_109_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_109               | 672 kB     |
 wres   | timeseriesvalue_lead_10_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_10                | 4859 MB    |
 wres   | timeseriesvalue_lead_10_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_10                | 3973 MB    |
 wres   | timeseriesvalue_lead_11                               | table | wres_user8 |                                        | 7428 MB    |
 wres   | timeseriesvalue_lead_110                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_110_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_110               | 856 kB     |
 wres   | timeseriesvalue_lead_110_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_110               | 672 kB     |
 wres   | timeseriesvalue_lead_111                              | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_111_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_111               | 848 kB     |
 wres   | timeseriesvalue_lead_111_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_111               | 672 kB     |
 wres   | timeseriesvalue_lead_112                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_112_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_112               | 848 kB     |
 wres   | timeseriesvalue_lead_112_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_112               | 672 kB     |
 wres   | timeseriesvalue_lead_113                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_113_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_113               | 856 kB     |
 wres   | timeseriesvalue_lead_113_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_113               | 672 kB     |
 wres   | timeseriesvalue_lead_114                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_114_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_114               | 888 kB     |
 wres   | timeseriesvalue_lead_114_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_114               | 664 kB     |
 wres   | timeseriesvalue_lead_115                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_115_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_115               | 856 kB     |
 wres   | timeseriesvalue_lead_115_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_115               | 672 kB     |
 wres   | timeseriesvalue_lead_116                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_116_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_116               | 896 kB     |
 wres   | timeseriesvalue_lead_116_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_116               | 672 kB     |
 wres   | timeseriesvalue_lead_117                              | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_117_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_117               | 856 kB     |
 wres   | timeseriesvalue_lead_117_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_117               | 672 kB     |
 wres   | timeseriesvalue_lead_118                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_118_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_118               | 864 kB     |
 wres   | timeseriesvalue_lead_118_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_118               | 672 kB     |
 wres   | timeseriesvalue_lead_119                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_119_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_119               | 848 kB     |
 wres   | timeseriesvalue_lead_119_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_119               | 672 kB     |
 wres   | timeseriesvalue_lead_11_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_11                | 4772 MB    |
 wres   | timeseriesvalue_lead_11_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_11                | 3906 MB    |
 wres   | timeseriesvalue_lead_12                               | table | wres_user8 |                                        | 9623 MB    |
 wres   | timeseriesvalue_lead_120                              | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_120_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_120               | 848 kB     |
 wres   | timeseriesvalue_lead_120_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_120               | 672 kB     |
 wres   | timeseriesvalue_lead_121                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_121_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_121               | 896 kB     |
 wres   | timeseriesvalue_lead_121_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_121               | 672 kB     |
 wres   | timeseriesvalue_lead_122                              | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_122_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_122               | 872 kB     |
 wres   | timeseriesvalue_lead_122_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_122               | 672 kB     |
 wres   | timeseriesvalue_lead_123                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_123_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_123               | 896 kB     |
 wres   | timeseriesvalue_lead_123_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_123               | 672 kB     |
 wres   | timeseriesvalue_lead_124                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_124_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_124               | 888 kB     |
 wres   | timeseriesvalue_lead_124_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_124               | 672 kB     |
 wres   | timeseriesvalue_lead_125                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_125_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_125               | 848 kB     |
 wres   | timeseriesvalue_lead_125_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_125               | 672 kB     |
 wres   | timeseriesvalue_lead_126                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_126_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_126               | 848 kB     |
 wres   | timeseriesvalue_lead_126_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_126               | 672 kB     |
 wres   | timeseriesvalue_lead_127                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_127_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_127               | 856 kB     |
 wres   | timeseriesvalue_lead_127_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_127               | 672 kB     |
 wres   | timeseriesvalue_lead_128                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_128_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_128               | 888 kB     |
 wres   | timeseriesvalue_lead_128_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_128               | 672 kB     |
 wres   | timeseriesvalue_lead_129                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_129_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_129               | 864 kB     |
 wres   | timeseriesvalue_lead_129_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_129               | 672 kB     |
 wres   | timeseriesvalue_lead_12_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_12                | 6183 MB    |
 wres   | timeseriesvalue_lead_12_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_12                | 5086 MB    |
 wres   | timeseriesvalue_lead_13                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_130                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_130_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_130               | 880 kB     |
 wres   | timeseriesvalue_lead_130_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_130               | 672 kB     |
 wres   | timeseriesvalue_lead_131                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_131_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_131               | 824 kB     |
 wres   | timeseriesvalue_lead_131_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_131               | 672 kB     |
 wres   | timeseriesvalue_lead_132                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_132_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_132               | 856 kB     |
 wres   | timeseriesvalue_lead_132_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_132               | 672 kB     |
 wres   | timeseriesvalue_lead_133                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_133_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_133               | 856 kB     |
 wres   | timeseriesvalue_lead_133_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_133               | 672 kB     |
 wres   | timeseriesvalue_lead_134                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_134_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_134               | 888 kB     |
 wres   | timeseriesvalue_lead_134_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_134               | 672 kB     |
 wres   | timeseriesvalue_lead_135                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_135_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_135               | 880 kB     |
 wres   | timeseriesvalue_lead_135_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_135               | 672 kB     |
 wres   | timeseriesvalue_lead_136                              | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_136_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_136               | 880 kB     |
 wres   | timeseriesvalue_lead_136_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_136               | 672 kB     |
 wres   | timeseriesvalue_lead_137                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_137_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_137               | 864 kB     |
 wres   | timeseriesvalue_lead_137_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_137               | 672 kB     |
 wres   | timeseriesvalue_lead_138                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_138_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_138               | 864 kB     |
 wres   | timeseriesvalue_lead_138_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_138               | 672 kB     |
 wres   | timeseriesvalue_lead_139                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_139_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_139               | 856 kB     |
 wres   | timeseriesvalue_lead_139_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_139               | 664 kB     |
 wres   | timeseriesvalue_lead_13_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_13                | 4623 MB    |
 wres   | timeseriesvalue_lead_13_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_13                | 3790 MB    |
 wres   | timeseriesvalue_lead_14                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_140                              | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_140_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_140               | 872 kB     |
 wres   | timeseriesvalue_lead_140_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_140               | 664 kB     |
 wres   | timeseriesvalue_lead_141                              | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_141_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_141               | 880 kB     |
 wres   | timeseriesvalue_lead_141_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_141               | 672 kB     |
 wres   | timeseriesvalue_lead_142                              | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_142_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_142               | 880 kB     |
 wres   | timeseriesvalue_lead_142_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_142               | 664 kB     |
 wres   | timeseriesvalue_lead_143                              | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_143_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_143               | 872 kB     |
 wres   | timeseriesvalue_lead_143_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_143               | 664 kB     |
 wres   | timeseriesvalue_lead_144                              | table | wres_user8 |                                        | 136 kB     |
 wres   | timeseriesvalue_lead_144_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_144               | 72 kB      |
 wres   | timeseriesvalue_lead_144_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_144               | 64 kB      |
 wres   | timeseriesvalue_lead_145                              | table | wres_user8 |                                        | 48 kB      |
 wres   | timeseriesvalue_lead_145_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_145               | 16 kB      |
 wres   | timeseriesvalue_lead_145_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_145               | 16 kB      |
 wres   | timeseriesvalue_lead_146                              | table | wres_user8 |                                        | 48 kB      |
 wres   | timeseriesvalue_lead_146_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_146               | 16 kB      |
 wres   | timeseriesvalue_lead_146_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_146               | 16 kB      |
 wres   | timeseriesvalue_lead_147                              | table | wres_user8 |                                        | 48 kB      |
 wres   | timeseriesvalue_lead_147_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_147               | 40 kB      |
 wres   | timeseriesvalue_lead_147_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_147               | 32 kB      |
 wres   | timeseriesvalue_lead_148                              | table | wres_user8 |                                        | 48 kB      |
 wres   | timeseriesvalue_lead_148_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_148               | 16 kB      |
 wres   | timeseriesvalue_lead_148_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_148               | 16 kB      |
 wres   | timeseriesvalue_lead_149                              | table | wres_user8 |                                        | 48 kB      |
 wres   | timeseriesvalue_lead_149_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_149               | 16 kB      |
 wres   | timeseriesvalue_lead_149_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_149               | 16 kB      |
 wres   | timeseriesvalue_lead_14_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_14                | 4625 MB    |
 wres   | timeseriesvalue_lead_14_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_14                | 3792 MB    |
 wres   | timeseriesvalue_lead_15                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_150                              | table | wres_user8 |                                        | 48 kB      |
 wres   | timeseriesvalue_lead_150_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_150               | 40 kB      |
 wres   | timeseriesvalue_lead_150_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_150               | 32 kB      |
 wres   | timeseriesvalue_lead_15_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_15                | 6163 MB    |
 wres   | timeseriesvalue_lead_15_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_15                | 5073 MB    |
 wres   | timeseriesvalue_lead_16                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_16_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_16                | 4624 MB    |
 wres   | timeseriesvalue_lead_16_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_16                | 3793 MB    |
 wres   | timeseriesvalue_lead_17                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_17_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_17                | 4623 MB    |
 wres   | timeseriesvalue_lead_17_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_17                | 3792 MB    |
 wres   | timeseriesvalue_lead_18                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_18_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_18                | 6163 MB    |
 wres   | timeseriesvalue_lead_18_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_18                | 5076 MB    |
 wres   | timeseriesvalue_lead_19                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_19_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_19                | 4624 MB    |
 wres   | timeseriesvalue_lead_19_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_19                | 3796 MB    |
 wres   | timeseriesvalue_lead_1_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_1                 | 5150 MB    |
 wres   | timeseriesvalue_lead_1_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_1                 | 4190 MB    |
 wres   | timeseriesvalue_lead_2                                | table | wres_user8 |                                        | 7911 MB    |
 wres   | timeseriesvalue_lead_20                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_20_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_20                | 4622 MB    |
 wres   | timeseriesvalue_lead_20_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_20                | 3797 MB    |
 wres   | timeseriesvalue_lead_21                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_21_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_21                | 6162 MB    |
 wres   | timeseriesvalue_lead_21_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_21                | 5078 MB    |
 wres   | timeseriesvalue_lead_22                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_22_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_22                | 4625 MB    |
 wres   | timeseriesvalue_lead_22_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_22                | 3798 MB    |
 wres   | timeseriesvalue_lead_23                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_23_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_23                | 4623 MB    |
 wres   | timeseriesvalue_lead_23_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_23                | 3800 MB    |
 wres   | timeseriesvalue_lead_24                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_24_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_24                | 6164 MB    |
 wres   | timeseriesvalue_lead_24_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_24                | 5082 MB    |
 wres   | timeseriesvalue_lead_25                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_25_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_25                | 4624 MB    |
 wres   | timeseriesvalue_lead_25_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_25                | 3794 MB    |
 wres   | timeseriesvalue_lead_26                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_26_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_26                | 4626 MB    |
 wres   | timeseriesvalue_lead_26_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_26                | 3802 MB    |
 wres   | timeseriesvalue_lead_27                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_27_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_27                | 6167 MB    |
 wres   | timeseriesvalue_lead_27_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_27                | 5084 MB    |
 wres   | timeseriesvalue_lead_28                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_28_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_28                | 4625 MB    |
 wres   | timeseriesvalue_lead_28_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_28                | 3802 MB    |
 wres   | timeseriesvalue_lead_29                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_29_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_29                | 4623 MB    |
 wres   | timeseriesvalue_lead_29_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_29                | 3796 MB    |
 wres   | timeseriesvalue_lead_2_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_2                 | 5085 MB    |
 wres   | timeseriesvalue_lead_2_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_2                 | 4141 MB    |
 wres   | timeseriesvalue_lead_3                                | table | wres_user8 |                                        | 10 GB      |
 wres   | timeseriesvalue_lead_30                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_30_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_30                | 6165 MB    |
 wres   | timeseriesvalue_lead_30_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_30                | 5084 MB    |
 wres   | timeseriesvalue_lead_31                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_31_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_31                | 4623 MB    |
 wres   | timeseriesvalue_lead_31_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_31                | 3798 MB    |
 wres   | timeseriesvalue_lead_32                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_32_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_32                | 4624 MB    |
 wres   | timeseriesvalue_lead_32_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_32                | 3799 MB    |
 wres   | timeseriesvalue_lead_33                               | table | wres_user8 |                                        | 9591 MB    |
 wres   | timeseriesvalue_lead_33_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_33                | 6162 MB    |
 wres   | timeseriesvalue_lead_33_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_33                | 5089 MB    |
 wres   | timeseriesvalue_lead_34                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_34_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_34                | 4623 MB    |
 wres   | timeseriesvalue_lead_34_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_34                | 3802 MB    |
 wres   | timeseriesvalue_lead_35                               | table | wres_user8 |                                        | 7195 MB    |
 wres   | timeseriesvalue_lead_35_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_35                | 4624 MB    |
 wres   | timeseriesvalue_lead_35_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_35                | 3801 MB    |
 wres   | timeseriesvalue_lead_36                               | table | wres_user8 |                                        | 2403 MB    |
 wres   | timeseriesvalue_lead_36_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_36                | 1542 MB    |
 wres   | timeseriesvalue_lead_36_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_36                | 1235 MB    |
 wres   | timeseriesvalue_lead_37                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_37_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_37                | 856 kB     |
 wres   | timeseriesvalue_lead_37_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_37                | 672 kB     |
 wres   | timeseriesvalue_lead_38                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_38_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_38                | 872 kB     |
 wres   | timeseriesvalue_lead_38_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_38                | 672 kB     |
 wres   | timeseriesvalue_lead_39                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_39_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_39                | 904 kB     |
 wres   | timeseriesvalue_lead_39_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_39                | 664 kB     |
 wres   | timeseriesvalue_lead_3_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_3                 | 6662 MB    |
 wres   | timeseriesvalue_lead_3_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_3                 | 5452 MB    |
 wres   | timeseriesvalue_lead_4                                | table | wres_user8 |                                        | 7866 MB    |
 wres   | timeseriesvalue_lead_40                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_40_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_40                | 912 kB     |
 wres   | timeseriesvalue_lead_40_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_40                | 672 kB     |
 wres   | timeseriesvalue_lead_41                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_41_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_41                | 856 kB     |
 wres   | timeseriesvalue_lead_41_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_41                | 672 kB     |
 wres   | timeseriesvalue_lead_42                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_42_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_42                | 864 kB     |
 wres   | timeseriesvalue_lead_42_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_42                | 672 kB     |
 wres   | timeseriesvalue_lead_43                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_43_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_43                | 872 kB     |
 wres   | timeseriesvalue_lead_43_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_43                | 672 kB     |
 wres   | timeseriesvalue_lead_44                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_44_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_44                | 880 kB     |
 wres   | timeseriesvalue_lead_44_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_44                | 672 kB     |
 wres   | timeseriesvalue_lead_45                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_45_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_45                | 864 kB     |
 wres   | timeseriesvalue_lead_45_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_45                | 672 kB     |
 wres   | timeseriesvalue_lead_46                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_46_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_46                | 896 kB     |
 wres   | timeseriesvalue_lead_46_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_46                | 672 kB     |
 wres   | timeseriesvalue_lead_47                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_47_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_47                | 872 kB     |
 wres   | timeseriesvalue_lead_47_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_47                | 672 kB     |
 wres   | timeseriesvalue_lead_48                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_48_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_48                | 880 kB     |
 wres   | timeseriesvalue_lead_48_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_48                | 672 kB     |
 wres   | timeseriesvalue_lead_49                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_49_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_49                | 888 kB     |
 wres   | timeseriesvalue_lead_49_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_49                | 672 kB     |
 wres   | timeseriesvalue_lead_4_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_4                 | 5055 MB    |
 wres   | timeseriesvalue_lead_4_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_4                 | 4120 MB    |
 wres   | timeseriesvalue_lead_5                                | table | wres_user8 |                                        | 7782 MB    |
 wres   | timeseriesvalue_lead_50                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_50_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_50                | 864 kB     |
 wres   | timeseriesvalue_lead_50_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_50                | 672 kB     |
 wres   | timeseriesvalue_lead_51                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_51_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_51                | 856 kB     |
 wres   | timeseriesvalue_lead_51_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_51                | 672 kB     |
 wres   | timeseriesvalue_lead_52                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_52_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_52                | 840 kB     |
 wres   | timeseriesvalue_lead_52_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_52                | 664 kB     |
 wres   | timeseriesvalue_lead_53                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_53_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_53                | 856 kB     |
 wres   | timeseriesvalue_lead_53_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_53                | 664 kB     |
 wres   | timeseriesvalue_lead_54                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_54_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_54                | 872 kB     |
 wres   | timeseriesvalue_lead_54_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_54                | 664 kB     |
 wres   | timeseriesvalue_lead_55                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_55_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_55                | 888 kB     |
 wres   | timeseriesvalue_lead_55_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_55                | 672 kB     |
 wres   | timeseriesvalue_lead_56                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_56_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_56                | 888 kB     |
 wres   | timeseriesvalue_lead_56_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_56                | 672 kB     |
 wres   | timeseriesvalue_lead_57                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_57_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_57                | 872 kB     |
 wres   | timeseriesvalue_lead_57_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_57                | 672 kB     |
 wres   | timeseriesvalue_lead_58                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_58_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_58                | 872 kB     |
 wres   | timeseriesvalue_lead_58_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_58                | 672 kB     |
 wres   | timeseriesvalue_lead_59                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_59_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_59                | 864 kB     |
 wres   | timeseriesvalue_lead_59_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_59                | 672 kB     |
 wres   | timeseriesvalue_lead_5_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_5                 | 5003 MB    |
 wres   | timeseriesvalue_lead_5_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_5                 | 4077 MB    |
 wres   | timeseriesvalue_lead_6                                | table | wres_user8 |                                        | 10235 MB   |
 wres   | timeseriesvalue_lead_60                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_60_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_60                | 888 kB     |
 wres   | timeseriesvalue_lead_60_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_60                | 672 kB     |
 wres   | timeseriesvalue_lead_61                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_61_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_61                | 872 kB     |
 wres   | timeseriesvalue_lead_61_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_61                | 672 kB     |
 wres   | timeseriesvalue_lead_62                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_62_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_62                | 840 kB     |
 wres   | timeseriesvalue_lead_62_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_62                | 672 kB     |
 wres   | timeseriesvalue_lead_63                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_63_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_63                | 816 kB     |
 wres   | timeseriesvalue_lead_63_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_63                | 672 kB     |
 wres   | timeseriesvalue_lead_64                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_64_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_64                | 872 kB     |
 wres   | timeseriesvalue_lead_64_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_64                | 672 kB     |
 wres   | timeseriesvalue_lead_65                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_65_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_65                | 856 kB     |
 wres   | timeseriesvalue_lead_65_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_65                | 672 kB     |
 wres   | timeseriesvalue_lead_66                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_66_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_66                | 872 kB     |
 wres   | timeseriesvalue_lead_66_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_66                | 672 kB     |
 wres   | timeseriesvalue_lead_67                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_67_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_67                | 872 kB     |
 wres   | timeseriesvalue_lead_67_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_67                | 672 kB     |
 wres   | timeseriesvalue_lead_68                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_68_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_68                | 872 kB     |
 wres   | timeseriesvalue_lead_68_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_68                | 672 kB     |
 wres   | timeseriesvalue_lead_69                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_69_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_69                | 912 kB     |
 wres   | timeseriesvalue_lead_69_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_69                | 672 kB     |
 wres   | timeseriesvalue_lead_6_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_6                 | 6577 MB    |
 wres   | timeseriesvalue_lead_6_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_6                 | 5390 MB    |
 wres   | timeseriesvalue_lead_7                                | table | wres_user8 |                                        | 7839 MB    |
 wres   | timeseriesvalue_lead_70                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_70_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_70                | 864 kB     |
 wres   | timeseriesvalue_lead_70_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_70                | 672 kB     |
 wres   | timeseriesvalue_lead_71                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_71_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_71                | 832 kB     |
 wres   | timeseriesvalue_lead_71_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_71                | 672 kB     |
 wres   | timeseriesvalue_lead_72                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_72_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_72                | 872 kB     |
 wres   | timeseriesvalue_lead_72_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_72                | 672 kB     |
 wres   | timeseriesvalue_lead_73                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_73_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_73                | 864 kB     |
 wres   | timeseriesvalue_lead_73_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_73                | 672 kB     |
 wres   | timeseriesvalue_lead_74                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_74_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_74                | 888 kB     |
 wres   | timeseriesvalue_lead_74_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_74                | 672 kB     |
 wres   | timeseriesvalue_lead_75                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_75_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_75                | 864 kB     |
 wres   | timeseriesvalue_lead_75_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_75                | 672 kB     |
 wres   | timeseriesvalue_lead_76                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_76_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_76                | 832 kB     |
 wres   | timeseriesvalue_lead_76_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_76                | 672 kB     |
 wres   | timeseriesvalue_lead_77                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_77_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_77                | 856 kB     |
 wres   | timeseriesvalue_lead_77_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_77                | 672 kB     |
 wres   | timeseriesvalue_lead_78                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_78_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_78                | 840 kB     |
 wres   | timeseriesvalue_lead_78_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_78                | 672 kB     |
 wres   | timeseriesvalue_lead_79                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_79_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_79                | 856 kB     |
 wres   | timeseriesvalue_lead_79_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_79                | 672 kB     |
 wres   | timeseriesvalue_lead_7_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_7                 | 5038 MB    |
 wres   | timeseriesvalue_lead_7_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_7                 | 4107 MB    |
 wres   | timeseriesvalue_lead_8                                | table | wres_user8 |                                        | 7751 MB    |
 wres   | timeseriesvalue_lead_80                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_80_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_80                | 888 kB     |
 wres   | timeseriesvalue_lead_80_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_80                | 672 kB     |
 wres   | timeseriesvalue_lead_81                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_81_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_81                | 880 kB     |
 wres   | timeseriesvalue_lead_81_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_81                | 672 kB     |
 wres   | timeseriesvalue_lead_82                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_82_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_82                | 880 kB     |
 wres   | timeseriesvalue_lead_82_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_82                | 672 kB     |
 wres   | timeseriesvalue_lead_83                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_83_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_83                | 888 kB     |
 wres   | timeseriesvalue_lead_83_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_83                | 672 kB     |
 wres   | timeseriesvalue_lead_84                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_84_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_84                | 832 kB     |
 wres   | timeseriesvalue_lead_84_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_84                | 672 kB     |
 wres   | timeseriesvalue_lead_85                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_85_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_85                | 904 kB     |
 wres   | timeseriesvalue_lead_85_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_85                | 672 kB     |
 wres   | timeseriesvalue_lead_86                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_86_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_86                | 840 kB     |
 wres   | timeseriesvalue_lead_86_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_86                | 672 kB     |
 wres   | timeseriesvalue_lead_87                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_87_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_87                | 856 kB     |
 wres   | timeseriesvalue_lead_87_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_87                | 672 kB     |
 wres   | timeseriesvalue_lead_88                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_88_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_88                | 872 kB     |
 wres   | timeseriesvalue_lead_88_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_88                | 672 kB     |
 wres   | timeseriesvalue_lead_89                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_89_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_89                | 872 kB     |
 wres   | timeseriesvalue_lead_89_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_89                | 672 kB     |
 wres   | timeseriesvalue_lead_8_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_8                 | 4978 MB    |
 wres   | timeseriesvalue_lead_8_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_8                 | 4065 MB    |
 wres   | timeseriesvalue_lead_9                                | table | wres_user8 |                                        | 10222 MB   |
 wres   | timeseriesvalue_lead_90                               | table | wres_user8 |                                        | 1352 kB    |
 wres   | timeseriesvalue_lead_90_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_90                | 856 kB     |
 wres   | timeseriesvalue_lead_90_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_90                | 672 kB     |
 wres   | timeseriesvalue_lead_91                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_91_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_91                | 840 kB     |
 wres   | timeseriesvalue_lead_91_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_91                | 672 kB     |
 wres   | timeseriesvalue_lead_92                               | table | wres_user8 |                                        | 1328 kB    |
 wres   | timeseriesvalue_lead_92_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_92                | 856 kB     |
 wres   | timeseriesvalue_lead_92_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_92                | 672 kB     |
 wres   | timeseriesvalue_lead_93                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_93_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_93                | 840 kB     |
 wres   | timeseriesvalue_lead_93_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_93                | 672 kB     |
 wres   | timeseriesvalue_lead_94                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_94_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_94                | 864 kB     |
 wres   | timeseriesvalue_lead_94_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_94                | 672 kB     |
 wres   | timeseriesvalue_lead_95                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_95_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_95                | 864 kB     |
 wres   | timeseriesvalue_lead_95_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_95                | 672 kB     |
 wres   | timeseriesvalue_lead_96                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_96_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_96                | 880 kB     |
 wres   | timeseriesvalue_lead_96_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_96                | 672 kB     |
 wres   | timeseriesvalue_lead_97                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_97_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_97                | 848 kB     |
 wres   | timeseriesvalue_lead_97_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_97                | 672 kB     |
 wres   | timeseriesvalue_lead_98                               | table | wres_user8 |                                        | 1344 kB    |
 wres   | timeseriesvalue_lead_98_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_98                | 872 kB     |
 wres   | timeseriesvalue_lead_98_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_98                | 672 kB     |
 wres   | timeseriesvalue_lead_99                               | table | wres_user8 |                                        | 1336 kB    |
 wres   | timeseriesvalue_lead_99_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_99                | 864 kB     |
 wres   | timeseriesvalue_lead_99_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_99                | 672 kB     |
 wres   | timeseriesvalue_lead_9_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_9                 | 6569 MB    |
 wres   | timeseriesvalue_lead_9_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_9                 | 5392 MB    |
 wres   | timeseriesvalue_lead_above_150                        | table | wres_user8 |                                        | 713 MB     |
 wres   | timeseriesvalue_lead_above_150_lead_idx               | index | wres_user8 | timeseriesvalue_lead_above_150         | 471 MB     |
 wres   | timeseriesvalue_lead_above_150_timeseries_idx         | index | wres_user8 | timeseriesvalue_lead_above_150         | 456 MB     |
 wres   | timeseriesvalue_lead_below_negative_10                | table | wres_user8 |                                        | 22 GB      |
 wres   | timeseriesvalue_lead_below_negative_10_lead_idx       | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 14 GB      |
 wres   | timeseriesvalue_lead_below_negative_10_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 14 GB      |
 wres   | timeseriesvalue_lead_negative_1                       | table | wres_user8 |                                        | 231 MB     |
 wres   | timeseriesvalue_lead_negative_10                      | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_10_lead_idx             | index | wres_user8 | timeseriesvalue_lead_negative_10       | 148 MB     |
 wres   | timeseriesvalue_lead_negative_10_timeseries_idx       | index | wres_user8 | timeseriesvalue_lead_negative_10       | 118 MB     |
 wres   | timeseriesvalue_lead_negative_1_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_1        | 149 MB     |
 wres   | timeseriesvalue_lead_negative_1_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_1        | 119 MB     |
 wres   | timeseriesvalue_lead_negative_2                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_2_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_2        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_2_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_2        | 119 MB     |
 wres   | timeseriesvalue_lead_negative_3                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_3_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_3        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_3_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_3        | 119 MB     |
 wres   | timeseriesvalue_lead_negative_4                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_4_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_4        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_4_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_4        | 118 MB     |
 wres   | timeseriesvalue_lead_negative_5                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_5_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_5        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_5_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_5        | 119 MB     |
 wres   | timeseriesvalue_lead_negative_6                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_6_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_6        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_6_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_6        | 118 MB     |
 wres   | timeseriesvalue_lead_negative_7                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_7_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_7        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_7_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_7        | 118 MB     |
 wres   | timeseriesvalue_lead_negative_8                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_8_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_8        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_8_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_8        | 118 MB     |
 wres   | timeseriesvalue_lead_negative_9                       | table | wres_user8 |                                        | 230 MB     |
 wres   | timeseriesvalue_lead_negative_9_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_9        | 148 MB     |
 wres   | timeseriesvalue_lead_negative_9_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_9        | 118 MB     |
 wres   | timeseriesvalue_timeseries_idx                        | index | wres       | timeseriesvalue                        | 8192 bytes |
 wres   | unitconversion                                        | table | wres       |                                        | 72 kB      |
 wres   | unitconversion_measurement_uix                        | index | wres       | unitconversion                         | 40 kB      |

!proddb_past_25_hrs_disk_latency_20210526T15-0600.png! !proddb_past_25_hrs_disk_throughput_20210526T15-0600.png! !proddb_past_25_hrs_disk_io_wait_time_20210526T15-0600.png! !proddb_past_25_hrs_cpu_usage_20210526T15-0600.png! !proddb_past_25_hrs_20210526T15-0600.png! !proddb_past_25_hrs_disk_size_20210526T15-0600.png! !proddb_past_25_hrs_page_faults_20210526T15-0600.png! !proddb_past_25_hrs_virtual_mem_overview_20210526T15-0600.png! !proddb_past_25_hrs_disk_q_length_20210526T15-0600.png!

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-16T13:06:30Z


Can we close this ticket? We haven't had an issue with HEFS evaluations in long time. Whatever was causing the query timeouts has likely been corrected or sufficiently mitigated that I think we should just add a new ticket if it happens again.

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2023-02-16T13:26:12Z


No, I don't think so. The problem has not gone away. The underlying problem is partitions, particularly bookend partitions, that contain too much data. We don't see that very often because the shapes of evaluation that exploit this problem are rare. But anything with long-range forecasts will fit into that category and dump a ton of time-series into the top partition by lead duration. We backed out the first cut at a structural solution (declarative partitioning by time-series rather than lead duration) because it was non-performant for the majority of retrievals that are filtered by lead duration. Of course, we can always mitigate with extra partitions, but there will always be a longer forecast...

epag commented 2 months ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2023-02-16T13:49:58Z


Oh, got it. I saw the description, saw it was on-hold, and questioned why when HEFS evaluations have been running fine for so long. But this ticket was the one tracking development related to to partitions. I do remember that issue and I know that it remains a concern.

Thanks for the reminder,

Hank

epag commented 2 months ago

Original Redmine Comment Author Name: James (James) Original Date: 2024-06-25T10:24:26Z


Retaining on the backlog as this is a very precise performance issue related to bookend partitions that fill up.