NOAA-OWP / wres

Code and scripts for the Water Resources Evaluation Service
Other
2 stars 1 forks source link

As a user, I do not want to suffer a significant slowdown when sources from many projects are all loaded at once #179

Open epag opened 3 weeks ago

epag commented 3 weeks ago

Author Name: Chris (Chris) Original Redmine Issue: 50327, https://vlab.noaa.gov/redmine/issues/50327 Original Date: 2018-05-11


There are reports of a significant slowdown when sources across many projects are all loaded at once. Measures need to be discovered and implemented so that database clean up isn't necessary (or as necessary).

Since databases are rarely loaded with many projects at once, this is not a problem that is frequently encountered. As a result, useful partitioning, clustering, and statistical strategies cannot be developed. Many projects will need to be loaded to do so.

This issue may not be as extreme as it sounds. Take scenario003 for instance. My clean execution runs for scenario003 that include an ingest take around 51.6s. Executions afterwards with only scenario003 data generally takes around 25.8s. When the system is loaded with scenario1001, 1000, 703, 702, 701, 700, 601, 600, 409, 405, 404, 403, 402, 200, 104, 100, 012, 008, and 007, the first run of 003 takes around 26.678 seconds. When that data set is first loaded (012 is the first scenario to use that data set) with 1001, 1000, 703, 702, 701, 700, 601, 600, 409, 405, 404, 403, 402, 200, 104, and 100 already loaded, it takes ~34.045 seconds to ingest and execute. Executions and ingests of 012 on clean databases generally take within that same ballpark. I have several of these cases taking between 32 and 34 seconds.

Attached is a screenshot of several runs of the application prior to asynchronization improvements. They were performed on the EDS server, in the WRES database instance.


Related issue(s): #216 Redmine related issue(s): 40271, 56944, 65122, 65903, 66354, 66665, 115356


epag commented 3 weeks ago

Original Redmine Comment Author Name: Chris (Chris) Original Date: 2018-05-11T13:37:19Z


Attached are two files containing explanations of the tables involved in queries and many examples of the queries themselves.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2018-06-11T14:54:45Z


I would call this urgent (feel free to downgrade if you disagree, just using the priority to express my opinion of the priority as we discussed in the 2018-06-07 retrospective).

The use of a database is not really a "cache", it is a means of organizing disparate source models into a canonical internal-to-WRES data model for single-path retrieval of data when doing evaluation.

Relational database management systems (RDBMSes) are purpose-built for organizing and sharing data. So it is a shame to not take advantage of the benefits of all the decades of optimization that the postgres people have built into postgres.

Aside: there is unnecessary hand-wringing over data duplication at OWP. Data duplication is not a crime against humanity. Data is duplicated every time data moves from disk to memory to CPU caches. Data is duplicated in production systems all the time on purpose for backup or failover or high availability. Data is also duplicated in internal data models of software. Data is also duplicated in caches. Data duplication can actually benefit humanity in that can help development time (whether starting development or maintaining existing software) and/or raw runtime performance of software (think nginx or varnish performance increases). Some kinds of duplication do become an issue when we run out of storage space and find that we stored the exact same file in 20 storage systems when we could have stored it in 2 or 3 (again as backup) or when we find that we have stored the exact same data in the exact same format twice within the same system. That is the kind of duplication that perhaps we can worry about. But there is plenty of duplication that is beneficial. End of aside.

I think we should take advantage of the performant concurrent-reads-on-larger-than-RAM-data that RDBMSes offer. One way to do that is to make sure that we can store many large projects together in our database instance and make sure that we are leaning toward optimization for read performance over ingest performance.

Another aside: too much complaint early on about ingest performance happened, with collateral damage to the software as the result. For example: no referential integrity (another thing databases help with, ensuring data invariants hold), suspension of indices on ingest (making multiple simultaneous users difficult if not impossible), and I don't know what else.

If we want the benefits of a database, let's really commit to making multiple users be able to run multiple projects of multiple sizes simultaneously on one database instance with acceptable execution time performance. That's what databases are for.

(Also, if we want the benefits of a canonical internal data model, let's really commit to making gridded data get into our database.)

epag commented 3 weeks ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-06-12T12:56:14Z


Jesse,

I have no problems leaving this as urgent. My tentative plan is for 1.2 and 1.3 to be primarily (not solely) focused on getting the repave the existing road instead of trying to pave new road. The database is definitely part of that existing road.

As for the duplication of data, resources is the primary reason I am leery of it, with performance being secondary (but still a major worry). Specifically, the EDS currently has 2 TB of space allocated to databases such as those needed for WRES. 2 TBs is nowhere near enough to store even a small NWM reforecast data sets, particularly when multiple variables and/or gridded variables are included. Such data sets, particularly gridded, will easily reach dozens or hundreds of TBs, I would think (could be wrong). In fact, 2 TBs is not even enough to handle HEFS Baseline Validation datasets, at least if we intend to store hindcasts and observations for all locations nation wide (Jim estimates that will take about 16 TBs of disk space, however that translates to Postgres space).

There is also the question of having to move that data from a disk somewhere or the object store into the WRES Postgres database. How much bandwidth will that requires? Will IT complain?

And this gets back to one topic we spoke about last week: allowing a user to write some sort of plug-in reader to the WRES so that WRES can pull data on an as needed basis from the external source of data, such as BPA. From James's description, it appears that they already have a huge database of data that would be impossible or unacceptable for them to have to duplicate in a WRES Postgres DB. This is similar to the NetCDF gridded reader we have now (which will apparently need significant optimization).

The bottom line for me, near term (1.2 and 1.3), is that I'd like for our architecture to at least allow us the option of coding an alternative to the Postgres DB if we choose to in the future. This means separating the logic of the database from the algorithmic code underlying WRES (pairing and metric computations). To a large extent, the separation has already happened: aggregation and pairing are conducted in the WRES application, while Postgres is only used to handle data/sources and identify time offsets. So why not just reengineer it a bit to make that separation formal?

I guess this is something we can discuss further later today.

My two cents,

Hank

epag commented 3 weeks ago

Original Redmine Comment Author Name: Chris (Chris) Original Date: 2018-06-12T13:14:19Z


Some things to note:

  1. The full ingest process is not finished. It will only be truly finished once two or more users can upload the same data to the same database at the same time without any sort of duplication or interruption. When this is done, User A won't have a giant set of data in Database A that is identical to the data that User B uploaded to Database B.
  2. The reason the schema is set up the way it is is mainly to save space. It's been a while since I checked, but I believe the data in the database is around 25-50% smaller than the data spread across PIXML files. Nothing will ever beat NetCDF, but the schema is certainly better than any sort of XML format.

While I'm not looking forward to writing it, I'm not totally against the different options for data retrieval (i.e. database vs flat file vs run time web service retrieval). Whether we like it or not, we'll need to implement the run time web service portion at some point, hopefully during the tail-end of SR2 (that's if WRDS truly gets off the ground).

epag commented 3 weeks ago

Original Redmine Comment Author Name: Hank (Hank) Original Date: 2018-06-20T18:04:15Z


I'm putting this into 1.2 because its been flagged as "Urgent". We call talk about it, decide just how urgent it is, and then schedule it accordingly.

Hank

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-07-22T15:12:04Z


Experiments in #56944 might not extrapolate exactly to larger RAM/disk sizes (because block sizes are fixed, maybe more reasons too). The size of @wres2@ in COWRES production-like environment is 38GiB total, but no one table or index exceeds 1GiB (1/32 of physical RAM), yet it felt like the slowdown of some of our users evaluations was worse-than-linear. Would need to (1) do an evaluation-only run at 38GiB, (2) increase the db to 76GiB in the same proportions across the larger tables, then (3) do an evaluation-only run at the 76GiB to see if the evaluation duration doubled, less-than-doubled, or more-than-doubled. The double-the-size (both sizes larger than physical RAM which is 32GiB on rdb) experiment may not currently be possible with USGS data because the USGS re-use is slim-to-none. That's why #45175 is related.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-07-23T18:20:26Z


RHOER running two evaluations simultaneously on the -ti COWRES, one is pretty small and would expect to finish quickly, but both are in the stage "Loading preliminary metadata", here are the long-running queries as of a few moments ago:

wres16=> select pid, state, substring( query, 0, 64 ), query_start, now() - query_start as duration from pg_stat_activity  where datname = 'wres16'
;
  pid   | state  |                            substring                            |          query_start          |    duration     
--------+--------+-----------------------------------------------------------------+-------------------------------+-----------------
   9670 | idle   | SELECT 1                                                        | 2019-07-23 18:17:04.274135+00 | 00:00:02.56165
  10051 | idle   | SELECT 1                                                        | 2019-07-23 18:17:06.755971+00 | 00:00:00.079814
  10052 | idle   | SELECT 1                                                        | 2019-07-23 18:17:06.755977+00 | 00:00:00.079808
   9891 | idle   | SELECT 1                                                        | 2019-07-23 18:17:04.274136+00 | 00:00:02.561649
  10053 | idle   | SELECT 1                                                        | 2019-07-23 18:17:06.755969+00 | 00:00:00.079816
  68125 | active | SELECT *                                                       +| 2019-07-23 15:28:06.067234+00 | 02:49:00.768551
        |        | FROM wres.Feature F                                            +|                               | 
        |        | WHERE EXISTS (                                                 +|                               | 
        |        |     SELECT 1                                                   +|                               | 
        |        |     FR                                                          |                               | 
  10056 | idle   | SELECT 1                                                        | 2019-07-23 18:17:05.824671+00 | 00:00:01.011114
  16240 | idle   |                                                                 | 2019-07-23 18:17:03.234792+00 | 00:00:03.600993
  16241 | idle   |                                                                 | 2019-07-23 18:17:03.235144+00 | 00:00:03.600641
   9682 | idle   | SELECT 1                                                        | 2019-07-23 18:17:03.729923+00 | 00:00:03.105862
   9830 | idle   | SELECT 1                                                        | 2019-07-23 18:17:03.729923+00 | 00:00:03.105862
  10060 | idle   | SELECT 1                                                        | 2019-07-23 18:17:05.82509+00  | 00:00:01.010695
  10066 | idle   | SHOW TRANSACTION ISOLATION LEVEL                                | 2019-07-23 18:17:05.946195+00 | 00:00:00.88959
 100522 | idle   |                                                                 | 2019-07-23 18:17:03.230863+00 | 00:00:03.604922
   6039 | active | select pid, state, substring( query, 0, 64 ), query_start, now( | 2019-07-23 18:17:06.835785+00 | 00:00:00
 100523 | idle   |                                                                 | 2019-07-23 18:17:03.23128+00  | 00:00:03.604505
 111402 | active | WITH differences AS                                            +| 2019-07-23 17:08:03.412618+00 | 01:09:03.423167
        |        | (                                                              +|                               | 
        |        |     SELECT DISTINCT lead - lag(lead) OVER                       |                               | 
(17 rows)
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-07-23T18:45:03Z


More regarding the state of wres16 on rdb

wres16=> select count(*) from wres.project;
 count 
-------
    12
(1 row)

wres16=> select count(*) from wres.projectsource ;
 count  
--------
 114543
(1 row)

wres16=> select count(*) from wres.source;
 count 
-------
 56681
(1 row)

wres16=> select count(*) from wres.timeseries;
  count  
---------
 9518604
(1 row)

wres16=> select count(*) from wres.observation ;
  count   
----------
 54791048
(1 row)

wres16=> select count(*) from wres.timeseriesvalue;
   count   
-----------
 441424253
(1 row)

wres16=> \dti+ wres.*;
                                                                    List of relations
 Schema |                         Name                          | Type  |    Owner    |                 Table                  |    Size    | Description 
--------+-------------------------------------------------------+-------+-------------+----------------------------------------+------------+-------------
 wres   | ensemble                                              | table | wres        |                                        | 376 kB     | 
 wres   | ensemble_pk                                           | index | wres        | ensemble                               | 16 kB      | 
 wres   | executionlog                                          | table | wres        |                                        | 64 kB      | 
 wres   | feature                                               | table | wres        |                                        | 1184 kB    | 
 wres   | feature_lid_idx                                       | index | wres        | feature                                | 336 kB     | 
 wres   | feature_pk                                            | index | wres        | feature                                | 248 kB     | 
 wres   | gridprojection                                        | table | wres        |                                        | 8192 bytes | 
 wres   | gridprojection_pk                                     | index | wres        | gridprojection                         | 8192 bytes | 
 wres   | indexqueue                                            | table | wres        |                                        | 8192 bytes | 
 wres   | indexqueue_pk                                         | index | wres        | indexqueue                             | 8192 bytes | 
 wres   | measurementunit                                       | table | wres        |                                        | 16 kB      | 
 wres   | measurementunit_unit_name_key                         | index | wres        | measurementunit                        | 16 kB      | 
 wres   | netcdfcoordinate                                      | table | wres        |                                        | 0 bytes    | 
 wres   | netcdfcoordinate_coordinate_idx                       | index | wres        | netcdfcoordinate                       | 8192 bytes | 
 wres   | observation                                           | table | wres        |                                        | 3564 MB    | 
 wres   | observation_source_idx                                | index | wres        | observation                            | 1489 MB    | 
 wres   | observation_time_idx                                  | index | wres        | observation                            | 1545 MB    | 
 wres   | observation_variablefeature_idx                       | index | wres        | observation                            | 1500 MB    | 
 wres   | pk_measurementunit                                    | index | wres        | measurementunit                        | 16 kB      | 
 wres   | project                                               | table | wres        |                                        | 16 kB      | 
 wres   | project_input_code_key                                | index | wres        | project                                | 16 kB      | 
 wres   | project_pk                                            | index | wres        | project                                | 16 kB      | 
 wres   | projectsource                                         | table | wres        |                                        | 4984 kB    | 
 wres   | projectsource_project_idx                             | index | wres        | projectsource                          | 3192 kB    | 
 wres   | projectsource_source_idx                              | index | wres        | projectsource                          | 2864 kB    | 
 wres   | source                                                | table | wres        |                                        | 24 MB      | 
 wres   | source_hash_idx                                       | index | wres        | source                                 | 8568 kB    | 
 wres   | source_output_time_idx                                | index | wres        | source                                 | 3128 kB    | 
 wres   | source_path_idx                                       | index | wres        | source                                 | 22 MB      | 
 wres   | source_pk                                             | index | wres        | source                                 | 3216 kB    | 
 wres   | sourcecompleted                                       | table | wres        |                                        | 3680 kB    | 
 wres   | sourcecompleted_source_idx                            | index | wres        | sourcecompleted                        | 1256 kB    | 
 wres   | timeseries                                            | table | wres        |                                        | 547 MB     | 
 wres   | timeseries_pk                                         | index | wres        | timeseries                             | 204 MB     | 
 wres   | timeseries_source_idx                                 | index | wres        | timeseriessource                       | 12 GB      | 
 wres   | timeseries_variablefeature_ix                         | index | wres        | timeseries                             | 265 MB     | 
 wres   | timeseriessource                                      | table | wres        |                                        | 18 GB      | 
 wres   | timeseriesvalue                                       | table | wres        |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_0                                | table | wres        |                                        | 4580 MB    | 
 wres   | timeseriesvalue_lead_0_lead_idx                       | index | wres        | timeseriesvalue_lead_0                 | 2965 MB    | 
 wres   | timeseriesvalue_lead_0_timeseries_idx                 | index | wres        | timeseriesvalue_lead_0                 | 3054 MB    | 
 wres   | timeseriesvalue_lead_1                                | table | wres        |                                        | 1517 MB    | 
 wres   | timeseriesvalue_lead_10                               | table | wres_user16 |                                        | 584 MB     | 
 wres   | timeseriesvalue_lead_100                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_100_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_100               | 888 kB     | 
 wres   | timeseriesvalue_lead_100_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_100               | 664 kB     | 
 wres   | timeseriesvalue_lead_101                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_101_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_101               | 816 kB     | 
 wres   | timeseriesvalue_lead_101_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_101               | 656 kB     | 
 wres   | timeseriesvalue_lead_102                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_102_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_102               | 832 kB     | 
 wres   | timeseriesvalue_lead_102_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_102               | 656 kB     | 
 wres   | timeseriesvalue_lead_103                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_103_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_103               | 872 kB     | 
 wres   | timeseriesvalue_lead_103_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_103               | 664 kB     | 
 wres   | timeseriesvalue_lead_104                              | table | wres_user16 |                                        | 1304 kB    | 
 wres   | timeseriesvalue_lead_104_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_104               | 784 kB     | 
 wres   | timeseriesvalue_lead_104_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_104               | 664 kB     | 
 wres   | timeseriesvalue_lead_105                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_105_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_105               | 856 kB     | 
 wres   | timeseriesvalue_lead_105_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_105               | 664 kB     | 
 wres   | timeseriesvalue_lead_106                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_106_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_106               | 864 kB     | 
 wres   | timeseriesvalue_lead_106_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_106               | 664 kB     | 
 wres   | timeseriesvalue_lead_107                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_107_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_107               | 864 kB     | 
 wres   | timeseriesvalue_lead_107_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_107               | 656 kB     | 
 wres   | timeseriesvalue_lead_108                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_108_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_108               | 864 kB     | 
 wres   | timeseriesvalue_lead_108_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_108               | 672 kB     | 
 wres   | timeseriesvalue_lead_109                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_109_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_109               | 832 kB     | 
 wres   | timeseriesvalue_lead_109_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_109               | 672 kB     | 
 wres   | timeseriesvalue_lead_10_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_10                | 379 MB     | 
 wres   | timeseriesvalue_lead_10_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_10                | 393 MB     | 
 wres   | timeseriesvalue_lead_11                               | table | wres_user16 |                                        | 185 MB     | 
 wres   | timeseriesvalue_lead_110                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_110_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_110               | 840 kB     | 
 wres   | timeseriesvalue_lead_110_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_110               | 664 kB     | 
 wres   | timeseriesvalue_lead_111                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_111_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_111               | 832 kB     | 
 wres   | timeseriesvalue_lead_111_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_111               | 664 kB     | 
 wres   | timeseriesvalue_lead_112                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_112_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_112               | 888 kB     | 
 wres   | timeseriesvalue_lead_112_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_112               | 664 kB     | 
 wres   | timeseriesvalue_lead_113                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_113_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_113               | 856 kB     | 
 wres   | timeseriesvalue_lead_113_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_113               | 664 kB     | 
 wres   | timeseriesvalue_lead_114                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_114_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_114               | 856 kB     | 
 wres   | timeseriesvalue_lead_114_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_114               | 672 kB     | 
 wres   | timeseriesvalue_lead_115                              | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_115_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_115               | 856 kB     | 
 wres   | timeseriesvalue_lead_115_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_115               | 672 kB     | 
 wres   | timeseriesvalue_lead_116                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_116_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_116               | 848 kB     | 
 wres   | timeseriesvalue_lead_116_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_116               | 664 kB     | 
 wres   | timeseriesvalue_lead_117                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_117_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_117               | 832 kB     | 
 wres   | timeseriesvalue_lead_117_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_117               | 664 kB     | 
 wres   | timeseriesvalue_lead_118                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_118_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_118               | 864 kB     | 
 wres   | timeseriesvalue_lead_118_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_118               | 664 kB     | 
 wres   | timeseriesvalue_lead_119                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_119_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_119               | 872 kB     | 
 wres   | timeseriesvalue_lead_119_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_119               | 664 kB     | 
 wres   | timeseriesvalue_lead_11_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_11                | 120 MB     | 
 wres   | timeseriesvalue_lead_11_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_11                | 120 MB     | 
 wres   | timeseriesvalue_lead_12                               | table | wres_user16 |                                        | 32 MB      | 
 wres   | timeseriesvalue_lead_120                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_120_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_120               | 848 kB     | 
 wres   | timeseriesvalue_lead_120_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_120               | 664 kB     | 
 wres   | timeseriesvalue_lead_121                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_121_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_121               | 824 kB     | 
 wres   | timeseriesvalue_lead_121_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_121               | 664 kB     | 
 wres   | timeseriesvalue_lead_122                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_122_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_122               | 848 kB     | 
 wres   | timeseriesvalue_lead_122_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_122               | 664 kB     | 
 wres   | timeseriesvalue_lead_123                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_123_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_123               | 848 kB     | 
 wres   | timeseriesvalue_lead_123_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_123               | 664 kB     | 
 wres   | timeseriesvalue_lead_124                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_124_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_124               | 864 kB     | 
 wres   | timeseriesvalue_lead_124_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_124               | 664 kB     | 
 wres   | timeseriesvalue_lead_125                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_125_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_125               | 864 kB     | 
 wres   | timeseriesvalue_lead_125_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_125               | 664 kB     | 
 wres   | timeseriesvalue_lead_126                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_126_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_126               | 856 kB     | 
 wres   | timeseriesvalue_lead_126_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_126               | 664 kB     | 
 wres   | timeseriesvalue_lead_127                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_127_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_127               | 856 kB     | 
 wres   | timeseriesvalue_lead_127_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_127               | 664 kB     | 
 wres   | timeseriesvalue_lead_128                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_128_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_128               | 856 kB     | 
 wres   | timeseriesvalue_lead_128_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_128               | 664 kB     | 
 wres   | timeseriesvalue_lead_129                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_129_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_129               | 896 kB     | 
 wres   | timeseriesvalue_lead_129_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_129               | 664 kB     | 
 wres   | timeseriesvalue_lead_12_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_12                | 21 MB      | 
 wres   | timeseriesvalue_lead_12_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_12                | 22 MB      | 
 wres   | timeseriesvalue_lead_13                               | table | wres_user16 |                                        | 1496 kB    | 
 wres   | timeseriesvalue_lead_130                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_130_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_130               | 856 kB     | 
 wres   | timeseriesvalue_lead_130_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_130               | 664 kB     | 
 wres   | timeseriesvalue_lead_131                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_131_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_131               | 832 kB     | 
 wres   | timeseriesvalue_lead_131_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_131               | 664 kB     | 
 wres   | timeseriesvalue_lead_132                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_132_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_132               | 832 kB     | 
 wres   | timeseriesvalue_lead_132_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_132               | 664 kB     | 
 wres   | timeseriesvalue_lead_133                              | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_133_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_133               | 832 kB     | 
 wres   | timeseriesvalue_lead_133_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_133               | 664 kB     | 
 wres   | timeseriesvalue_lead_134                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_134_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_134               | 856 kB     | 
 wres   | timeseriesvalue_lead_134_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_134               | 664 kB     | 
 wres   | timeseriesvalue_lead_135                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_135_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_135               | 856 kB     | 
 wres   | timeseriesvalue_lead_135_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_135               | 664 kB     | 
 wres   | timeseriesvalue_lead_136                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_136_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_136               | 864 kB     | 
 wres   | timeseriesvalue_lead_136_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_136               | 664 kB     | 
 wres   | timeseriesvalue_lead_137                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_137_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_137               | 840 kB     | 
 wres   | timeseriesvalue_lead_137_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_137               | 664 kB     | 
 wres   | timeseriesvalue_lead_138                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_138_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_138               | 872 kB     | 
 wres   | timeseriesvalue_lead_138_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_138               | 664 kB     | 
 wres   | timeseriesvalue_lead_139                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_139_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_139               | 832 kB     | 
 wres   | timeseriesvalue_lead_139_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_139               | 664 kB     | 
 wres   | timeseriesvalue_lead_13_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_13                | 984 kB     | 
 wres   | timeseriesvalue_lead_13_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_13                | 752 kB     | 
 wres   | timeseriesvalue_lead_14                               | table | wres_user16 |                                        | 1464 kB    | 
 wres   | timeseriesvalue_lead_140                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_140_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_140               | 880 kB     | 
 wres   | timeseriesvalue_lead_140_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_140               | 664 kB     | 
 wres   | timeseriesvalue_lead_141                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_141_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_141               | 864 kB     | 
 wres   | timeseriesvalue_lead_141_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_141               | 672 kB     | 
 wres   | timeseriesvalue_lead_142                              | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_142_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_142               | 864 kB     | 
 wres   | timeseriesvalue_lead_142_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_142               | 712 kB     | 
 wres   | timeseriesvalue_lead_143                              | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_143_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_143               | 872 kB     | 
 wres   | timeseriesvalue_lead_143_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_143               | 664 kB     | 
 wres   | timeseriesvalue_lead_144                              | table | wres_user16 |                                        | 120 kB     | 
 wres   | timeseriesvalue_lead_144_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_144               | 64 kB      | 
 wres   | timeseriesvalue_lead_144_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_144               | 48 kB      | 
 wres   | timeseriesvalue_lead_145                              | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_145_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_145               | 8192 bytes | 
 wres   | timeseriesvalue_lead_145_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_145               | 8192 bytes | 
 wres   | timeseriesvalue_lead_146                              | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_146_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_146               | 8192 bytes | 
 wres   | timeseriesvalue_lead_146_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_146               | 8192 bytes | 
 wres   | timeseriesvalue_lead_147                              | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_147_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_147               | 8192 bytes | 
 wres   | timeseriesvalue_lead_147_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_147               | 8192 bytes | 
 wres   | timeseriesvalue_lead_148                              | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_148_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_148               | 8192 bytes | 
 wres   | timeseriesvalue_lead_148_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_148               | 8192 bytes | 
 wres   | timeseriesvalue_lead_149                              | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_149_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_149               | 8192 bytes | 
 wres   | timeseriesvalue_lead_149_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_149               | 8192 bytes | 
 wres   | timeseriesvalue_lead_14_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_14                | 944 kB     | 
 wres   | timeseriesvalue_lead_14_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_14                | 728 kB     | 
 wres   | timeseriesvalue_lead_15                               | table | wres_user16 |                                        | 1424 kB    | 
 wres   | timeseriesvalue_lead_150                              | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_150_lead_idx                     | index | wres_user16 | timeseriesvalue_lead_150               | 8192 bytes | 
 wres   | timeseriesvalue_lead_150_timeseries_idx               | index | wres_user16 | timeseriesvalue_lead_150               | 8192 bytes | 
 wres   | timeseriesvalue_lead_15_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_15                | 952 kB     | 
 wres   | timeseriesvalue_lead_15_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_15                | 712 kB     | 
 wres   | timeseriesvalue_lead_16                               | table | wres_user16 |                                        | 1448 kB    | 
 wres   | timeseriesvalue_lead_16_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_16                | 944 kB     | 
 wres   | timeseriesvalue_lead_16_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_16                | 712 kB     | 
 wres   | timeseriesvalue_lead_17                               | table | wres_user16 |                                        | 1408 kB    | 
 wres   | timeseriesvalue_lead_17_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_17                | 936 kB     | 
 wres   | timeseriesvalue_lead_17_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_17                | 696 kB     | 
 wres   | timeseriesvalue_lead_18                               | table | wres_user16 |                                        | 1408 kB    | 
 wres   | timeseriesvalue_lead_18_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_18                | 920 kB     | 
 wres   | timeseriesvalue_lead_18_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_18                | 688 kB     | 
 wres   | timeseriesvalue_lead_19                               | table | wres_user16 |                                        | 1424 kB    | 
 wres   | timeseriesvalue_lead_19_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_19                | 920 kB     | 
 wres   | timeseriesvalue_lead_19_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_19                | 696 kB     | 
 wres   | timeseriesvalue_lead_1_lead_idx                       | index | wres        | timeseriesvalue_lead_1                 | 983 MB     | 
 wres   | timeseriesvalue_lead_1_timeseries_idx                 | index | wres        | timeseriesvalue_lead_1                 | 1001 MB    | 
 wres   | timeseriesvalue_lead_2                                | table | wres        |                                        | 1300 MB    | 
 wres   | timeseriesvalue_lead_20                               | table | wres_user16 |                                        | 1416 kB    | 
 wres   | timeseriesvalue_lead_20_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_20                | 920 kB     | 
 wres   | timeseriesvalue_lead_20_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_20                | 696 kB     | 
 wres   | timeseriesvalue_lead_21                               | table | wres_user16 |                                        | 1400 kB    | 
 wres   | timeseriesvalue_lead_21_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_21                | 904 kB     | 
 wres   | timeseriesvalue_lead_21_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_21                | 688 kB     | 
 wres   | timeseriesvalue_lead_22                               | table | wres_user16 |                                        | 1392 kB    | 
 wres   | timeseriesvalue_lead_22_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_22                | 944 kB     | 
 wres   | timeseriesvalue_lead_22_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_22                | 696 kB     | 
 wres   | timeseriesvalue_lead_23                               | table | wres_user16 |                                        | 1408 kB    | 
 wres   | timeseriesvalue_lead_23_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_23                | 968 kB     | 
 wres   | timeseriesvalue_lead_23_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_23                | 696 kB     | 
 wres   | timeseriesvalue_lead_24                               | table | wres_user16 |                                        | 1384 kB    | 
 wres   | timeseriesvalue_lead_24_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_24                | 896 kB     | 
 wres   | timeseriesvalue_lead_24_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_24                | 688 kB     | 
 wres   | timeseriesvalue_lead_25                               | table | wres_user16 |                                        | 1424 kB    | 
 wres   | timeseriesvalue_lead_25_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_25                | 912 kB     | 
 wres   | timeseriesvalue_lead_25_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_25                | 696 kB     | 
 wres   | timeseriesvalue_lead_26                               | table | wres_user16 |                                        | 1368 kB    | 
 wres   | timeseriesvalue_lead_26_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_26                | 928 kB     | 
 wres   | timeseriesvalue_lead_26_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_26                | 696 kB     | 
 wres   | timeseriesvalue_lead_27                               | table | wres_user16 |                                        | 1384 kB    | 
 wres   | timeseriesvalue_lead_27_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_27                | 928 kB     | 
 wres   | timeseriesvalue_lead_27_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_27                | 688 kB     | 
 wres   | timeseriesvalue_lead_28                               | table | wres_user16 |                                        | 1432 kB    | 
 wres   | timeseriesvalue_lead_28_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_28                | 928 kB     | 
 wres   | timeseriesvalue_lead_28_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_28                | 696 kB     | 
 wres   | timeseriesvalue_lead_29                               | table | wres_user16 |                                        | 1424 kB    | 
 wres   | timeseriesvalue_lead_29_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_29                | 936 kB     | 
 wres   | timeseriesvalue_lead_29_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_29                | 696 kB     | 
 wres   | timeseriesvalue_lead_2_lead_idx                       | index | wres        | timeseriesvalue_lead_2                 | 844 MB     | 
 wres   | timeseriesvalue_lead_2_timeseries_idx                 | index | wres        | timeseriesvalue_lead_2                 | 841 MB     | 
 wres   | timeseriesvalue_lead_3                                | table | wres_user16 |                                        | 1514 MB    | 
 wres   | timeseriesvalue_lead_30                               | table | wres_user16 |                                        | 1384 kB    | 
 wres   | timeseriesvalue_lead_30_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_30                | 896 kB     | 
 wres   | timeseriesvalue_lead_30_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_30                | 696 kB     | 
 wres   | timeseriesvalue_lead_31                               | table | wres_user16 |                                        | 1408 kB    | 
 wres   | timeseriesvalue_lead_31_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_31                | 936 kB     | 
 wres   | timeseriesvalue_lead_31_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_31                | 696 kB     | 
 wres   | timeseriesvalue_lead_32                               | table | wres_user16 |                                        | 1376 kB    | 
 wres   | timeseriesvalue_lead_32_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_32                | 920 kB     | 
 wres   | timeseriesvalue_lead_32_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_32                | 696 kB     | 
 wres   | timeseriesvalue_lead_33                               | table | wres_user16 |                                        | 1368 kB    | 
 wres   | timeseriesvalue_lead_33_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_33                | 912 kB     | 
 wres   | timeseriesvalue_lead_33_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_33                | 680 kB     | 
 wres   | timeseriesvalue_lead_34                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_34_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_34                | 824 kB     | 
 wres   | timeseriesvalue_lead_34_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_34                | 672 kB     | 
 wres   | timeseriesvalue_lead_35                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_35_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_35                | 848 kB     | 
 wres   | timeseriesvalue_lead_35_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_35                | 672 kB     | 
 wres   | timeseriesvalue_lead_36                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_36_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_36                | 872 kB     | 
 wres   | timeseriesvalue_lead_36_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_36                | 672 kB     | 
 wres   | timeseriesvalue_lead_37                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_37_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_37                | 848 kB     | 
 wres   | timeseriesvalue_lead_37_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_37                | 672 kB     | 
 wres   | timeseriesvalue_lead_38                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_38_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_38                | 864 kB     | 
 wres   | timeseriesvalue_lead_38_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_38                | 672 kB     | 
 wres   | timeseriesvalue_lead_39                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_39_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_39                | 848 kB     | 
 wres   | timeseriesvalue_lead_39_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_39                | 672 kB     | 
 wres   | timeseriesvalue_lead_3_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_3                 | 982 MB     | 
 wres   | timeseriesvalue_lead_3_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_3                 | 1000 MB    | 
 wres   | timeseriesvalue_lead_4                                | table | wres_user16 |                                        | 1514 MB    | 
 wres   | timeseriesvalue_lead_40                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_40_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_40                | 856 kB     | 
 wres   | timeseriesvalue_lead_40_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_40                | 672 kB     | 
 wres   | timeseriesvalue_lead_41                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_41_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_41                | 872 kB     | 
 wres   | timeseriesvalue_lead_41_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_41                | 672 kB     | 
 wres   | timeseriesvalue_lead_42                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_42_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_42                | 848 kB     | 
 wres   | timeseriesvalue_lead_42_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_42                | 672 kB     | 
 wres   | timeseriesvalue_lead_43                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_43_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_43                | 856 kB     | 
 wres   | timeseriesvalue_lead_43_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_43                | 672 kB     | 
 wres   | timeseriesvalue_lead_44                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_44_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_44                | 872 kB     | 
 wres   | timeseriesvalue_lead_44_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_44                | 672 kB     | 
 wres   | timeseriesvalue_lead_45                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_45_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_45                | 856 kB     | 
 wres   | timeseriesvalue_lead_45_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_45                | 672 kB     | 
 wres   | timeseriesvalue_lead_46                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_46_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_46                | 824 kB     | 
 wres   | timeseriesvalue_lead_46_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_46                | 672 kB     | 
 wres   | timeseriesvalue_lead_47                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_47_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_47                | 864 kB     | 
 wres   | timeseriesvalue_lead_47_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_47                | 672 kB     | 
 wres   | timeseriesvalue_lead_48                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_48_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_48                | 848 kB     | 
 wres   | timeseriesvalue_lead_48_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_48                | 672 kB     | 
 wres   | timeseriesvalue_lead_49                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_49_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_49                | 832 kB     | 
 wres   | timeseriesvalue_lead_49_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_49                | 672 kB     | 
 wres   | timeseriesvalue_lead_4_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_4                 | 982 MB     | 
 wres   | timeseriesvalue_lead_4_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_4                 | 999 MB     | 
 wres   | timeseriesvalue_lead_5                                | table | wres_user16 |                                        | 1298 MB    | 
 wres   | timeseriesvalue_lead_50                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_50_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_50                | 840 kB     | 
 wres   | timeseriesvalue_lead_50_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_50                | 672 kB     | 
 wres   | timeseriesvalue_lead_51                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_51_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_51                | 856 kB     | 
 wres   | timeseriesvalue_lead_51_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_51                | 672 kB     | 
 wres   | timeseriesvalue_lead_52                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_52_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_52                | 824 kB     | 
 wres   | timeseriesvalue_lead_52_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_52                | 672 kB     | 
 wres   | timeseriesvalue_lead_53                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_53_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_53                | 848 kB     | 
 wres   | timeseriesvalue_lead_53_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_53                | 672 kB     | 
 wres   | timeseriesvalue_lead_54                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_54_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_54                | 840 kB     | 
 wres   | timeseriesvalue_lead_54_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_54                | 672 kB     | 
 wres   | timeseriesvalue_lead_55                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_55_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_55                | 840 kB     | 
 wres   | timeseriesvalue_lead_55_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_55                | 672 kB     | 
 wres   | timeseriesvalue_lead_56                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_56_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_56                | 848 kB     | 
 wres   | timeseriesvalue_lead_56_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_56                | 672 kB     | 
 wres   | timeseriesvalue_lead_57                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_57_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_57                | 872 kB     | 
 wres   | timeseriesvalue_lead_57_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_57                | 664 kB     | 
 wres   | timeseriesvalue_lead_58                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_58_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_58                | 896 kB     | 
 wres   | timeseriesvalue_lead_58_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_58                | 656 kB     | 
 wres   | timeseriesvalue_lead_59                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_59_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_59                | 896 kB     | 
 wres   | timeseriesvalue_lead_59_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_59                | 656 kB     | 
 wres   | timeseriesvalue_lead_5_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_5                 | 842 MB     | 
 wres   | timeseriesvalue_lead_5_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_5                 | 841 MB     | 
 wres   | timeseriesvalue_lead_6                                | table | wres_user16 |                                        | 1509 MB    | 
 wres   | timeseriesvalue_lead_60                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_60_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_60                | 864 kB     | 
 wres   | timeseriesvalue_lead_60_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_60                | 656 kB     | 
 wres   | timeseriesvalue_lead_61                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_61_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_61                | 848 kB     | 
 wres   | timeseriesvalue_lead_61_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_61                | 656 kB     | 
 wres   | timeseriesvalue_lead_62                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_62_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_62                | 864 kB     | 
 wres   | timeseriesvalue_lead_62_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_62                | 656 kB     | 
 wres   | timeseriesvalue_lead_63                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_63_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_63                | 872 kB     | 
 wres   | timeseriesvalue_lead_63_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_63                | 656 kB     | 
 wres   | timeseriesvalue_lead_64                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_64_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_64                | 840 kB     | 
 wres   | timeseriesvalue_lead_64_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_64                | 664 kB     | 
 wres   | timeseriesvalue_lead_65                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_65_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_65                | 872 kB     | 
 wres   | timeseriesvalue_lead_65_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_65                | 656 kB     | 
 wres   | timeseriesvalue_lead_66                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_66_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_66                | 888 kB     | 
 wres   | timeseriesvalue_lead_66_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_66                | 656 kB     | 
 wres   | timeseriesvalue_lead_67                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_67_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_67                | 832 kB     | 
 wres   | timeseriesvalue_lead_67_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_67                | 656 kB     | 
 wres   | timeseriesvalue_lead_68                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_68_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_68                | 840 kB     | 
 wres   | timeseriesvalue_lead_68_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_68                | 656 kB     | 
 wres   | timeseriesvalue_lead_69                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_69_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_69                | 864 kB     | 
 wres   | timeseriesvalue_lead_69_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_69                | 656 kB     | 
 wres   | timeseriesvalue_lead_6_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_6                 | 979 MB     | 
 wres   | timeseriesvalue_lead_6_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_6                 | 999 MB     | 
 wres   | timeseriesvalue_lead_7                                | table | wres_user16 |                                        | 1509 MB    | 
 wres   | timeseriesvalue_lead_70                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_70_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_70                | 848 kB     | 
 wres   | timeseriesvalue_lead_70_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_70                | 656 kB     | 
 wres   | timeseriesvalue_lead_71                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_71_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_71                | 880 kB     | 
 wres   | timeseriesvalue_lead_71_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_71                | 656 kB     | 
 wres   | timeseriesvalue_lead_72                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_72_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_72                | 832 kB     | 
 wres   | timeseriesvalue_lead_72_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_72                | 656 kB     | 
 wres   | timeseriesvalue_lead_73                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_73_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_73                | 840 kB     | 
 wres   | timeseriesvalue_lead_73_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_73                | 656 kB     | 
 wres   | timeseriesvalue_lead_74                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_74_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_74                | 848 kB     | 
 wres   | timeseriesvalue_lead_74_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_74                | 656 kB     | 
 wres   | timeseriesvalue_lead_75                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_75_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_75                | 848 kB     | 
 wres   | timeseriesvalue_lead_75_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_75                | 656 kB     | 
 wres   | timeseriesvalue_lead_76                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_76_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_76                | 848 kB     | 
 wres   | timeseriesvalue_lead_76_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_76                | 656 kB     | 
 wres   | timeseriesvalue_lead_77                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_77_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_77                | 856 kB     | 
 wres   | timeseriesvalue_lead_77_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_77                | 656 kB     | 
 wres   | timeseriesvalue_lead_78                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_78_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_78                | 864 kB     | 
 wres   | timeseriesvalue_lead_78_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_78                | 664 kB     | 
 wres   | timeseriesvalue_lead_79                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_79_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_79                | 864 kB     | 
 wres   | timeseriesvalue_lead_79_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_79                | 656 kB     | 
 wres   | timeseriesvalue_lead_7_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_7                 | 979 MB     | 
 wres   | timeseriesvalue_lead_7_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_7                 | 999 MB     | 
 wres   | timeseriesvalue_lead_8                                | table | wres_user16 |                                        | 1292 MB    | 
 wres   | timeseriesvalue_lead_80                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_80_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_80                | 824 kB     | 
 wres   | timeseriesvalue_lead_80_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_80                | 656 kB     | 
 wres   | timeseriesvalue_lead_81                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_81_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_81                | 880 kB     | 
 wres   | timeseriesvalue_lead_81_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_81                | 656 kB     | 
 wres   | timeseriesvalue_lead_82                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_82_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_82                | 832 kB     | 
 wres   | timeseriesvalue_lead_82_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_82                | 656 kB     | 
 wres   | timeseriesvalue_lead_83                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_83_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_83                | 832 kB     | 
 wres   | timeseriesvalue_lead_83_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_83                | 656 kB     | 
 wres   | timeseriesvalue_lead_84                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_84_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_84                | 832 kB     | 
 wres   | timeseriesvalue_lead_84_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_84                | 656 kB     | 
 wres   | timeseriesvalue_lead_85                               | table | wres_user16 |                                        | 1304 kB    | 
 wres   | timeseriesvalue_lead_85_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_85                | 824 kB     | 
 wres   | timeseriesvalue_lead_85_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_85                | 656 kB     | 
 wres   | timeseriesvalue_lead_86                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_86_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_86                | 872 kB     | 
 wres   | timeseriesvalue_lead_86_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_86                | 656 kB     | 
 wres   | timeseriesvalue_lead_87                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_87_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_87                | 864 kB     | 
 wres   | timeseriesvalue_lead_87_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_87                | 656 kB     | 
 wres   | timeseriesvalue_lead_88                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_88_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_88                | 856 kB     | 
 wres   | timeseriesvalue_lead_88_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_88                | 656 kB     | 
 wres   | timeseriesvalue_lead_89                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_89_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_89                | 808 kB     | 
 wres   | timeseriesvalue_lead_89_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_89                | 656 kB     | 
 wres   | timeseriesvalue_lead_8_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_8                 | 838 MB     | 
 wres   | timeseriesvalue_lead_8_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_8                 | 838 MB     | 
 wres   | timeseriesvalue_lead_9                                | table | wres_user16 |                                        | 1505 MB    | 
 wres   | timeseriesvalue_lead_90                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_90_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_90                | 856 kB     | 
 wres   | timeseriesvalue_lead_90_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_90                | 656 kB     | 
 wres   | timeseriesvalue_lead_91                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_91_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_91                | 800 kB     | 
 wres   | timeseriesvalue_lead_91_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_91                | 656 kB     | 
 wres   | timeseriesvalue_lead_92                               | table | wres_user16 |                                        | 1336 kB    | 
 wres   | timeseriesvalue_lead_92_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_92                | 856 kB     | 
 wres   | timeseriesvalue_lead_92_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_92                | 656 kB     | 
 wres   | timeseriesvalue_lead_93                               | table | wres_user16 |                                        | 1328 kB    | 
 wres   | timeseriesvalue_lead_93_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_93                | 832 kB     | 
 wres   | timeseriesvalue_lead_93_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_93                | 664 kB     | 
 wres   | timeseriesvalue_lead_94                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_94_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_94                | 832 kB     | 
 wres   | timeseriesvalue_lead_94_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_94                | 664 kB     | 
 wres   | timeseriesvalue_lead_95                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_95_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_95                | 896 kB     | 
 wres   | timeseriesvalue_lead_95_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_95                | 664 kB     | 
 wres   | timeseriesvalue_lead_96                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_96_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_96                | 824 kB     | 
 wres   | timeseriesvalue_lead_96_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_96                | 664 kB     | 
 wres   | timeseriesvalue_lead_97                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_97_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_97                | 832 kB     | 
 wres   | timeseriesvalue_lead_97_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_97                | 664 kB     | 
 wres   | timeseriesvalue_lead_98                               | table | wres_user16 |                                        | 1320 kB    | 
 wres   | timeseriesvalue_lead_98_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_98                | 824 kB     | 
 wres   | timeseriesvalue_lead_98_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_98                | 664 kB     | 
 wres   | timeseriesvalue_lead_99                               | table | wres_user16 |                                        | 1312 kB    | 
 wres   | timeseriesvalue_lead_99_lead_idx                      | index | wres_user16 | timeseriesvalue_lead_99                | 864 kB     | 
 wres   | timeseriesvalue_lead_99_timeseries_idx                | index | wres_user16 | timeseriesvalue_lead_99                | 656 kB     | 
 wres   | timeseriesvalue_lead_9_lead_idx                       | index | wres_user16 | timeseriesvalue_lead_9                 | 976 MB     | 
 wres   | timeseriesvalue_lead_9_timeseries_idx                 | index | wres_user16 | timeseriesvalue_lead_9                 | 998 MB     | 
 wres   | timeseriesvalue_lead_above_150                        | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_above_150_lead_idx               | index | wres_user16 | timeseriesvalue_lead_above_150         | 8192 bytes | 
 wres   | timeseriesvalue_lead_above_150_timeseries_idx         | index | wres_user16 | timeseriesvalue_lead_above_150         | 8192 bytes | 
 wres   | timeseriesvalue_lead_below_negative_10                | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_below_negative_10_lead_idx       | index | wres_user16 | timeseriesvalue_lead_below_negative_10 | 8192 bytes | 
 wres   | timeseriesvalue_lead_below_negative_10_timeseries_idx | index | wres_user16 | timeseriesvalue_lead_below_negative_10 | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_1                       | table | wres_user16 |                                        | 48 kB      | 
 wres   | timeseriesvalue_lead_negative_10                      | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_10_lead_idx             | index | wres_user16 | timeseriesvalue_lead_negative_10       | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_10_timeseries_idx       | index | wres_user16 | timeseriesvalue_lead_negative_10       | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_1_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_1        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_1_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_1        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_2                       | table | wres_user16 |                                        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_2_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_2        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_2_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_2        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_3                       | table | wres_user16 |                                        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_3_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_3        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_3_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_3        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_4                       | table | wres_user16 |                                        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_4_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_4        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_4_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_4        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_5                       | table | wres_user16 |                                        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_5_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_5        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_5_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_5        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_6                       | table | wres_user16 |                                        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_6_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_6        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_6_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_6        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_7                       | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_7_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_7        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_7_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_7        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_8                       | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_8_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_8        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_8_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_8        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_9                       | table | wres_user16 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_9_lead_idx              | index | wres_user16 | timeseriesvalue_lead_negative_9        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_9_timeseries_idx        | index | wres_user16 | timeseriesvalue_lead_negative_9        | 8192 bytes | 
 wres   | timeseriesvalue_timeseries_idx                        | index | wres        | timeseriesvalue                        | 8192 bytes | 
 wres   | unitconversion                                        | table | wres        |                                        | 56 kB      | 
 wres   | unitconversion_measurement_uix                        | index | wres        | unitconversion                         | 32 kB      | 
 wres   | usgsparameter                                         | table | wres        |                                        | 16 kB      | 
 wres   | usgsparameter_name_agg_measurement_idx                | index | wres        | usgsparameter                          | 16 kB      | 
 wres   | variable                                              | table | wres        |                                        | 16 kB      | 
 wres   | variable_pk                                           | index | wres        | variable                               | 16 kB      | 
 wres   | variable_variable_name_idx                            | index | wres        | variable                               | 16 kB      | 
 wres   | variable_variable_name_key                            | index | wres        | variable                               | 16 kB      | 
 wres   | variablefeature                                       | table | wres        |                                        | 696 kB     | 
 wres   | variablefeature_feature_uix                           | index | wres        | variablefeature                        | 280 kB     | 
 wres   | variablefeature_pk                                    | index | wres        | variablefeature                        | 256 kB     | 
(539 rows)
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-07-23T18:45:52Z


One thing jumping out is wres.timeseriessource is larger than -RAM- shared buffers.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-07-23T22:04:08Z


rdb has around 31.2GiB RAM and shared buffers is set at 12GiB.

Only @wres.timeseriessource@ exceeds shared buffers but does not exceed db server RAM.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-07-23T22:21:31Z


Ideas for handling larger-than-RAM data.

  1. Unify timeseries data.

    • Put observation timeseries data into @wres.timeseriesvalue@ and
    • a slimmer @wres.timeseries@,
    • put forecast-related data into @wres.forecast@ (which has a fk to @wres.timeseries@),
    • use valid datetime in @wres.timeseriesvalue@ rather than lead.
  2. Make NWM data look the same to the system as normal timeseries data

    • Remove @wres.timeseriessource@ table entirely (or at least make it historical and not part of queries),
    • create fancier readers for NWM data that present individual timeseries to the WRES.
  3. Forget about sources, identify timeseries instead

    • Remove @wres.source@ table entirely (or at least make it historical and not part of queries),
    • remove @wres.projectsource@ table entirely,
    • add @wres.projecttimeseries@ table,
    • use hash of the timeseries to identify a timeseries, include in @wres.timeseries@ as natural key
  4. Cap the growth of tables expected to grow beyond 1 million rows

    • Partition both @wres.timeseries@ and @wres.timeseriesvalue@ tables by @timeseries_id@,
    • partition the new table @wres.projecttimeseries@ also by @timeseries_id@,
    • when partitioning by @timeseries_id@ use the same bounds on all tables so that we can truncate all the partitions referencing the earliest timeseries in one fell swoop without affecting later timeseries,
    • have the liquibase migration both remove and add partitions every time it runs (only if needed), to keep a rolling window of several dozen partitions across all these tables.
  5. Use 64-bit integer keys so that the one database is sustainable for a while.

Shorter term I think we could eliminate or modify some queries. But longer term, we could allow for an ever-growing database.

This is an expansion of the suggestion in #61488-13, #61488-16, #61488-17, #65216-1, #65216-10. (Edit: formatting, edit to allow for different use of @wres.timeseriessource@, add other ticket refs, qualify liquibase idea)

epag commented 3 weeks ago

Original Redmine Comment Author Name: James (James) Original Date: 2019-07-23T23:33:29Z


I don't believe that you're looking for my agreement on #50327-19, but I'll offer it anyway, because I've reached the same conclusions, at least in the big-picture sense.

I believe it's Chris that we need to convince about the big picture, and I'm not saying that he doesn't have some legitimate concerns, performance-wise, but I think it's also hard to predict performance and it is secondary to correctness and arguably to maintainability and other related things too.

I have some minor quibbles or questions about the proposed implementation that need further discussion, such as the hash of the time-series as a unique identifier (I think that poses a non-trivial risk of collision, particularly for short time-series or intermittent variables like precipitation). However, any smaller quibbles or questions should not detract from my overall agreement.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Chris (Chris) Original Date: 2019-07-24T13:44:45Z


bq. Put observation timeseries data into wres.timeseriesvalue and

Vehemently disagree; access patterns for observations and our forecast data are totally different and our data modelling needs to reflect this.

bq. a slimmer wres.timeseries

If you're talking about normalizing a bit, there are some pros and cons; denormalization speeds up joins since it'll remove the need for different joins. That benefit isn't enormous though; there are only a few queries that actually rely on that data and they aren't responsible for for actually retrieving anything. I'm imagining that the denormalization would mean moving the scale information out. Anything else is short sighted through and through. While moving those columns probably wouldn't reduce that much of a slowdown, I'm not sure if the work involved doesn't really balance out the benefit, though that's just a rough opinion on task priorities and not the functionality itself.

bq. put forecast-related data into wres.forecast (which has a fk to wres.timeseries)

@wres.TimeSeries@ is and was @wres.Forecast@, but there was a tremendous amount of arguing about whether or not it was an appropriate name. The only reason that it is titled @wres.TimeSeries@ was to quell pointless arguments on nomenclature. This is just a rehash of prior arguments. What names would you prefer? This is forecast data, not observation data; assumptions based on the name don't change that. Would shifting back to the name @wres.Forecast@ again help?

bq. use valid datetime in wres.timeseriesvalue rather than lead

This keeps getting brought up and it's wrong every single time. Adding the valid datetime in addition to the lead isn't a bad idea, but removing the primary method of identifying the data we want is a terrible idea. Sure, you can always determine that information by performing an expression within a @where@ or @join@, but then you're performing an expression within a @where@ or @join@ which is a terrible idea.

bq. Remove wres.timeseriessource table entirely (or at least make it historical and not part of queries),

I hate @wres.TimeSeriesSource@ and getting rid of it and moving to a join model akin to @wres.Observation@ would benefit us greatly. Finding a way to bundle the grouping of data for NWM is the hard part. Part of that difficulty is that we'd need a good deal of precognition to identify what values correspond to what. NWM 2.X grants us a TON of tools for that though, so we might have a chance of getting a good solution now. That solution wouldn't be compatible with 1.X data, though, so a decision would need to be made for that.

remove wres.projectsource table entirely, add wres.projecttimeseries table,

What's the solution for observation vs forecast data? I like @wres.ProjectTimeSeries@, but linking individual values for an observation is a bit of a conundrum. If we use @wres.ProjectSource@, we have one ID identifying an enormous amount of values. It's the same situation with @wres.TimeSeries@, though it isn't as strong with NWM data, though we'd have 18 records in @wres.ProjectSource@ rather than ~8000 in @wres.ProjectTimeSeries@ per source.

bq. partition the new table wres.projecttimeseries also by timeseries_id,

If we create @wres.ProjectTimeSeries@, I don't have any idea how else you'd do it, so that works.

bq. use hash of the timeseries to identify a timeseries, include in wres.timeseries as natural key

NWM data is again the wrench here since we'd need a good amount of precognition to determine the actual hash of a time series as we load everything. We'd also run into an issue where we'd have to load every time series we encounter in order to determine whether or not we should do anything with it. That would be leagues worse that what we already have, which is already lacking.

Partition both wres.timeseries and wres.timeseriesvalue tables by timeseries_id partition the new table wres.projecttimeseries also by timeseries_id,

Partitioning @wres.TimeSeries@ isn't a terrible idea, we'd just need to invest time into rewriting queries to take advantage of it. I'd say that's worth the effort. I've said this for years, YEARS, though: partitioning @wres.TimeSeriesValue@ by @timeseries_id@ is worse than a terrible idea. I've shown the math. The point of the partitions isn't for storage concerns, it's for access concerns, which partitioning by our access patterns actually addresses.

when partitioning by timeseries_id use the same bounds on all tables so that we can truncate all the partitions referencing the earliest timeseries in one fell swoop without affecting later timeseries have the liquibase migration both remove and add partitions every time it runs (only if needed), to keep a rolling window of several dozen partitions across all these tables

Not a terrible idea if you're talking about the partitioning on @wres.ProjectTimeSeries@. It would have to be done in a really intelligent way, however; I can't think of a way to do it that wouldn't result in mountains of orphaned data or grind ingest and total operation on the database on the host due to cascading deletes. We'd need an expert to consult on that sort of operation.

bq. Use 64-bit integer keys so that the one database is sustainable for a while.

Sounds reasonable.

bq. Shorter term I think we could eliminate or modify some queries. But longer term, we could allow for an ever-growing database.

Yes, yes, yes, yes, yes. Since there are a TON of new restrictions on what must be in a project configuration, we can probably remove a lot of queries that were compensating for week declarations.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-08-05T18:38:26Z


Xia reports that an evaluation that was taking around 4 minutes, for a single location, now is taking longer than 15 minutes, seems to be stuck on the "Loading preliminary data" step. This is on COWRES 1.13, with the shared single database instance.

Active query shows

wres8=> select datname, query, xact_start, now() - xact_start as duration from pg_stat_activity where state = 'active' and query not like '%schnitzelbank%';
 datname |                    query                    |          xact_start           |    duration     
---------+---------------------------------------------+-------------------------------+-----------------
 wres8   | ANALYZE wres.timeseriesvalue_lead_above_150 | 2019-08-05 18:23:11.727733+00 | 00:11:43.402944
(1 row)

Sizes of tables and indices:

wres8=> \dti+ wres.*;
                                                                    List of relations
 Schema |                         Name                          | Type  |   Owner    |                 Table                  |    Size    | Description 
--------+-------------------------------------------------------+-------+------------+----------------------------------------+------------+-------------
 wres   | ensemble                                              | table | wres       |                                        | 48 kB      | 
 wres   | ensemble_pk                                           | index | wres       | ensemble                               | 16 kB      | 
 wres   | executionlog                                          | table | wres       |                                        | 200 kB     | 
 wres   | feature                                               | table | wres       |                                        | 1184 kB    | 
 wres   | feature_lid_idx                                       | index | wres       | feature                                | 336 kB     | 
 wres   | feature_pk                                            | index | wres       | feature                                | 248 kB     | 
 wres   | gridprojection                                        | table | wres       |                                        | 8192 bytes | 
 wres   | gridprojection_pk                                     | index | wres       | gridprojection                         | 8192 bytes | 
 wres   | indexqueue                                            | table | wres       |                                        | 8192 bytes | 
 wres   | indexqueue_pk                                         | index | wres       | indexqueue                             | 8192 bytes | 
 wres   | measurementunit                                       | table | wres       |                                        | 16 kB      | 
 wres   | measurementunit_unit_name_key                         | index | wres       | measurementunit                        | 16 kB      | 
 wres   | netcdfcoordinate                                      | table | wres       |                                        | 0 bytes    | 
 wres   | netcdfcoordinate_coordinate_idx                       | index | wres       | netcdfcoordinate                       | 8192 bytes | 
 wres   | observation                                           | table | wres       |                                        | 1430 MB    | 
 wres   | observation_source_idx                                | index | wres       | observation                            | 597 MB     | 
 wres   | observation_time_idx                                  | index | wres       | observation                            | 621 MB     | 
 wres   | observation_variablefeature_idx                       | index | wres       | observation                            | 605 MB     | 
 wres   | pk_measurementunit                                    | index | wres       | measurementunit                        | 16 kB      | 
 wres   | project                                               | table | wres       |                                        | 16 kB      | 
 wres   | project_input_code_key                                | index | wres       | project                                | 16 kB      | 
 wres   | project_pk                                            | index | wres       | project                                | 16 kB      | 
 wres   | projectsource                                         | table | wres       |                                        | 3464 kB    | 
 wres   | projectsource_project_idx                             | index | wres       | projectsource                          | 2280 kB    | 
 wres   | projectsource_source_idx                              | index | wres       | projectsource                          | 2232 kB    | 
 wres   | source                                                | table | wres       |                                        | 4048 kB    | 
 wres   | source_hash_idx                                       | index | wres       | source                                 | 1016 kB    | 
 wres   | source_output_time_idx                                | index | wres       | source                                 | 320 kB     | 
 wres   | source_path_idx                                       | index | wres       | source                                 | 3448 kB    | 
 wres   | source_pk                                             | index | wres       | source                                 | 296 kB     | 
 wres   | sourcecompleted                                       | table | wres       |                                        | 968 kB     | 
 wres   | sourcecompleted_source_idx                            | index | wres       | sourcecompleted                        | 296 kB     | 
 wres   | timeseries                                            | table | wres       |                                        | 29 MB      | 
 wres   | timeseries_pk                                         | index | wres       | timeseries                             | 11 MB      | 
 wres   | timeseries_source_idx                                 | index | wres       | timeseriessource                       | 463 MB     | 
 wres   | timeseries_variablefeature_ix                         | index | wres       | timeseries                             | 14 MB      | 
 wres   | timeseriessource                                      | table | wres       |                                        | 704 MB     | 
 wres   | timeseriesvalue                                       | table | wres       |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_0                                | table | wres       |                                        | 105 MB     | 
 wres   | timeseriesvalue_lead_0_lead_idx                       | index | wres       | timeseriesvalue_lead_0                 | 68 MB      | 
 wres   | timeseriesvalue_lead_0_timeseries_idx                 | index | wres       | timeseriesvalue_lead_0                 | 62 MB      | 
 wres   | timeseriesvalue_lead_1                                | table | wres       |                                        | 113 MB     | 
 wres   | timeseriesvalue_lead_10                               | table | wres_user8 |                                        | 50 MB      | 
 wres   | timeseriesvalue_lead_100                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_100_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_100               | 17 MB      | 
 wres   | timeseriesvalue_lead_100_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_100               | 14 MB      | 
 wres   | timeseriesvalue_lead_101                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_101_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_101               | 17 MB      | 
 wres   | timeseriesvalue_lead_101_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_101               | 14 MB      | 
 wres   | timeseriesvalue_lead_102                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_102_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_102               | 23 MB      | 
 wres   | timeseriesvalue_lead_102_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_102               | 18 MB      | 
 wres   | timeseriesvalue_lead_103                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_103_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_103               | 17 MB      | 
 wres   | timeseriesvalue_lead_103_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_103               | 14 MB      | 
 wres   | timeseriesvalue_lead_104                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_104_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_104               | 17 MB      | 
 wres   | timeseriesvalue_lead_104_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_104               | 14 MB      | 
 wres   | timeseriesvalue_lead_105                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_105_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_105               | 23 MB      | 
 wres   | timeseriesvalue_lead_105_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_105               | 18 MB      | 
 wres   | timeseriesvalue_lead_106                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_106_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_106               | 17 MB      | 
 wres   | timeseriesvalue_lead_106_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_106               | 14 MB      | 
 wres   | timeseriesvalue_lead_107                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_107_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_107               | 17 MB      | 
 wres   | timeseriesvalue_lead_107_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_107               | 14 MB      | 
 wres   | timeseriesvalue_lead_108                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_108_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_108               | 23 MB      | 
 wres   | timeseriesvalue_lead_108_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_108               | 18 MB      | 
 wres   | timeseriesvalue_lead_109                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_109_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_109               | 17 MB      | 
 wres   | timeseriesvalue_lead_109_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_109               | 14 MB      | 
 wres   | timeseriesvalue_lead_10_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_10                | 32 MB      | 
 wres   | timeseriesvalue_lead_10_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_10                | 28 MB      | 
 wres   | timeseriesvalue_lead_11                               | table | wres_user8 |                                        | 29 MB      | 
 wres   | timeseriesvalue_lead_110                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_110_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_110               | 17 MB      | 
 wres   | timeseriesvalue_lead_110_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_110               | 14 MB      | 
 wres   | timeseriesvalue_lead_111                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_111_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_111               | 23 MB      | 
 wres   | timeseriesvalue_lead_111_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_111               | 18 MB      | 
 wres   | timeseriesvalue_lead_112                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_112_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_112               | 17 MB      | 
 wres   | timeseriesvalue_lead_112_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_112               | 14 MB      | 
 wres   | timeseriesvalue_lead_113                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_113_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_113               | 17 MB      | 
 wres   | timeseriesvalue_lead_113_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_113               | 14 MB      | 
 wres   | timeseriesvalue_lead_114                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_114_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_114               | 23 MB      | 
 wres   | timeseriesvalue_lead_114_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_114               | 18 MB      | 
 wres   | timeseriesvalue_lead_115                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_115_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_115               | 17 MB      | 
 wres   | timeseriesvalue_lead_115_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_115               | 14 MB      | 
 wres   | timeseriesvalue_lead_116                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_116_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_116               | 17 MB      | 
 wres   | timeseriesvalue_lead_116_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_116               | 14 MB      | 
 wres   | timeseriesvalue_lead_117                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_117_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_117               | 23 MB      | 
 wres   | timeseriesvalue_lead_117_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_117               | 18 MB      | 
 wres   | timeseriesvalue_lead_118                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_118_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_118               | 17 MB      | 
 wres   | timeseriesvalue_lead_118_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_118               | 14 MB      | 
 wres   | timeseriesvalue_lead_119                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_119_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_119               | 17 MB      | 
 wres   | timeseriesvalue_lead_119_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_119               | 14 MB      | 
 wres   | timeseriesvalue_lead_11_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_11                | 18 MB      | 
 wres   | timeseriesvalue_lead_11_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_11                | 14 MB      | 
 wres   | timeseriesvalue_lead_12                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_120                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_120_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_120               | 23 MB      | 
 wres   | timeseriesvalue_lead_120_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_120               | 18 MB      | 
 wres   | timeseriesvalue_lead_121                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_121_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_121               | 17 MB      | 
 wres   | timeseriesvalue_lead_121_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_121               | 14 MB      | 
 wres   | timeseriesvalue_lead_122                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_122_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_122               | 17 MB      | 
 wres   | timeseriesvalue_lead_122_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_122               | 14 MB      | 
 wres   | timeseriesvalue_lead_123                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_123_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_123               | 23 MB      | 
 wres   | timeseriesvalue_lead_123_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_123               | 18 MB      | 
 wres   | timeseriesvalue_lead_124                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_124_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_124               | 17 MB      | 
 wres   | timeseriesvalue_lead_124_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_124               | 14 MB      | 
 wres   | timeseriesvalue_lead_125                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_125_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_125               | 17 MB      | 
 wres   | timeseriesvalue_lead_125_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_125               | 14 MB      | 
 wres   | timeseriesvalue_lead_126                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_126_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_126               | 23 MB      | 
 wres   | timeseriesvalue_lead_126_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_126               | 18 MB      | 
 wres   | timeseriesvalue_lead_127                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_127_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_127               | 17 MB      | 
 wres   | timeseriesvalue_lead_127_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_127               | 14 MB      | 
 wres   | timeseriesvalue_lead_128                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_128_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_128               | 17 MB      | 
 wres   | timeseriesvalue_lead_128_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_128               | 14 MB      | 
 wres   | timeseriesvalue_lead_129                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_129_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_129               | 23 MB      | 
 wres   | timeseriesvalue_lead_129_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_129               | 18 MB      | 
 wres   | timeseriesvalue_lead_12_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_12                | 23 MB      | 
 wres   | timeseriesvalue_lead_12_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_12                | 18 MB      | 
 wres   | timeseriesvalue_lead_13                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_130                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_130_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_130               | 17 MB      | 
 wres   | timeseriesvalue_lead_130_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_130               | 14 MB      | 
 wres   | timeseriesvalue_lead_131                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_131_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_131               | 17 MB      | 
 wres   | timeseriesvalue_lead_131_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_131               | 14 MB      | 
 wres   | timeseriesvalue_lead_132                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_132_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_132               | 23 MB      | 
 wres   | timeseriesvalue_lead_132_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_132               | 18 MB      | 
 wres   | timeseriesvalue_lead_133                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_133_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_133               | 17 MB      | 
 wres   | timeseriesvalue_lead_133_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_133               | 14 MB      | 
 wres   | timeseriesvalue_lead_134                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_134_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_134               | 17 MB      | 
 wres   | timeseriesvalue_lead_134_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_134               | 14 MB      | 
 wres   | timeseriesvalue_lead_135                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_135_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_135               | 23 MB      | 
 wres   | timeseriesvalue_lead_135_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_135               | 18 MB      | 
 wres   | timeseriesvalue_lead_136                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_136_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_136               | 17 MB      | 
 wres   | timeseriesvalue_lead_136_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_136               | 14 MB      | 
 wres   | timeseriesvalue_lead_137                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_137_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_137               | 17 MB      | 
 wres   | timeseriesvalue_lead_137_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_137               | 14 MB      | 
 wres   | timeseriesvalue_lead_138                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_138_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_138               | 23 MB      | 
 wres   | timeseriesvalue_lead_138_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_138               | 18 MB      | 
 wres   | timeseriesvalue_lead_139                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_139_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_139               | 17 MB      | 
 wres   | timeseriesvalue_lead_139_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_139               | 14 MB      | 
 wres   | timeseriesvalue_lead_13_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_13                | 17 MB      | 
 wres   | timeseriesvalue_lead_13_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_13                | 14 MB      | 
 wres   | timeseriesvalue_lead_14                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_140                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_140_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_140               | 17 MB      | 
 wres   | timeseriesvalue_lead_140_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_140               | 14 MB      | 
 wres   | timeseriesvalue_lead_141                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_141_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_141               | 23 MB      | 
 wres   | timeseriesvalue_lead_141_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_141               | 18 MB      | 
 wres   | timeseriesvalue_lead_142                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_142_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_142               | 17 MB      | 
 wres   | timeseriesvalue_lead_142_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_142               | 14 MB      | 
 wres   | timeseriesvalue_lead_143                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_143_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_143               | 17 MB      | 
 wres   | timeseriesvalue_lead_143_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_143               | 14 MB      | 
 wres   | timeseriesvalue_lead_144                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_144_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_144               | 23 MB      | 
 wres   | timeseriesvalue_lead_144_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_144               | 18 MB      | 
 wres   | timeseriesvalue_lead_145                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_145_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_145               | 17 MB      | 
 wres   | timeseriesvalue_lead_145_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_145               | 14 MB      | 
 wres   | timeseriesvalue_lead_146                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_146_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_146               | 17 MB      | 
 wres   | timeseriesvalue_lead_146_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_146               | 14 MB      | 
 wres   | timeseriesvalue_lead_147                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_147_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_147               | 23 MB      | 
 wres   | timeseriesvalue_lead_147_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_147               | 18 MB      | 
 wres   | timeseriesvalue_lead_148                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_148_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_148               | 17 MB      | 
 wres   | timeseriesvalue_lead_148_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_148               | 14 MB      | 
 wres   | timeseriesvalue_lead_149                              | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_149_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_149               | 17 MB      | 
 wres   | timeseriesvalue_lead_149_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_149               | 14 MB      | 
 wres   | timeseriesvalue_lead_14_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_14                | 17 MB      | 
 wres   | timeseriesvalue_lead_14_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_14                | 14 MB      | 
 wres   | timeseriesvalue_lead_15                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_150                              | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_150_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_150               | 23 MB      | 
 wres   | timeseriesvalue_lead_150_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_150               | 18 MB      | 
 wres   | timeseriesvalue_lead_15_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_15                | 23 MB      | 
 wres   | timeseriesvalue_lead_15_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_15                | 18 MB      | 
 wres   | timeseriesvalue_lead_16                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_16_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_16                | 17 MB      | 
 wres   | timeseriesvalue_lead_16_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_16                | 14 MB      | 
 wres   | timeseriesvalue_lead_17                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_17_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_17                | 17 MB      | 
 wres   | timeseriesvalue_lead_17_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_17                | 14 MB      | 
 wres   | timeseriesvalue_lead_18                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_18_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_18                | 23 MB      | 
 wres   | timeseriesvalue_lead_18_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_18                | 18 MB      | 
 wres   | timeseriesvalue_lead_19                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_19_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_19                | 17 MB      | 
 wres   | timeseriesvalue_lead_19_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_19                | 14 MB      | 
 wres   | timeseriesvalue_lead_1_lead_idx                       | index | wres       | timeseriesvalue_lead_1                 | 73 MB      | 
 wres   | timeseriesvalue_lead_1_timeseries_idx                 | index | wres       | timeseriesvalue_lead_1                 | 69 MB      | 
 wres   | timeseriesvalue_lead_2                                | table | wres       |                                        | 102 MB     | 
 wres   | timeseriesvalue_lead_20                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_20_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_20                | 17 MB      | 
 wres   | timeseriesvalue_lead_20_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_20                | 14 MB      | 
 wres   | timeseriesvalue_lead_21                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_21_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_21                | 23 MB      | 
 wres   | timeseriesvalue_lead_21_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_21                | 18 MB      | 
 wres   | timeseriesvalue_lead_22                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_22_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_22                | 17 MB      | 
 wres   | timeseriesvalue_lead_22_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_22                | 14 MB      | 
 wres   | timeseriesvalue_lead_23                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_23_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_23                | 17 MB      | 
 wres   | timeseriesvalue_lead_23_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_23                | 14 MB      | 
 wres   | timeseriesvalue_lead_24                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_24_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_24                | 23 MB      | 
 wres   | timeseriesvalue_lead_24_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_24                | 18 MB      | 
 wres   | timeseriesvalue_lead_25                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_25_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_25                | 17 MB      | 
 wres   | timeseriesvalue_lead_25_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_25                | 14 MB      | 
 wres   | timeseriesvalue_lead_26                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_26_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_26                | 17 MB      | 
 wres   | timeseriesvalue_lead_26_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_26                | 14 MB      | 
 wres   | timeseriesvalue_lead_27                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_27_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_27                | 23 MB      | 
 wres   | timeseriesvalue_lead_27_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_27                | 18 MB      | 
 wres   | timeseriesvalue_lead_28                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_28_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_28                | 17 MB      | 
 wres   | timeseriesvalue_lead_28_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_28                | 14 MB      | 
 wres   | timeseriesvalue_lead_29                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_29_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_29                | 17 MB      | 
 wres   | timeseriesvalue_lead_29_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_29                | 14 MB      | 
 wres   | timeseriesvalue_lead_2_lead_idx                       | index | wres       | timeseriesvalue_lead_2                 | 66 MB      | 
 wres   | timeseriesvalue_lead_2_timeseries_idx                 | index | wres       | timeseriesvalue_lead_2                 | 62 MB      | 
 wres   | timeseriesvalue_lead_3                                | table | wres_user8 |                                        | 120 MB     | 
 wres   | timeseriesvalue_lead_30                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_30_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_30                | 23 MB      | 
 wres   | timeseriesvalue_lead_30_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_30                | 18 MB      | 
 wres   | timeseriesvalue_lead_31                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_31_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_31                | 17 MB      | 
 wres   | timeseriesvalue_lead_31_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_31                | 14 MB      | 
 wres   | timeseriesvalue_lead_32                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_32_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_32                | 17 MB      | 
 wres   | timeseriesvalue_lead_32_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_32                | 14 MB      | 
 wres   | timeseriesvalue_lead_33                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_33_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_33                | 23 MB      | 
 wres   | timeseriesvalue_lead_33_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_33                | 18 MB      | 
 wres   | timeseriesvalue_lead_34                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_34_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_34                | 17 MB      | 
 wres   | timeseriesvalue_lead_34_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_34                | 14 MB      | 
 wres   | timeseriesvalue_lead_35                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_35_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_35                | 17 MB      | 
 wres   | timeseriesvalue_lead_35_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_35                | 14 MB      | 
 wres   | timeseriesvalue_lead_36                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_36_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_36                | 23 MB      | 
 wres   | timeseriesvalue_lead_36_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_36                | 18 MB      | 
 wres   | timeseriesvalue_lead_37                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_37_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_37                | 17 MB      | 
 wres   | timeseriesvalue_lead_37_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_37                | 14 MB      | 
 wres   | timeseriesvalue_lead_38                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_38_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_38                | 17 MB      | 
 wres   | timeseriesvalue_lead_38_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_38                | 14 MB      | 
 wres   | timeseriesvalue_lead_39                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_39_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_39                | 23 MB      | 
 wres   | timeseriesvalue_lead_39_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_39                | 18 MB      | 
 wres   | timeseriesvalue_lead_3_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_3                 | 78 MB      | 
 wres   | timeseriesvalue_lead_3_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_3                 | 71 MB      | 
 wres   | timeseriesvalue_lead_4                                | table | wres_user8 |                                        | 111 MB     | 
 wres   | timeseriesvalue_lead_40                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_40_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_40                | 17 MB      | 
 wres   | timeseriesvalue_lead_40_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_40                | 14 MB      | 
 wres   | timeseriesvalue_lead_41                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_41_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_41                | 17 MB      | 
 wres   | timeseriesvalue_lead_41_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_41                | 14 MB      | 
 wres   | timeseriesvalue_lead_42                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_42_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_42                | 23 MB      | 
 wres   | timeseriesvalue_lead_42_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_42                | 18 MB      | 
 wres   | timeseriesvalue_lead_43                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_43_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_43                | 17 MB      | 
 wres   | timeseriesvalue_lead_43_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_43                | 14 MB      | 
 wres   | timeseriesvalue_lead_44                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_44_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_44                | 17 MB      | 
 wres   | timeseriesvalue_lead_44_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_44                | 14 MB      | 
 wres   | timeseriesvalue_lead_45                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_45_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_45                | 23 MB      | 
 wres   | timeseriesvalue_lead_45_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_45                | 18 MB      | 
 wres   | timeseriesvalue_lead_46                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_46_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_46                | 17 MB      | 
 wres   | timeseriesvalue_lead_46_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_46                | 14 MB      | 
 wres   | timeseriesvalue_lead_47                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_47_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_47                | 17 MB      | 
 wres   | timeseriesvalue_lead_47_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_47                | 14 MB      | 
 wres   | timeseriesvalue_lead_48                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_48_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_48                | 23 MB      | 
 wres   | timeseriesvalue_lead_48_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_48                | 18 MB      | 
 wres   | timeseriesvalue_lead_49                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_49_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_49                | 17 MB      | 
 wres   | timeseriesvalue_lead_49_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_49                | 14 MB      | 
 wres   | timeseriesvalue_lead_4_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_4                 | 72 MB      | 
 wres   | timeseriesvalue_lead_4_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_4                 | 68 MB      | 
 wres   | timeseriesvalue_lead_5                                | table | wres_user8 |                                        | 100 MB     | 
 wres   | timeseriesvalue_lead_50                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_50_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_50                | 17 MB      | 
 wres   | timeseriesvalue_lead_50_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_50                | 14 MB      | 
 wres   | timeseriesvalue_lead_51                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_51_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_51                | 23 MB      | 
 wres   | timeseriesvalue_lead_51_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_51                | 18 MB      | 
 wres   | timeseriesvalue_lead_52                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_52_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_52                | 17 MB      | 
 wres   | timeseriesvalue_lead_52_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_52                | 14 MB      | 
 wres   | timeseriesvalue_lead_53                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_53_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_53                | 17 MB      | 
 wres   | timeseriesvalue_lead_53_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_53                | 14 MB      | 
 wres   | timeseriesvalue_lead_54                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_54_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_54                | 23 MB      | 
 wres   | timeseriesvalue_lead_54_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_54                | 18 MB      | 
 wres   | timeseriesvalue_lead_55                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_55_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_55                | 17 MB      | 
 wres   | timeseriesvalue_lead_55_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_55                | 14 MB      | 
 wres   | timeseriesvalue_lead_56                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_56_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_56                | 17 MB      | 
 wres   | timeseriesvalue_lead_56_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_56                | 14 MB      | 
 wres   | timeseriesvalue_lead_57                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_57_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_57                | 23 MB      | 
 wres   | timeseriesvalue_lead_57_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_57                | 18 MB      | 
 wres   | timeseriesvalue_lead_58                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_58_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_58                | 17 MB      | 
 wres   | timeseriesvalue_lead_58_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_58                | 14 MB      | 
 wres   | timeseriesvalue_lead_59                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_59_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_59                | 17 MB      | 
 wres   | timeseriesvalue_lead_59_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_59                | 14 MB      | 
 wres   | timeseriesvalue_lead_5_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_5                 | 65 MB      | 
 wres   | timeseriesvalue_lead_5_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_5                 | 60 MB      | 
 wres   | timeseriesvalue_lead_6                                | table | wres_user8 |                                        | 114 MB     | 
 wres   | timeseriesvalue_lead_60                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_60_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_60                | 23 MB      | 
 wres   | timeseriesvalue_lead_60_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_60                | 18 MB      | 
 wres   | timeseriesvalue_lead_61                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_61_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_61                | 17 MB      | 
 wres   | timeseriesvalue_lead_61_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_61                | 14 MB      | 
 wres   | timeseriesvalue_lead_62                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_62_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_62                | 17 MB      | 
 wres   | timeseriesvalue_lead_62_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_62                | 14 MB      | 
 wres   | timeseriesvalue_lead_63                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_63_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_63                | 23 MB      | 
 wres   | timeseriesvalue_lead_63_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_63                | 18 MB      | 
 wres   | timeseriesvalue_lead_64                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_64_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_64                | 17 MB      | 
 wres   | timeseriesvalue_lead_64_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_64                | 14 MB      | 
 wres   | timeseriesvalue_lead_65                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_65_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_65                | 17 MB      | 
 wres   | timeseriesvalue_lead_65_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_65                | 14 MB      | 
 wres   | timeseriesvalue_lead_66                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_66_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_66                | 23 MB      | 
 wres   | timeseriesvalue_lead_66_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_66                | 18 MB      | 
 wres   | timeseriesvalue_lead_67                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_67_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_67                | 17 MB      | 
 wres   | timeseriesvalue_lead_67_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_67                | 14 MB      | 
 wres   | timeseriesvalue_lead_68                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_68_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_68                | 17 MB      | 
 wres   | timeseriesvalue_lead_68_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_68                | 14 MB      | 
 wres   | timeseriesvalue_lead_69                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_69_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_69                | 23 MB      | 
 wres   | timeseriesvalue_lead_69_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_69                | 18 MB      | 
 wres   | timeseriesvalue_lead_6_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_6                 | 73 MB      | 
 wres   | timeseriesvalue_lead_6_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_6                 | 68 MB      | 
 wres   | timeseriesvalue_lead_7                                | table | wres_user8 |                                        | 105 MB     | 
 wres   | timeseriesvalue_lead_70                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_70_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_70                | 17 MB      | 
 wres   | timeseriesvalue_lead_70_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_70                | 14 MB      | 
 wres   | timeseriesvalue_lead_71                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_71_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_71                | 17 MB      | 
 wres   | timeseriesvalue_lead_71_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_71                | 14 MB      | 
 wres   | timeseriesvalue_lead_72                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_72_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_72                | 23 MB      | 
 wres   | timeseriesvalue_lead_72_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_72                | 18 MB      | 
 wres   | timeseriesvalue_lead_73                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_73_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_73                | 17 MB      | 
 wres   | timeseriesvalue_lead_73_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_73                | 14 MB      | 
 wres   | timeseriesvalue_lead_74                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_74_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_74                | 17 MB      | 
 wres   | timeseriesvalue_lead_74_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_74                | 14 MB      | 
 wres   | timeseriesvalue_lead_75                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_75_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_75                | 23 MB      | 
 wres   | timeseriesvalue_lead_75_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_75                | 18 MB      | 
 wres   | timeseriesvalue_lead_76                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_76_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_76                | 17 MB      | 
 wres   | timeseriesvalue_lead_76_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_76                | 14 MB      | 
 wres   | timeseriesvalue_lead_77                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_77_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_77                | 17 MB      | 
 wres   | timeseriesvalue_lead_77_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_77                | 14 MB      | 
 wres   | timeseriesvalue_lead_78                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_78_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_78                | 23 MB      | 
 wres   | timeseriesvalue_lead_78_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_78                | 18 MB      | 
 wres   | timeseriesvalue_lead_79                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_79_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_79                | 17 MB      | 
 wres   | timeseriesvalue_lead_79_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_79                | 14 MB      | 
 wres   | timeseriesvalue_lead_7_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_7                 | 68 MB      | 
 wres   | timeseriesvalue_lead_7_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_7                 | 64 MB      | 
 wres   | timeseriesvalue_lead_8                                | table | wres_user8 |                                        | 92 MB      | 
 wres   | timeseriesvalue_lead_80                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_80_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_80                | 17 MB      | 
 wres   | timeseriesvalue_lead_80_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_80                | 14 MB      | 
 wres   | timeseriesvalue_lead_81                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_81_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_81                | 23 MB      | 
 wres   | timeseriesvalue_lead_81_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_81                | 18 MB      | 
 wres   | timeseriesvalue_lead_82                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_82_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_82                | 17 MB      | 
 wres   | timeseriesvalue_lead_82_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_82                | 14 MB      | 
 wres   | timeseriesvalue_lead_83                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_83_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_83                | 17 MB      | 
 wres   | timeseriesvalue_lead_83_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_83                | 14 MB      | 
 wres   | timeseriesvalue_lead_84                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_84_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_84                | 23 MB      | 
 wres   | timeseriesvalue_lead_84_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_84                | 18 MB      | 
 wres   | timeseriesvalue_lead_85                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_85_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_85                | 17 MB      | 
 wres   | timeseriesvalue_lead_85_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_85                | 14 MB      | 
 wres   | timeseriesvalue_lead_86                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_86_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_86                | 17 MB      | 
 wres   | timeseriesvalue_lead_86_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_86                | 14 MB      | 
 wres   | timeseriesvalue_lead_87                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_87_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_87                | 23 MB      | 
 wres   | timeseriesvalue_lead_87_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_87                | 18 MB      | 
 wres   | timeseriesvalue_lead_88                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_88_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_88                | 17 MB      | 
 wres   | timeseriesvalue_lead_88_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_88                | 14 MB      | 
 wres   | timeseriesvalue_lead_89                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_89_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_89                | 17 MB      | 
 wres   | timeseriesvalue_lead_89_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_89                | 14 MB      | 
 wres   | timeseriesvalue_lead_8_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_8                 | 59 MB      | 
 wres   | timeseriesvalue_lead_8_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_8                 | 56 MB      | 
 wres   | timeseriesvalue_lead_9                                | table | wres_user8 |                                        | 109 MB     | 
 wres   | timeseriesvalue_lead_90                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_90_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_90                | 23 MB      | 
 wres   | timeseriesvalue_lead_90_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_90                | 18 MB      | 
 wres   | timeseriesvalue_lead_91                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_91_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_91                | 17 MB      | 
 wres   | timeseriesvalue_lead_91_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_91                | 14 MB      | 
 wres   | timeseriesvalue_lead_92                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_92_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_92                | 17 MB      | 
 wres   | timeseriesvalue_lead_92_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_92                | 14 MB      | 
 wres   | timeseriesvalue_lead_93                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_93_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_93                | 23 MB      | 
 wres   | timeseriesvalue_lead_93_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_93                | 18 MB      | 
 wres   | timeseriesvalue_lead_94                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_94_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_94                | 17 MB      | 
 wres   | timeseriesvalue_lead_94_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_94                | 14 MB      | 
 wres   | timeseriesvalue_lead_95                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_95_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_95                | 17 MB      | 
 wres   | timeseriesvalue_lead_95_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_95                | 14 MB      | 
 wres   | timeseriesvalue_lead_96                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_96_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_96                | 23 MB      | 
 wres   | timeseriesvalue_lead_96_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_96                | 18 MB      | 
 wres   | timeseriesvalue_lead_97                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_97_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_97                | 17 MB      | 
 wres   | timeseriesvalue_lead_97_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_97                | 14 MB      | 
 wres   | timeseriesvalue_lead_98                               | table | wres_user8 |                                        | 27 MB      | 
 wres   | timeseriesvalue_lead_98_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_98                | 17 MB      | 
 wres   | timeseriesvalue_lead_98_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_98                | 14 MB      | 
 wres   | timeseriesvalue_lead_99                               | table | wres_user8 |                                        | 36 MB      | 
 wres   | timeseriesvalue_lead_99_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_99                | 23 MB      | 
 wres   | timeseriesvalue_lead_99_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_99                | 18 MB      | 
 wres   | timeseriesvalue_lead_9_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_9                 | 70 MB      | 
 wres   | timeseriesvalue_lead_9_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_9                 | 66 MB      | 
 wres   | timeseriesvalue_lead_above_150                        | table | wres_user8 |                                        | 14 GB      | 
 wres   | timeseriesvalue_lead_above_150_lead_idx               | index | wres_user8 | timeseriesvalue_lead_above_150         | 9518 MB    | 
 wres   | timeseriesvalue_lead_above_150_timeseries_idx         | index | wres_user8 | timeseriesvalue_lead_above_150         | 9818 MB    | 
 wres   | timeseriesvalue_lead_below_negative_10                | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_below_negative_10_lead_idx       | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 8192 bytes | 
 wres   | timeseriesvalue_lead_below_negative_10_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_1                       | table | wres_user8 |                                        | 40 kB      | 
 wres   | timeseriesvalue_lead_negative_10                      | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_10_lead_idx             | index | wres_user8 | timeseriesvalue_lead_negative_10       | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_10_timeseries_idx       | index | wres_user8 | timeseriesvalue_lead_negative_10       | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_1_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_1        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_1_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_1        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_2                       | table | wres_user8 |                                        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_2_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_2        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_2_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_2        | 16 kB      | 
 wres   | timeseriesvalue_lead_negative_3                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_3_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_3        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_3_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_3        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_4                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_4_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_4        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_4_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_4        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_5                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_5_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_5        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_5_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_5        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_6                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_6_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_6        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_6_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_6        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_7                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_7_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_7        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_7_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_7        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_8                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_8_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_8        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_8_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_8        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_9                       | table | wres_user8 |                                        | 0 bytes    | 
 wres   | timeseriesvalue_lead_negative_9_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_9        | 8192 bytes | 
 wres   | timeseriesvalue_lead_negative_9_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_9        | 8192 bytes | 
 wres   | timeseriesvalue_timeseries_idx                        | index | wres       | timeseriesvalue                        | 8192 bytes | 
 wres   | unitconversion                                        | table | wres       |                                        | 56 kB      | 
 wres   | unitconversion_measurement_uix                        | index | wres       | unitconversion                         | 32 kB      | 
 wres   | usgsparameter                                         | table | wres       |                                        | 16 kB      | 
 wres   | usgsparameter_name_agg_measurement_idx                | index | wres       | usgsparameter                          | 16 kB      | 
 wres   | variable                                              | table | wres       |                                        | 16 kB      | 
 wres   | variable_pk                                           | index | wres       | variable                               | 16 kB      | 
 wres   | variable_variable_name_idx                            | index | wres       | variable                               | 16 kB      | 
 wres   | variable_variable_name_key                            | index | wres       | variable                               | 16 kB      | 
 wres   | variablefeature                                       | table | wres       |                                        | 624 kB     | 
 wres   | variablefeature_feature_uix                           | index | wres       | variablefeature                        | 328 kB     | 
 wres   | variablefeature_pk                                    | index | wres       | variablefeature                        | 256 kB     | 
(539 rows)

Four prior datasets present:

wres8=> select * from wres.project;
 project_id | input_code  |                       project_name                       
------------+-------------+----------------------------------------------------------
          1 | -2117656521 | esp flavor eval
          2 | -1847996207 | AHPS 144 hours Evaluation
          3 |    75345536 | AHPS 144 hours Evaluation
          4 | -1745458678 | AHPS 144 hours Evaluation
          5 | -1645977094 | V2.0 MRF 20181202-20181205 1_Basin_Ens_Metrics WRES_1.13
(5 rows)
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-08-05T18:59:21Z


wres8=> select datname, query, xact_start, now() - xact_start as duration from pg_stat_activity where state = 'active' and query not like '%schnitzelbank%';
 datname |                    query                    |          xact_start           |    duration    
---------+---------------------------------------------+-------------------------------+----------------
 wres8   | ANALYZE wres.timeseriesvalue_lead_above_150 | 2019-08-05 18:23:11.727733+00 | 00:33:29.23338
(1 row)
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-08-05T20:36:35Z


Taking dump of wres8 before cleaning db.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-08-05T21:04:53Z


Can find dumps with prefix @/wres_share/dbdumps/rdb_wres8_20190805@ on the ti01.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-08-05T21:06:44Z


Running clean... 5168251927000198665 on prod-like env.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-08-05T21:06:59Z


@INFO Main The function 'cleandatabase' took PT1M25.197345S@

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-10-15T21:48:58Z


Took another wres8 dump:

scripts/pg_dump_wres.sh -f /wres_share/dbdumps/rdb_wres8_20191015 -h nwcal-rdb.[host] -d wres8 -U wres_user8 -p /usr/pgsql-9.6/bin/pg_dump
Creating dump_file /wres_share/dbdumps/rdb_wres8_20191015.pgdump (does not yet exist)
Creating changelog_dump_file /wres_share/dbdumps/rdb_wres8_20191015_changelog.pgdump (does not yet exist)
Using pg_dump executable /usr/pgsql-9.6/bin/pg_dump (exists) (executable)
Using database host nwcal-rdb.[host] (resolves)
Using database name wres8
Using database username wres_user8
Please ctrl-c if that is not correct, any key otherwise...
2019-10-15T21:35:13,608358650+0000
++ /usr/pgsql-9.6/bin/pg_dump -h nwcal-rdb.[host] -d wres8 -Fc -U wres_user8 -n wres
pg_dump: LOG:  duration: 594874.755 ms  statement: COPY wres.observation (variablefeature_id, observation_time, observed_value, measurementunit_id, source_id, scale_period, scale_function) TO stdout;
++ /usr/pgsql-9.6/bin/pg_dump -h nwcal-rdb.[host] -d wres8 -Fc -U wres_user8 -n public --table databasechangelog --table databasechangeloglock
++ set +x
2019-10-15T21:46:59,516899482+0000
Dump took around 706 seconds
-rw-rw-r--. 1 user.name wres 229K Oct 15 21:46 /wres_share/dbdumps/rdb_wres8_20191015_changelog.pgdump
-rw-rw-r--. 1 user.name wres 1.3G Oct 15 21:46 /wres_share/dbdumps/rdb_wres8_20191015.pgdump
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-10-15T21:50:13Z


Reason for taking a backup of the wres8 from rdb? User said

I tried to run my evaluations today again but it was taking forever. So I decided to kill them and run them without using COWRES

I would like to run a clean database once I see zero evaluations running (which may be a while, because evaluations are indeed taking many hours).

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2019-12-18T21:41:45Z


I ran a clean database operation (through the front door of COWRES, using script) about a half hour ago because the ANALYZE operation on wres.observation was taking a long time, the table was 5900MiB, and I was going to show Ross how to use COWRES to get several years of USGS data.

epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-17T19:16:32Z


Mitigated to some extent by (edit: changes up through) 5.0: Fewer joins (fewer tables), simplified up-front queries, using auto-analyze instead of explicit analyze.

Prod is now ~232GiB but we can run evaluations that take under thirty seconds to complete.

Tables, several of which exceed the 7.3GiB shared_buffers:

wres8=> \dti+ wres.*;
                                                                    List of relations
 Schema |                         Name                          | Type  |   Owner    |                 Table                  |    Size    | Description
--------+-------------------------------------------------------+-------+------------+----------------------------------------+------------+-------------
 wres   | ensemble                                              | table | wres       |                                        | 48 kB      |
 wres   | ensemble_pk                                           | index | wres       | ensemble                               | 16 kB      |
 wres   | executionlog                                          | table | wres       |                                        | 45 MB      |
 wres   | feature                                               | table | wres_user8 |                                        | 1152 kB    |
 wres   | feature_name_description_srid_wkt_key                 | index | wres_user8 | feature                                | 1096 kB    |
 wres   | feature_pkey                                          | index | wres_user8 | feature                                | 240 kB     |
 wres   | gridprojection                                        | table | wres       |                                        | 16 kB      |
 wres   | gridprojection_pk                                     | index | wres       | gridprojection                         | 16 kB      |
 wres   | measurementunit                                       | table | wres       |                                        | 48 kB      |
 wres   | measurementunit_pkey                                  | index | wres       | measurementunit                        | 16 kB      |
 wres   | measurementunit_unit_name_key                         | index | wres       | measurementunit                        | 16 kB      |
 wres   | netcdfcoordinate                                      | table | wres       |                                        | 1152 MB    |
 wres   | netcdfcoordinate_coordinate_idx                       | index | wres       | netcdfcoordinate                       | 1534 MB    |
 wres   | project                                               | table | wres       |                                        | 80 kB      |
 wres   | project_input_code_key                                | index | wres       | project                                | 40 kB      |
 wres   | project_pk                                            | index | wres       | project                                | 32 kB      |
 wres   | projectsource                                         | table | wres       |                                        | 1339 MB    |
 wres   | projectsource_project_idx                             | index | wres       | projectsource                          | 1585 MB    |
 wres   | projectsource_source_idx                              | index | wres       | projectsource                          | 1195 MB    |
 wres   | source                                                | table | wres       |                                        | 2355 MB    |
 wres   | source_hash_idx                                       | index | wres       | source                                 | 392 MB     |
 wres   | source_output_time_idx                                | index | wres       | source                                 | 119 MB     |
 wres   | source_path_idx                                       | index | wres       | source                                 | 4382 MB    |
 wres   | source_pk                                             | index | wres       | source                                 | 115 MB     |
 wres   | sourcecompleted                                       | table | wres       |                                        | 146 MB     |
 wres   | sourcecompleted_source_idx                            | index | wres       | sourcecompleted                        | 115 MB     |
 wres   | timeseries                                            | table | wres_user8 |                                        | 600 MB     |
 wres   | timeseries_feature_index                              | index | wres_user8 | timeseries                             | 187 MB     |
 wres   | timeseries_pk                                         | index | wres_user8 | timeseries                             | 217 MB     |
 wres   | timeseries_source_index                               | index | wres_user8 | timeseries                             | 217 MB     |
 wres   | timeseriesvalue                                       | table | wres       |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_0                                | table | wres_user8 |                                        | 5620 MB    |
 wres   | timeseriesvalue_lead_0_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_0                 | 3618 MB    |
 wres   | timeseriesvalue_lead_0_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_0                 | 2980 MB    |
 wres   | timeseriesvalue_lead_1                                | table | wres_user8 |                                        | 2561 MB    |
 wres   | timeseriesvalue_lead_10                               | table | wres_user8 |                                        | 1271 MB    |
 wres   | timeseriesvalue_lead_100                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_100_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_100               | 1648 kB    |
 wres   | timeseriesvalue_lead_100_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_100               | 1304 kB    |
 wres   | timeseriesvalue_lead_101                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_101_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_101               | 1656 kB    |
 wres   | timeseriesvalue_lead_101_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_101               | 1296 kB    |
 wres   | timeseriesvalue_lead_102                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_102_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_102               | 1656 kB    |
 wres   | timeseriesvalue_lead_102_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_102               | 1328 kB    |
 wres   | timeseriesvalue_lead_103                              | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_103_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_103               | 1664 kB    |
 wres   | timeseriesvalue_lead_103_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_103               | 1304 kB    |
 wres   | timeseriesvalue_lead_104                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_104_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_104               | 1648 kB    |
 wres   | timeseriesvalue_lead_104_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_104               | 1304 kB    |
 wres   | timeseriesvalue_lead_105                              | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_105_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_105               | 1696 kB    |
 wres   | timeseriesvalue_lead_105_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_105               | 1304 kB    |
 wres   | timeseriesvalue_lead_106                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_106_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_106               | 1696 kB    |
 wres   | timeseriesvalue_lead_106_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_106               | 1304 kB    |
 wres   | timeseriesvalue_lead_107                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_107_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_107               | 1704 kB    |
 wres   | timeseriesvalue_lead_107_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_107               | 1304 kB    |
 wres   | timeseriesvalue_lead_108                              | table | wres_user8 |                                        | 2600 kB    |
 wres   | timeseriesvalue_lead_108_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_108               | 1624 kB    |
 wres   | timeseriesvalue_lead_108_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_108               | 1304 kB    |
 wres   | timeseriesvalue_lead_109                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_109_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_109               | 1648 kB    |
 wres   | timeseriesvalue_lead_109_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_109               | 1304 kB    |
 wres   | timeseriesvalue_lead_10_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_10                | 812 MB     |
 wres   | timeseriesvalue_lead_10_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_10                | 640 MB     |
 wres   | timeseriesvalue_lead_11                               | table | wres_user8 |                                        | 744 MB     |
 wres   | timeseriesvalue_lead_110                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_110_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_110               | 1656 kB    |
 wres   | timeseriesvalue_lead_110_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_110               | 1304 kB    |
 wres   | timeseriesvalue_lead_111                              | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_111_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_111               | 1680 kB    |
 wres   | timeseriesvalue_lead_111_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_111               | 1296 kB    |
 wres   | timeseriesvalue_lead_112                              | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_112_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_112               | 1640 kB    |
 wres   | timeseriesvalue_lead_112_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_112               | 1304 kB    |
 wres   | timeseriesvalue_lead_113                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_113_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_113               | 1672 kB    |
 wres   | timeseriesvalue_lead_113_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_113               | 1296 kB    |
 wres   | timeseriesvalue_lead_114                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_114_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_114               | 1648 kB    |
 wres   | timeseriesvalue_lead_114_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_114               | 1296 kB    |
 wres   | timeseriesvalue_lead_115                              | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_115_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_115               | 1672 kB    |
 wres   | timeseriesvalue_lead_115_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_115               | 1304 kB    |
 wres   | timeseriesvalue_lead_116                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_116_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_116               | 1656 kB    |
 wres   | timeseriesvalue_lead_116_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_116               | 1304 kB    |
 wres   | timeseriesvalue_lead_117                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_117_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_117               | 1696 kB    |
 wres   | timeseriesvalue_lead_117_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_117               | 1296 kB    |
 wres   | timeseriesvalue_lead_118                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_118_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_118               | 1656 kB    |
 wres   | timeseriesvalue_lead_118_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_118               | 1296 kB    |
 wres   | timeseriesvalue_lead_119                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_119_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_119               | 1648 kB    |
 wres   | timeseriesvalue_lead_119_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_119               | 1296 kB    |
 wres   | timeseriesvalue_lead_11_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_11                | 474 MB     |
 wres   | timeseriesvalue_lead_11_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_11                | 374 MB     |
 wres   | timeseriesvalue_lead_12                               | table | wres_user8 |                                        | 291 MB     |
 wres   | timeseriesvalue_lead_120                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_120_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_120               | 1608 kB    |
 wres   | timeseriesvalue_lead_120_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_120               | 1304 kB    |
 wres   | timeseriesvalue_lead_121                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_121_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_121               | 1688 kB    |
 wres   | timeseriesvalue_lead_121_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_121               | 1304 kB    |
 wres   | timeseriesvalue_lead_122                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_122_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_122               | 1672 kB    |
 wres   | timeseriesvalue_lead_122_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_122               | 1304 kB    |
 wres   | timeseriesvalue_lead_123                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_123_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_123               | 1680 kB    |
 wres   | timeseriesvalue_lead_123_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_123               | 1304 kB    |
 wres   | timeseriesvalue_lead_124                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_124_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_124               | 1664 kB    |
 wres   | timeseriesvalue_lead_124_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_124               | 1312 kB    |
 wres   | timeseriesvalue_lead_125                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_125_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_125               | 1640 kB    |
 wres   | timeseriesvalue_lead_125_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_125               | 1304 kB    |
 wres   | timeseriesvalue_lead_126                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_126_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_126               | 1672 kB    |
 wres   | timeseriesvalue_lead_126_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_126               | 1304 kB    |
 wres   | timeseriesvalue_lead_127                              | table | wres_user8 |                                        | 2640 kB    |
 wres   | timeseriesvalue_lead_127_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_127               | 1672 kB    |
 wres   | timeseriesvalue_lead_127_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_127               | 1312 kB    |
 wres   | timeseriesvalue_lead_128                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_128_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_128               | 1664 kB    |
 wres   | timeseriesvalue_lead_128_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_128               | 1312 kB    |
 wres   | timeseriesvalue_lead_129                              | table | wres_user8 |                                        | 2600 kB    |
 wres   | timeseriesvalue_lead_129_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_129               | 1680 kB    |
 wres   | timeseriesvalue_lead_129_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_129               | 1304 kB    |
 wres   | timeseriesvalue_lead_12_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_12                | 183 MB     |
 wres   | timeseriesvalue_lead_12_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_12                | 144 MB     |
 wres   | timeseriesvalue_lead_13                               | table | wres_user8 |                                        | 165 MB     |
 wres   | timeseriesvalue_lead_130                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_130_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_130               | 1656 kB    |
 wres   | timeseriesvalue_lead_130_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_130               | 1304 kB    |
 wres   | timeseriesvalue_lead_131                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_131_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_131               | 1664 kB    |
 wres   | timeseriesvalue_lead_131_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_131               | 1304 kB    |
 wres   | timeseriesvalue_lead_132                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_132_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_132               | 1672 kB    |
 wres   | timeseriesvalue_lead_132_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_132               | 1432 kB    |
 wres   | timeseriesvalue_lead_133                              | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_133_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_133               | 1696 kB    |
 wres   | timeseriesvalue_lead_133_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_133               | 1432 kB    |
 wres   | timeseriesvalue_lead_134                              | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_134_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_134               | 1680 kB    |
 wres   | timeseriesvalue_lead_134_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_134               | 1304 kB    |
 wres   | timeseriesvalue_lead_135                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_135_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_135               | 1656 kB    |
 wres   | timeseriesvalue_lead_135_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_135               | 1304 kB    |
 wres   | timeseriesvalue_lead_136                              | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_136_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_136               | 1664 kB    |
 wres   | timeseriesvalue_lead_136_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_136               | 1432 kB    |
 wres   | timeseriesvalue_lead_137                              | table | wres_user8 |                                        | 2600 kB    |
 wres   | timeseriesvalue_lead_137_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_137               | 1656 kB    |
 wres   | timeseriesvalue_lead_137_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_137               | 1320 kB    |
 wres   | timeseriesvalue_lead_138                              | table | wres_user8 |                                        | 2600 kB    |
 wres   | timeseriesvalue_lead_138_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_138               | 1664 kB    |
 wres   | timeseriesvalue_lead_138_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_138               | 1304 kB    |
 wres   | timeseriesvalue_lead_139                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_139_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_139               | 1632 kB    |
 wres   | timeseriesvalue_lead_139_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_139               | 1304 kB    |
 wres   | timeseriesvalue_lead_13_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_13                | 105 MB     |
 wres   | timeseriesvalue_lead_13_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_13                | 83 MB      |
 wres   | timeseriesvalue_lead_14                               | table | wres_user8 |                                        | 166 MB     |
 wres   | timeseriesvalue_lead_140                              | table | wres_user8 |                                        | 2640 kB    |
 wres   | timeseriesvalue_lead_140_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_140               | 1688 kB    |
 wres   | timeseriesvalue_lead_140_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_140               | 1304 kB    |
 wres   | timeseriesvalue_lead_141                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_141_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_141               | 1704 kB    |
 wres   | timeseriesvalue_lead_141_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_141               | 1312 kB    |
 wres   | timeseriesvalue_lead_142                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_142_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_142               | 1664 kB    |
 wres   | timeseriesvalue_lead_142_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_142               | 1312 kB    |
 wres   | timeseriesvalue_lead_143                              | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_143_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_143               | 1672 kB    |
 wres   | timeseriesvalue_lead_143_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_143               | 1320 kB    |
 wres   | timeseriesvalue_lead_144                              | table | wres_user8 |                                        | 200 kB     |
 wres   | timeseriesvalue_lead_144_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_144               | 96 kB      |
 wres   | timeseriesvalue_lead_144_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_144               | 80 kB      |
 wres   | timeseriesvalue_lead_145                              | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_145_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_145               | 8192 bytes |
 wres   | timeseriesvalue_lead_145_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_145               | 8192 bytes |
 wres   | timeseriesvalue_lead_146                              | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_146_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_146               | 8192 bytes |
 wres   | timeseriesvalue_lead_146_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_146               | 8192 bytes |
 wres   | timeseriesvalue_lead_147                              | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_147_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_147               | 8192 bytes |
 wres   | timeseriesvalue_lead_147_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_147               | 8192 bytes |
 wres   | timeseriesvalue_lead_148                              | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_148_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_148               | 8192 bytes |
 wres   | timeseriesvalue_lead_148_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_148               | 8192 bytes |
 wres   | timeseriesvalue_lead_149                              | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_149_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_149               | 8192 bytes |
 wres   | timeseriesvalue_lead_149_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_149               | 8192 bytes |
 wres   | timeseriesvalue_lead_14_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_14                | 105 MB     |
 wres   | timeseriesvalue_lead_14_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_14                | 83 MB      |
 wres   | timeseriesvalue_lead_15                               | table | wres_user8 |                                        | 190 MB     |
 wres   | timeseriesvalue_lead_150                              | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_150_lead_idx                     | index | wres_user8 | timeseriesvalue_lead_150               | 8192 bytes |
 wres   | timeseriesvalue_lead_150_timeseries_idx               | index | wres_user8 | timeseriesvalue_lead_150               | 8192 bytes |
 wres   | timeseriesvalue_lead_15_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_15                | 121 MB     |
 wres   | timeseriesvalue_lead_15_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_15                | 95 MB      |
 wres   | timeseriesvalue_lead_16                               | table | wres_user8 |                                        | 167 MB     |
 wres   | timeseriesvalue_lead_16_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_16                | 106 MB     |
 wres   | timeseriesvalue_lead_16_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_16                | 84 MB      |
 wres   | timeseriesvalue_lead_17                               | table | wres_user8 |                                        | 166 MB     |
 wres   | timeseriesvalue_lead_17_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_17                | 106 MB     |
 wres   | timeseriesvalue_lead_17_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_17                | 83 MB      |
 wres   | timeseriesvalue_lead_18                               | table | wres_user8 |                                        | 189 MB     |
 wres   | timeseriesvalue_lead_18_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_18                | 120 MB     |
 wres   | timeseriesvalue_lead_18_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_18                | 95 MB      |
 wres   | timeseriesvalue_lead_19                               | table | wres_user8 |                                        | 167 MB     |
 wres   | timeseriesvalue_lead_19_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_19                | 106 MB     |
 wres   | timeseriesvalue_lead_19_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_19                | 83 MB      |
 wres   | timeseriesvalue_lead_1_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_1                 | 1645 MB    |
 wres   | timeseriesvalue_lead_1_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_1                 | 1300 MB    |
 wres   | timeseriesvalue_lead_2                                | table | wres_user8 |                                        | 2251 MB    |
 wres   | timeseriesvalue_lead_20                               | table | wres_user8 |                                        | 166 MB     |
 wres   | timeseriesvalue_lead_20_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_20                | 106 MB     |
 wres   | timeseriesvalue_lead_20_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_20                | 83 MB      |
 wres   | timeseriesvalue_lead_21                               | table | wres_user8 |                                        | 189 MB     |
 wres   | timeseriesvalue_lead_21_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_21                | 120 MB     |
 wres   | timeseriesvalue_lead_21_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_21                | 95 MB      |
 wres   | timeseriesvalue_lead_22                               | table | wres_user8 |                                        | 163 MB     |
 wres   | timeseriesvalue_lead_22_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_22                | 104 MB     |
 wres   | timeseriesvalue_lead_22_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_22                | 82 MB      |
 wres   | timeseriesvalue_lead_23                               | table | wres_user8 |                                        | 163 MB     |
 wres   | timeseriesvalue_lead_23_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_23                | 104 MB     |
 wres   | timeseriesvalue_lead_23_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_23                | 82 MB      |
 wres   | timeseriesvalue_lead_24                               | table | wres_user8 |                                        | 186 MB     |
 wres   | timeseriesvalue_lead_24_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_24                | 119 MB     |
 wres   | timeseriesvalue_lead_24_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_24                | 93 MB      |
 wres   | timeseriesvalue_lead_25                               | table | wres_user8 |                                        | 164 MB     |
 wres   | timeseriesvalue_lead_25_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_25                | 104 MB     |
 wres   | timeseriesvalue_lead_25_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_25                | 82 MB      |
 wres   | timeseriesvalue_lead_26                               | table | wres_user8 |                                        | 163 MB     |
 wres   | timeseriesvalue_lead_26_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_26                | 104 MB     |
 wres   | timeseriesvalue_lead_26_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_26                | 82 MB      |
 wres   | timeseriesvalue_lead_27                               | table | wres_user8 |                                        | 186 MB     |
 wres   | timeseriesvalue_lead_27_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_27                | 118 MB     |
 wres   | timeseriesvalue_lead_27_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_27                | 93 MB      |
 wres   | timeseriesvalue_lead_28                               | table | wres_user8 |                                        | 164 MB     |
 wres   | timeseriesvalue_lead_28_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_28                | 104 MB     |
 wres   | timeseriesvalue_lead_28_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_28                | 82 MB      |
 wres   | timeseriesvalue_lead_29                               | table | wres_user8 |                                        | 163 MB     |
 wres   | timeseriesvalue_lead_29_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_29                | 104 MB     |
 wres   | timeseriesvalue_lead_29_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_29                | 82 MB      |
 wres   | timeseriesvalue_lead_2_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_2                 | 1444 MB    |
 wres   | timeseriesvalue_lead_2_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_2                 | 1140 MB    |
 wres   | timeseriesvalue_lead_3                                | table | wres_user8 |                                        | 2554 MB    |
 wres   | timeseriesvalue_lead_30                               | table | wres_user8 |                                        | 186 MB     |
 wres   | timeseriesvalue_lead_30_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_30                | 119 MB     |
 wres   | timeseriesvalue_lead_30_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_30                | 93 MB      |
 wres   | timeseriesvalue_lead_31                               | table | wres_user8 |                                        | 164 MB     |
 wres   | timeseriesvalue_lead_31_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_31                | 104 MB     |
 wres   | timeseriesvalue_lead_31_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_31                | 82 MB      |
 wres   | timeseriesvalue_lead_32                               | table | wres_user8 |                                        | 163 MB     |
 wres   | timeseriesvalue_lead_32_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_32                | 104 MB     |
 wres   | timeseriesvalue_lead_32_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_32                | 82 MB      |
 wres   | timeseriesvalue_lead_33                               | table | wres_user8 |                                        | 187 MB     |
 wres   | timeseriesvalue_lead_33_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_33                | 119 MB     |
 wres   | timeseriesvalue_lead_33_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_33                | 94 MB      |
 wres   | timeseriesvalue_lead_34                               | table | wres_user8 |                                        | 166 MB     |
 wres   | timeseriesvalue_lead_34_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_34                | 105 MB     |
 wres   | timeseriesvalue_lead_34_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_34                | 83 MB      |
 wres   | timeseriesvalue_lead_35                               | table | wres_user8 |                                        | 166 MB     |
 wres   | timeseriesvalue_lead_35_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_35                | 106 MB     |
 wres   | timeseriesvalue_lead_35_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_35                | 83 MB      |
 wres   | timeseriesvalue_lead_36                               | table | wres_user8 |                                        | 120 MB     |
 wres   | timeseriesvalue_lead_36_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_36                | 76 MB      |
 wres   | timeseriesvalue_lead_36_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_36                | 60 MB      |
 wres   | timeseriesvalue_lead_37                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_37_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_37                | 61 MB      |
 wres   | timeseriesvalue_lead_37_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_37                | 48 MB      |
 wres   | timeseriesvalue_lead_38                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_38_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_38                | 61 MB      |
 wres   | timeseriesvalue_lead_38_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_38                | 48 MB      |
 wres   | timeseriesvalue_lead_39                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_39_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_39                | 61 MB      |
 wres   | timeseriesvalue_lead_39_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_39                | 48 MB      |
 wres   | timeseriesvalue_lead_3_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_3                 | 1638 MB    |
 wres   | timeseriesvalue_lead_3_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_3                 | 1295 MB    |
 wres   | timeseriesvalue_lead_4                                | table | wres_user8 |                                        | 2440 MB    |
 wres   | timeseriesvalue_lead_40                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_40_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_40                | 61 MB      |
 wres   | timeseriesvalue_lead_40_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_40                | 48 MB      |
 wres   | timeseriesvalue_lead_41                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_41_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_41                | 61 MB      |
 wres   | timeseriesvalue_lead_41_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_41                | 48 MB      |
 wres   | timeseriesvalue_lead_42                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_42_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_42                | 61 MB      |
 wres   | timeseriesvalue_lead_42_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_42                | 48 MB      |
 wres   | timeseriesvalue_lead_43                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_43_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_43                | 61 MB      |
 wres   | timeseriesvalue_lead_43_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_43                | 48 MB      |
 wres   | timeseriesvalue_lead_44                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_44_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_44                | 61 MB      |
 wres   | timeseriesvalue_lead_44_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_44                | 48 MB      |
 wres   | timeseriesvalue_lead_45                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_45_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_45                | 61 MB      |
 wres   | timeseriesvalue_lead_45_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_45                | 48 MB      |
 wres   | timeseriesvalue_lead_46                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_46_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_46                | 61 MB      |
 wres   | timeseriesvalue_lead_46_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_46                | 48 MB      |
 wres   | timeseriesvalue_lead_47                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_47_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_47                | 61 MB      |
 wres   | timeseriesvalue_lead_47_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_47                | 48 MB      |
 wres   | timeseriesvalue_lead_48                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_48_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_48                | 61 MB      |
 wres   | timeseriesvalue_lead_48_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_48                | 48 MB      |
 wres   | timeseriesvalue_lead_49                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_49_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_49                | 61 MB      |
 wres   | timeseriesvalue_lead_49_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_49                | 48 MB      |
 wres   | timeseriesvalue_lead_4_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_4                 | 1565 MB    |
 wres   | timeseriesvalue_lead_4_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_4                 | 1236 MB    |
 wres   | timeseriesvalue_lead_5                                | table | wres_user8 |                                        | 2133 MB    |
 wres   | timeseriesvalue_lead_50                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_50_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_50                | 61 MB      |
 wres   | timeseriesvalue_lead_50_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_50                | 48 MB      |
 wres   | timeseriesvalue_lead_51                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_51_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_51                | 61 MB      |
 wres   | timeseriesvalue_lead_51_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_51                | 48 MB      |
 wres   | timeseriesvalue_lead_52                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_52_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_52                | 61 MB      |
 wres   | timeseriesvalue_lead_52_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_52                | 48 MB      |
 wres   | timeseriesvalue_lead_53                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_53_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_53                | 61 MB      |
 wres   | timeseriesvalue_lead_53_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_53                | 48 MB      |
 wres   | timeseriesvalue_lead_54                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_54_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_54                | 61 MB      |
 wres   | timeseriesvalue_lead_54_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_54                | 48 MB      |
 wres   | timeseriesvalue_lead_55                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_55_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_55                | 61 MB      |
 wres   | timeseriesvalue_lead_55_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_55                | 48 MB      |
 wres   | timeseriesvalue_lead_56                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_56_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_56                | 61 MB      |
 wres   | timeseriesvalue_lead_56_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_56                | 48 MB      |
 wres   | timeseriesvalue_lead_57                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_57_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_57                | 61 MB      |
 wres   | timeseriesvalue_lead_57_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_57                | 48 MB      |
 wres   | timeseriesvalue_lead_58                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_58_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_58                | 61 MB      |
 wres   | timeseriesvalue_lead_58_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_58                | 48 MB      |
 wres   | timeseriesvalue_lead_59                               | table | wres_user8 |                                        | 97 MB      |
 wres   | timeseriesvalue_lead_59_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_59                | 61 MB      |
 wres   | timeseriesvalue_lead_59_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_59                | 48 MB      |
 wres   | timeseriesvalue_lead_5_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_5                 | 1366 MB    |
 wres   | timeseriesvalue_lead_5_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_5                 | 1078 MB    |
 wres   | timeseriesvalue_lead_6                                | table | wres_user8 |                                        | 2398 MB    |
 wres   | timeseriesvalue_lead_60                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_60_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_60                | 60 MB      |
 wres   | timeseriesvalue_lead_60_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_60                | 47 MB      |
 wres   | timeseriesvalue_lead_61                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_61_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_61                | 59 MB      |
 wres   | timeseriesvalue_lead_61_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_61                | 47 MB      |
 wres   | timeseriesvalue_lead_62                               | table | wres_user8 |                                        | 93 MB      |
 wres   | timeseriesvalue_lead_62_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_62                | 59 MB      |
 wres   | timeseriesvalue_lead_62_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_62                | 47 MB      |
 wres   | timeseriesvalue_lead_63                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_63_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_63                | 59 MB      |
 wres   | timeseriesvalue_lead_63_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_63                | 47 MB      |
 wres   | timeseriesvalue_lead_64                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_64_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_64                | 59 MB      |
 wres   | timeseriesvalue_lead_64_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_64                | 47 MB      |
 wres   | timeseriesvalue_lead_65                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_65_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_65                | 59 MB      |
 wres   | timeseriesvalue_lead_65_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_65                | 47 MB      |
 wres   | timeseriesvalue_lead_66                               | table | wres_user8 |                                        | 93 MB      |
 wres   | timeseriesvalue_lead_66_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_66                | 59 MB      |
 wres   | timeseriesvalue_lead_66_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_66                | 47 MB      |
 wres   | timeseriesvalue_lead_67                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_67_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_67                | 59 MB      |
 wres   | timeseriesvalue_lead_67_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_67                | 47 MB      |
 wres   | timeseriesvalue_lead_68                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_68_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_68                | 59 MB      |
 wres   | timeseriesvalue_lead_68_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_68                | 47 MB      |
 wres   | timeseriesvalue_lead_69                               | table | wres_user8 |                                        | 93 MB      |
 wres   | timeseriesvalue_lead_69_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_69                | 59 MB      |
 wres   | timeseriesvalue_lead_69_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_69                | 46 MB      |
 wres   | timeseriesvalue_lead_6_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_6                 | 1537 MB    |
 wres   | timeseriesvalue_lead_6_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_6                 | 1215 MB    |
 wres   | timeseriesvalue_lead_7                                | table | wres_user8 |                                        | 2374 MB    |
 wres   | timeseriesvalue_lead_70                               | table | wres_user8 |                                        | 93 MB      |
 wres   | timeseriesvalue_lead_70_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_70                | 59 MB      |
 wres   | timeseriesvalue_lead_70_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_70                | 47 MB      |
 wres   | timeseriesvalue_lead_71                               | table | wres_user8 |                                        | 94 MB      |
 wres   | timeseriesvalue_lead_71_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_71                | 59 MB      |
 wres   | timeseriesvalue_lead_71_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_71                | 47 MB      |
 wres   | timeseriesvalue_lead_72                               | table | wres_user8 |                                        | 26 MB      |
 wres   | timeseriesvalue_lead_72_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_72                | 16 MB      |
 wres   | timeseriesvalue_lead_72_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_72                | 13 MB      |
 wres   | timeseriesvalue_lead_73                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_73_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_73                | 1688 kB    |
 wres   | timeseriesvalue_lead_73_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_73                | 1304 kB    |
 wres   | timeseriesvalue_lead_74                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_74_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_74                | 1696 kB    |
 wres   | timeseriesvalue_lead_74_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_74                | 1304 kB    |
 wres   | timeseriesvalue_lead_75                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_75_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_75                | 1704 kB    |
 wres   | timeseriesvalue_lead_75_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_75                | 1304 kB    |
 wres   | timeseriesvalue_lead_76                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_76_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_76                | 1696 kB    |
 wres   | timeseriesvalue_lead_76_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_76                | 1304 kB    |
 wres   | timeseriesvalue_lead_77                               | table | wres_user8 |                                        | 2600 kB    |
 wres   | timeseriesvalue_lead_77_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_77                | 1664 kB    |
 wres   | timeseriesvalue_lead_77_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_77                | 1304 kB    |
 wres   | timeseriesvalue_lead_78                               | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_78_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_78                | 1632 kB    |
 wres   | timeseriesvalue_lead_78_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_78                | 1304 kB    |
 wres   | timeseriesvalue_lead_79                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_79_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_79                | 1648 kB    |
 wres   | timeseriesvalue_lead_79_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_79                | 1304 kB    |
 wres   | timeseriesvalue_lead_7_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_7                 | 1522 MB    |
 wres   | timeseriesvalue_lead_7_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_7                 | 1204 MB    |
 wres   | timeseriesvalue_lead_8                                | table | wres_user8 |                                        | 2052 MB    |
 wres   | timeseriesvalue_lead_80                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_80_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_80                | 1672 kB    |
 wres   | timeseriesvalue_lead_80_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_80                | 1304 kB    |
 wres   | timeseriesvalue_lead_81                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_81_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_81                | 1680 kB    |
 wres   | timeseriesvalue_lead_81_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_81                | 1304 kB    |
 wres   | timeseriesvalue_lead_82                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_82_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_82                | 1688 kB    |
 wres   | timeseriesvalue_lead_82_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_82                | 1304 kB    |
 wres   | timeseriesvalue_lead_83                               | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_83_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_83                | 1656 kB    |
 wres   | timeseriesvalue_lead_83_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_83                | 1312 kB    |
 wres   | timeseriesvalue_lead_84                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_84_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_84                | 1664 kB    |
 wres   | timeseriesvalue_lead_84_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_84                | 1304 kB    |
 wres   | timeseriesvalue_lead_85                               | table | wres_user8 |                                        | 2600 kB    |
 wres   | timeseriesvalue_lead_85_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_85                | 1712 kB    |
 wres   | timeseriesvalue_lead_85_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_85                | 1304 kB    |
 wres   | timeseriesvalue_lead_86                               | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_86_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_86                | 1624 kB    |
 wres   | timeseriesvalue_lead_86_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_86                | 1304 kB    |
 wres   | timeseriesvalue_lead_87                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_87_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_87                | 1656 kB    |
 wres   | timeseriesvalue_lead_87_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_87                | 1304 kB    |
 wres   | timeseriesvalue_lead_88                               | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_88_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_88                | 1664 kB    |
 wres   | timeseriesvalue_lead_88_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_88                | 1304 kB    |
 wres   | timeseriesvalue_lead_89                               | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_89_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_89                | 1640 kB    |
 wres   | timeseriesvalue_lead_89_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_89                | 1304 kB    |
 wres   | timeseriesvalue_lead_8_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_8                 | 1315 MB    |
 wres   | timeseriesvalue_lead_8_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_8                 | 1037 MB    |
 wres   | timeseriesvalue_lead_9                                | table | wres_user8 |                                        | 2349 MB    |
 wres   | timeseriesvalue_lead_90                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_90_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_90                | 1688 kB    |
 wres   | timeseriesvalue_lead_90_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_90                | 1304 kB    |
 wres   | timeseriesvalue_lead_91                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_91_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_91                | 1680 kB    |
 wres   | timeseriesvalue_lead_91_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_91                | 1312 kB    |
 wres   | timeseriesvalue_lead_92                               | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_92_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_92                | 1600 kB    |
 wres   | timeseriesvalue_lead_92_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_92                | 1304 kB    |
 wres   | timeseriesvalue_lead_93                               | table | wres_user8 |                                        | 2592 kB    |
 wres   | timeseriesvalue_lead_93_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_93                | 1688 kB    |
 wres   | timeseriesvalue_lead_93_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_93                | 1304 kB    |
 wres   | timeseriesvalue_lead_94                               | table | wres_user8 |                                        | 2632 kB    |
 wres   | timeseriesvalue_lead_94_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_94                | 1624 kB    |
 wres   | timeseriesvalue_lead_94_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_94                | 1304 kB    |
 wres   | timeseriesvalue_lead_95                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_95_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_95                | 1688 kB    |
 wres   | timeseriesvalue_lead_95_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_95                | 1304 kB    |
 wres   | timeseriesvalue_lead_96                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_96_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_96                | 1648 kB    |
 wres   | timeseriesvalue_lead_96_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_96                | 1304 kB    |
 wres   | timeseriesvalue_lead_97                               | table | wres_user8 |                                        | 2624 kB    |
 wres   | timeseriesvalue_lead_97_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_97                | 1696 kB    |
 wres   | timeseriesvalue_lead_97_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_97                | 1304 kB    |
 wres   | timeseriesvalue_lead_98                               | table | wres_user8 |                                        | 2608 kB    |
 wres   | timeseriesvalue_lead_98_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_98                | 1688 kB    |
 wres   | timeseriesvalue_lead_98_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_98                | 1304 kB    |
 wres   | timeseriesvalue_lead_99                               | table | wres_user8 |                                        | 2616 kB    |
 wres   | timeseriesvalue_lead_99_lead_idx                      | index | wres_user8 | timeseriesvalue_lead_99                | 1688 kB    |
 wres   | timeseriesvalue_lead_99_timeseries_idx                | index | wres_user8 | timeseriesvalue_lead_99                | 1304 kB    |
 wres   | timeseriesvalue_lead_9_lead_idx                       | index | wres_user8 | timeseriesvalue_lead_9                 | 1505 MB    |
 wres   | timeseriesvalue_lead_9_timeseries_idx                 | index | wres_user8 | timeseriesvalue_lead_9                 | 1197 MB    |
 wres   | timeseriesvalue_lead_above_150                        | table | wres_user8 |                                        | 0 bytes    |
 wres   | timeseriesvalue_lead_above_150_lead_idx               | index | wres_user8 | timeseriesvalue_lead_above_150         | 8192 bytes |
 wres   | timeseriesvalue_lead_above_150_timeseries_idx         | index | wres_user8 | timeseriesvalue_lead_above_150         | 8192 bytes |
 wres   | timeseriesvalue_lead_below_negative_10                | table | wres_user8 |                                        | 39 GB      |
 wres   | timeseriesvalue_lead_below_negative_10_lead_idx       | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 25 GB      |
 wres   | timeseriesvalue_lead_below_negative_10_timeseries_idx | index | wres_user8 | timeseriesvalue_lead_below_negative_10 | 25 GB      |
 wres   | timeseriesvalue_lead_negative_1                       | table | wres_user8 |                                        | 3060 MB    |
 wres   | timeseriesvalue_lead_negative_10                      | table | wres_user8 |                                        | 1367 MB    |
 wres   | timeseriesvalue_lead_negative_10_lead_idx             | index | wres_user8 | timeseriesvalue_lead_negative_10       | 876 MB     |
 wres   | timeseriesvalue_lead_negative_10_timeseries_idx       | index | wres_user8 | timeseriesvalue_lead_negative_10       | 696 MB     |
 wres   | timeseriesvalue_lead_negative_1_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_1        | 1971 MB    |
 wres   | timeseriesvalue_lead_negative_1_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_1        | 1609 MB    |
 wres   | timeseriesvalue_lead_negative_2                       | table | wres_user8 |                                        | 2957 MB    |
 wres   | timeseriesvalue_lead_negative_2_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_2        | 1904 MB    |
 wres   | timeseriesvalue_lead_negative_2_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_2        | 1554 MB    |
 wres   | timeseriesvalue_lead_negative_3                       | table | wres_user8 |                                        | 2822 MB    |
 wres   | timeseriesvalue_lead_negative_3_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_3        | 1819 MB    |
 wres   | timeseriesvalue_lead_negative_3_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_3        | 1481 MB    |
 wres   | timeseriesvalue_lead_negative_4                       | table | wres_user8 |                                        | 2728 MB    |
 wres   | timeseriesvalue_lead_negative_4_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_4        | 1757 MB    |
 wres   | timeseriesvalue_lead_negative_4_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_4        | 1429 MB    |
 wres   | timeseriesvalue_lead_negative_5                       | table | wres_user8 |                                        | 2650 MB    |
 wres   | timeseriesvalue_lead_negative_5_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_5        | 1706 MB    |
 wres   | timeseriesvalue_lead_negative_5_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_5        | 1386 MB    |
 wres   | timeseriesvalue_lead_negative_6                       | table | wres_user8 |                                        | 2556 MB    |
 wres   | timeseriesvalue_lead_negative_6_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_6        | 1644 MB    |
 wres   | timeseriesvalue_lead_negative_6_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_6        | 1332 MB    |
 wres   | timeseriesvalue_lead_negative_7                       | table | wres_user8 |                                        | 2477 MB    |
 wres   | timeseriesvalue_lead_negative_7_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_7        | 1592 MB    |
 wres   | timeseriesvalue_lead_negative_7_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_7        | 1285 MB    |
 wres   | timeseriesvalue_lead_negative_8                       | table | wres_user8 |                                        | 1808 MB    |
 wres   | timeseriesvalue_lead_negative_8_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_8        | 1160 MB    |
 wres   | timeseriesvalue_lead_negative_8_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_8        | 921 MB     |
 wres   | timeseriesvalue_lead_negative_9                       | table | wres_user8 |                                        | 1369 MB    |
 wres   | timeseriesvalue_lead_negative_9_lead_idx              | index | wres_user8 | timeseriesvalue_lead_negative_9        | 878 MB     |
 wres   | timeseriesvalue_lead_negative_9_timeseries_idx        | index | wres_user8 | timeseriesvalue_lead_negative_9        | 699 MB     |
 wres   | timeseriesvalue_timeseries_idx                        | index | wres       | timeseriesvalue                        | 8192 bytes |
 wres   | unitconversion                                        | table | wres       |                                        | 64 kB      |
 wres   | unitconversion_measurement_uix                        | index | wres       | unitconversion                         | 40 kB      |
(523 rows)
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2020-12-17T20:23:28Z


More on the state of @wres8@ in prod from shortly after the previous command:

wres8=> select count(*) from wres.project;
 count
-------
   479
(1 row)

wres8=> select count(*) from wres.projectsource ;
  count
----------
 31694626
(1 row)

wres8=> select count(*) from wres.source;
  count
---------
 4043037
(1 row)

wres8=> select count(*) from wres.sourcecompleted ;
  count
---------
 4043037
(1 row)

wres8=> select count(*) from wres.timeseries;
  count
---------
 6788040
(1 row)

wres8=> select count(*) from wres.timeseriesvalue;
   count
------------
 2376424918
(1 row)
epag commented 3 weeks ago

Original Redmine Comment Author Name: Jesse (Jesse) Original Date: 2021-04-15T22:06:08Z


Prod had over 900GiB of data across over 500 projects with WRES 5.7 earlier today before we cleaned it out for the 5.8 deployment. Sure, there might have been some slower executions due to the size of the database instance, but we are in much better shape than before 5.0.