search

timescale / promscale

[DEPRECATED] Promscale is a unified metric and trace observability backend for Prometheus, Jaeger and OpenTelemetry built on PostgreSQL and TimescaleDB.
https://www.timescale.com/promscale
Apache License 2.0
1.33k stars 169 forks source link

Upgrade to pgx v5 #1809

Closed alejandrodnm closed 1 year ago

alejandrodnm commented 1 year ago

Version 5 of pgx introduced several breaking changes that affected our code base. You can read more about them here:

https://github.com/jackc/pgx/blob/29ad306e47c491a0ecc52d502241539aedd636bd/CHANGELOG.md#codec-and-value-split

In summary:

Previously, types had a Status field that could be Undefined, Null, or Present. This has been changed to a Valid bool field to harmonize with how database/sql represents NULL and to make the zero value useable.

Previously, the type system combined decoding and encoding values with the value types... This concepts have been separated. A Codec only has responsibility for encoding and decoding values. Value types are generally defined by implementing an interface that a particular Codec understands (e.g. PointScanner and PointValuer for the PostgreSQL point type).

All array types are now handled by ArrayCodec instead of using code generation for each new array type...

JSON and JSONB types removed. Use []byte or string directly.

There are some changes that don't seem to appear in the CHANGELOG:

To deal with the custom types, we added the register logic in the afterConnect method of the pool. Every time a connection is created a new pgtype.Map is generated with all the default PostgreSQL, meaning that whenever a connection is created, we need to register our custom types with it.

Description

Merge requirements

Please take into account the following non-code changes that you may need to make with your PR:

alejandrodnm commented 1 year ago

I'm going to take a day off this PR and then re-review everything with a clear mind.

github-actions[bot] commented 1 year ago

@alejandrodnm

Benchmark Result for 79aa08c638b4c54bb5203d7f6ddd20cd04501dd4

Master vs PR

name                                                                                      old time/op      new time/op      delta%0Apkg:github.com/timescale/promscale/pkg/clockcache goos:linux goarch:amd64%0AIntCache                                                                                       598ns ± 3%       586ns ± 2%   -2.13%  (p=0.004 n=9+8)%0AStringCache                                                                                    735ns ± 7%       709ns ± 5%   -3.51%  (p=0.008 n=9+8)%0AEviction/500                                                                                  1.53µs ± 0%      1.57µs ± 0%   +2.26%  (p=0.000 n=10+9)%0AEviction/5000                                                                                 10.5µs ± 0%      10.6µs ± 0%   +0.55%  (p=0.000 n=10+10)%0AEviction/50000                                                                                96.9µs ± 0%      96.3µs ± 0%   -0.57%  (p=0.000 n=9+9)%0AMembership/500                                                                                 164ns ± 0%       170ns ± 1%   +3.72%  (p=0.000 n=9+10)%0AMembership/5000                                                                                168ns ± 0%       175ns ± 1%   +3.67%  (p=0.000 n=9+10)%0AMembership/50000                                                                               225ns ± 4%       209ns ± 3%   -7.14%  (p=0.000 n=10+10)%0AMembership/500000                                                                              448ns ± 2%       422ns ± 3%   -5.91%  (p=0.000 n=10+9)%0ANotFound/500                                                                                   152ns ± 0%       158ns ± 1%   +3.91%  (p=0.000 n=10+10)%0ANotFound/5000                                                                                  158ns ± 1%       165ns ± 1%   +4.32%  (p=0.000 n=10+10)%0ANotFound/50000                                                                                 175ns ± 2%       180ns ± 1%   +3.10%  (p=0.000 n=10+10)%0ANotFound/500000                                                                                322ns ± 3%       320ns ± 1%   -0.81%  (p=0.045 n=10+10)%0AInsertUnderCapacity/500                                                                        169ns ± 3%       166ns ± 1%   -1.65%  (p=0.011 n=10+10)%0AInsertUnderCapacity/5000                                                                       169ns ± 4%       165ns ± 1%   -2.33%  (p=0.000 n=10+9)%0AInsertUnderCapacity/50000                                                                      183ns ± 0%       188ns ± 4%   +3.17%  (p=0.000 n=7+10)%0AInsertUnderCapacity/500000                                                                     280ns ± 2%       301ns ± 1%   +7.37%  (p=0.000 n=10+10)%0AInsertOverCapacity/500                                                                         530ns ± 0%       564ns ± 5%   +6.51%  (p=0.000 n=9+10)%0AInsertOverCapacity/5000                                                                        477ns ± 1%       509ns ± 2%   +6.63%  (p=0.000 n=10+10)%0AInsertOverCapacity/50000                                                                       473ns ± 1%       500ns ± 2%   +5.80%  (p=0.000 n=9+10)%0AInsertOverCapacity/500000                                                                      534ns ± 4%       538ns ± 5%     ~     (p=0.529 n=10+10)%0AInsertConcurrent/500                                                                           532ns ± 1%       560ns ± 6%   +5.36%  (p=0.001 n=10+10)%0AInsertConcurrent/5000                                                                          479ns ± 0%       502ns ± 2%   +4.87%  (p=0.000 n=10+10)%0AInsertConcurrent/50000                                                                         478ns ± 1%       503ns ± 2%   +5.25%  (p=0.000 n=9+10)%0AInsertConcurrent/500000                                                                        537ns ± 4%       536ns ± 1%     ~     (p=0.922 n=10+9)%0AMemoryEmptyCache                                                                              47.2ns ± 1%      46.0ns ± 3%   -2.57%  (p=0.000 n=9+8)%0Apkg:github.com/timescale/promscale/pkg/pgmodel/model/pgutf8str goos:linux goarch:amd64%0AMapWithContains/No_nulls_sanitize                                                             7.09ns ± 1%      6.23ns ± 2%  -12.14%  (p=0.000 n=10+9)%0AMapWithContains/No_nulls_revert                                                               19.4ns ± 0%      19.3ns ± 0%     ~     (p=0.675 n=10+9)%0AMapWithContains/With_nulls_sanitize                                                            148ns ± 1%       156ns ± 0%   +4.91%  (p=0.000 n=9+9)%0AMapWithContains/With_nulls_revert                                                              163ns ± 1%       158ns ± 1%   -3.09%  (p=0.000 n=10+10)%0AMapWithoutContains/No_nulls_sanitize                                                          82.1ns ± 0%      90.3ns ± 1%  +10.05%  (p=0.000 n=10+10)%0AMapWithoutContains/No_nulls_revert                                                            90.8ns ± 0%      81.8ns ± 0%   -9.95%  (p=0.000 n=9+10)%0AMapWithoutContains/With_nulls_sanitize                                                        72.6ns ± 0%      84.7ns ± 0%  +16.64%  (p=0.000 n=10+9)%0AMapWithoutContains/With_nulls_revert                                                          78.8ns ± 0%      72.5ns ± 0%   -8.00%  (p=0.000 n=10+10)%0Apkg:github.com/timescale/promscale/pkg/rules goos:linux goarch:amd64%0AVectorConversion/yolo                                                                        0.00ns ±100%      0.00ns ±67%     ~     (p=0.337 n=10+10)%0AVectorConversion/looping                                                                      0.00ns ± 5%      0.00ns ± 4%   +3.49%  (p=0.014 n=10+10)%0Apkg:github.com/timescale/promscale/pkg/tests/end_to_end_tests goos:linux goarch:amd64%0AMetricIngest                                                                                   26.1s ± 5%       26.7s ± 6%     ~     (p=0.190 n=10+10)%0ANewSeriesIngestion/small_series_metrics_100_seriesPerMetric_10_labels_4_batchSize_100          1.90s ± 3%       1.97s ± 5%   +3.24%  (p=0.035 n=9+10)%0ANewSeriesIngestion/medium_series_metrics_100_seriesPerMetric_100_labels_4_batchSize_1000       4.43s ± 4%       4.58s ± 2%   +3.48%  (p=0.000 n=10+10)%0ANewSeriesIngestion/large_series_metrics_100_seriesPerMetric_500_labels_4_batchSize_10000       15.1s ± 4%       14.8s ± 3%     ~     (p=0.143 n=10+10)%0AGetSeriesIDForKeyValueArrayExistingSeries                                                      986µs ± 2%      1015µs ± 6%   +2.95%  (p=0.029 n=10+10)%0AGetSeriesIDForKeyValueArrayNewSeriesExistingLabels                                            1.44ms ± 1%      1.46ms ± 2%   +1.68%  (p=0.014 n=9+9)%0AGetSeriesIDForKeyValueArrayNewMetric                                                          42.8ms ± 6%      41.6ms ± 5%   -2.82%  (p=0.019 n=10+10)%0AGetSeriesIDForKeyValueArrayNewSeriesNewLabels                                                 1.66ms ± 6%      1.68ms ± 3%     ~     (p=0.156 n=10+9)%0AKeyValueArrayToLabelArrayCreateNewLabels                                                       965µs ± 8%      1029µs ± 6%   +6.59%  (p=0.001 n=10+10)%0AKeyValueArrayToLabelArrayExistingLabels                                                        578µs ± 2%       613µs ± 3%   +6.01%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayCreateNewLabelKeys                                                    950µs ± 1%      1015µs ± 7%   +6.88%  (p=0.001 n=9+10)%0AGetOrCreateMetricTableName                                                                     358µs ± 3%       355µs ± 7%     ~     (p=0.497 n=9+10)%0A%0Aname                                                                                      old alloc/op     new alloc/op     delta%0Apkg:github.com/timescale/promscale/pkg/clockcache goos:linux goarch:amd64%0AIntCache                                                                                       48.0B ± 0%       48.0B ± 0%     ~     (all equal)%0AStringCache                                                                                    49.9B ± 4%       50.4B ± 5%     ~     (p=0.250 n=10+10)%0AEviction/500                                                                                   48.0B ± 0%       48.0B ± 0%     ~     (all equal)%0AEviction/5000                                                                                  48.0B ± 0%       48.0B ± 0%     ~     (all equal)%0AEviction/50000                                                                                 49.0B ± 0%       49.0B ± 0%     ~     (all equal)%0AMembership/500                                                                                 0.00B            0.00B          ~     (all equal)%0AMembership/5000                                                                                0.00B            0.00B          ~     (all equal)%0AMembership/50000                                                                               0.00B            0.00B          ~     (all equal)%0AMembership/500000                                                                              0.00B            0.00B          ~     (all equal)%0ANotFound/500                                                                                   0.00B            0.00B          ~     (all equal)%0ANotFound/5000                                                                                  0.00B            0.00B          ~     (all equal)%0ANotFound/50000                                                                                 0.00B            0.00B          ~     (all equal)%0ANotFound/500000                                                                                0.00B            0.00B          ~     (all equal)%0AInsertUnderCapacity/500                                                                        0.00B            0.00B          ~     (all equal)%0AInsertUnderCapacity/5000                                                                       0.00B            0.00B          ~     (all equal)%0AInsertUnderCapacity/50000                                                                      0.00B            0.00B          ~     (all equal)%0AInsertUnderCapacity/500000                                                                     2.00B ± 0%       2.00B ± 0%     ~     (all equal)%0AInsertOverCapacity/500                                                                         36.0B ± 0%       36.0B ± 0%     ~     (all equal)%0AInsertOverCapacity/5000                                                                        27.0B ± 0%       27.0B ± 0%     ~     (all equal)%0AInsertOverCapacity/50000                                                                       21.0B ± 0%       21.0B ± 0%     ~     (all equal)%0AInsertOverCapacity/500000                                                                      16.0B ± 0%       16.0B ± 0%     ~     (all equal)%0AInsertConcurrent/500                                                                           36.0B ± 0%       36.0B ± 0%     ~     (all equal)%0AInsertConcurrent/5000                                                                          27.0B ± 0%       27.0B ± 0%     ~     (all equal)%0AInsertConcurrent/50000                                                                         21.0B ± 0%       21.0B ± 0%     ~     (all equal)%0AInsertConcurrent/500000                                                                        16.0B ± 0%       16.0B ± 0%     ~     (all equal)%0AMemoryEmptyCache                                                                                112B ± 1%        110B ± 1%   -1.68%  (p=0.000 n=9+9)%0Apkg:github.com/timescale/promscale/pkg/pgmodel/model/pgutf8str goos:linux goarch:amd64%0AMapWithContains/No_nulls_sanitize                                                              0.00B            0.00B          ~     (all equal)%0AMapWithContains/No_nulls_revert                                                                0.00B            0.00B          ~     (all equal)%0AMapWithContains/With_nulls_sanitize                                                            24.0B ± 0%       24.0B ± 0%     ~     (all equal)%0AMapWithContains/With_nulls_revert                                                              32.0B ± 0%       32.0B ± 0%     ~     (all equal)%0AMapWithoutContains/No_nulls_sanitize                                                           0.00B            0.00B          ~     (all equal)%0AMapWithoutContains/No_nulls_revert                                                             0.00B            0.00B          ~     (all equal)%0AMapWithoutContains/With_nulls_sanitize                                                         0.00B            0.00B          ~     (all equal)%0AMapWithoutContains/With_nulls_revert                                                           0.00B            0.00B          ~     (all equal)%0Apkg:github.com/timescale/promscale/pkg/rules goos:linux goarch:amd64%0AVectorConversion/yolo                                                                          0.00B            0.00B          ~     (all equal)%0AVectorConversion/looping                                                                       0.00B            0.00B          ~     (all equal)%0Apkg:github.com/timescale/promscale/pkg/tests/end_to_end_tests goos:linux goarch:amd64%0AMetricIngest                                                                                  2.17GB ± 5%      1.05GB ± 0%  -51.56%  (p=0.000 n=10+10)%0ANewSeriesIngestion/small_series_metrics_100_seriesPerMetric_10_labels_4_batchSize_100          172MB ± 0%       167MB ± 0%   -2.91%  (p=0.000 n=9+10)%0ANewSeriesIngestion/medium_series_metrics_100_seriesPerMetric_100_labels_4_batchSize_1000       215MB ± 0%       218MB ± 0%   +1.13%  (p=0.000 n=10+10)%0ANewSeriesIngestion/large_series_metrics_100_seriesPerMetric_500_labels_4_batchSize_10000       401MB ± 0%       438MB ± 1%   +9.22%  (p=0.000 n=9+10)%0AGetSeriesIDForKeyValueArrayExistingSeries                                                     1.37kB ± 0%      1.55kB ± 1%  +12.62%  (p=0.000 n=9+10)%0AGetSeriesIDForKeyValueArrayNewSeriesExistingLabels                                            1.38kB ± 1%      1.55kB ± 0%  +12.39%  (p=0.000 n=10+8)%0AGetSeriesIDForKeyValueArrayNewMetric                                                          1.38kB ±11%      1.53kB ± 0%  +11.04%  (p=0.000 n=10+10)%0AGetSeriesIDForKeyValueArrayNewSeriesNewLabels                                                 1.38kB ± 0%      1.55kB ± 0%  +12.36%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayCreateNewLabels                                                      1.17kB ± 1%      1.31kB ± 0%  +11.38%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayExistingLabels                                                       1.17kB ± 1%      1.30kB ± 0%  +11.08%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayCreateNewLabelKeys                                                   1.17kB ± 1%      1.35kB ± 0%  +14.98%  (p=0.000 n=10+10)%0AGetOrCreateMetricTableName                                                                      562B ± 0%        640B ± 1%  +13.89%  (p=0.000 n=10+10)%0A%0Aname                                                                                      old allocs/op    new allocs/op    delta%0Apkg:github.com/timescale/promscale/pkg/clockcache goos:linux goarch:amd64%0AIntCache                                                                                        1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AStringCache                                                                                     1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AEviction/500                                                                                    1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AEviction/5000                                                                                   1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AEviction/50000                                                                                  1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AMembership/500                                                                                  0.00             0.00          ~     (all equal)%0AMembership/5000                                                                                 0.00             0.00          ~     (all equal)%0AMembership/50000                                                                                0.00             0.00          ~     (all equal)%0AMembership/500000                                                                               0.00             0.00          ~     (all equal)%0ANotFound/500                                                                                    0.00             0.00          ~     (all equal)%0ANotFound/5000                                                                                   0.00             0.00          ~     (all equal)%0ANotFound/50000                                                                                  0.00             0.00          ~     (all equal)%0ANotFound/500000                                                                                 0.00             0.00          ~     (all equal)%0AInsertUnderCapacity/500                                                                         0.00             0.00          ~     (all equal)%0AInsertUnderCapacity/5000                                                                        0.00             0.00          ~     (all equal)%0AInsertUnderCapacity/50000                                                                       0.00             0.00          ~     (all equal)%0AInsertUnderCapacity/500000                                                                      0.00             0.00          ~     (all equal)%0AInsertOverCapacity/500                                                                          0.00             0.00          ~     (all equal)%0AInsertOverCapacity/5000                                                                         0.00             0.00          ~     (all equal)%0AInsertOverCapacity/50000                                                                        0.00             0.00          ~     (all equal)%0AInsertOverCapacity/500000                                                                       0.00             0.00          ~     (all equal)%0AInsertConcurrent/500                                                                            0.00             0.00          ~     (all equal)%0AInsertConcurrent/5000                                                                           0.00             0.00          ~     (all equal)%0AInsertConcurrent/50000                                                                          0.00             0.00          ~     (all equal)%0AInsertConcurrent/500000                                                                         0.00             0.00          ~     (all equal)%0AMemoryEmptyCache                                                                                0.00             0.00          ~     (all equal)%0Apkg:github.com/timescale/promscale/pkg/pgmodel/model/pgutf8str goos:linux goarch:amd64%0AMapWithContains/No_nulls_sanitize                                                               0.00             0.00          ~     (all equal)%0AMapWithContains/No_nulls_revert                                                                 0.00             0.00          ~     (all equal)%0AMapWithContains/With_nulls_sanitize                                                             1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AMapWithContains/With_nulls_revert                                                               1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AMapWithoutContains/No_nulls_sanitize                                                            0.00             0.00          ~     (all equal)%0AMapWithoutContains/No_nulls_revert                                                              0.00             0.00          ~     (all equal)%0AMapWithoutContains/With_nulls_sanitize                                                          0.00             0.00          ~     (all equal)%0AMapWithoutContains/With_nulls_revert                                                            0.00             0.00          ~     (all equal)%0Apkg:github.com/timescale/promscale/pkg/rules goos:linux goarch:amd64%0AVectorConversion/yolo                                                                           0.00             0.00          ~     (all equal)%0AVectorConversion/looping                                                                        0.00             0.00          ~     (all equal)%0Apkg:github.com/timescale/promscale/pkg/tests/end_to_end_tests goos:linux goarch:amd64%0AMetricIngest                                                                                   10.5M ± 1%       14.9M ± 1%  +41.75%  (p=0.000 n=10+10)%0ANewSeriesIngestion/small_series_metrics_100_seriesPerMetric_10_labels_4_batchSize_100          74.3k ± 0%      113.2k ± 0%  +52.34%  (p=0.000 n=10+8)%0ANewSeriesIngestion/medium_series_metrics_100_seriesPerMetric_100_labels_4_batchSize_1000        528k ± 0%        839k ± 0%  +58.92%  (p=0.000 n=10+9)%0ANewSeriesIngestion/large_series_metrics_100_seriesPerMetric_500_labels_4_batchSize_10000       2.53M ± 0%       4.05M ± 0%  +60.09%  (p=0.000 n=10+10)%0AGetSeriesIDForKeyValueArrayExistingSeries                                                       31.0 ± 0%        58.0 ± 0%  +87.10%  (p=0.000 n=10+10)%0AGetSeriesIDForKeyValueArrayNewSeriesExistingLabels                                              31.0 ± 0%        58.0 ± 0%  +87.10%  (p=0.000 n=10+10)%0AGetSeriesIDForKeyValueArrayNewMetric                                                            32.0 ± 0%        59.0 ± 0%  +84.38%  (p=0.000 n=10+10)%0AGetSeriesIDForKeyValueArrayNewSeriesNewLabels                                                   31.0 ± 0%        58.0 ± 0%  +87.10%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayCreateNewLabels                                                        26.0 ± 0%        51.0 ± 0%  +96.15%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayExistingLabels                                                         26.0 ± 0%        51.0 ± 0%  +96.15%  (p=0.000 n=10+10)%0AKeyValueArrayToLabelArrayCreateNewLabelKeys                                                     26.0 ± 0%        51.0 ± 0%  +96.15%  (p=0.000 n=10+10)%0AGetOrCreateMetricTableName                                                                      9.00 ± 0%       14.00 ± 0%  +55.56%  (p=0.000 n=10+10)%0A%0Aname                                                                                      old inserts      new inserts      delta%0Apkg:github.com/timescale/promscale/pkg/clockcache goos:linux goarch:amd64%0AInsertUnderCapacity/500                                                                        7.04M ± 3%       7.26M ± 1%   +3.24%  (p=0.004 n=10+9)%0AInsertUnderCapacity/5000                                                                       6.95M ± 4%       7.20M ± 1%   +3.70%  (p=0.001 n=10+10)%0AInsertUnderCapacity/50000                                                                      6.19M ± 3%       5.87M ± 4%   -5.19%  (p=0.000 n=9+10)%0AInsertUnderCapacity/500000                                                                     3.77M ± 2%       3.45M ± 0%   -8.50%  (p=0.000 n=10+9)%0AInsertOverCapacity/500                                                                         2.26M ± 1%       2.13M ± 5%   -5.58%  (p=0.000 n=10+10)%0AInsertOverCapacity/5000                                                                        2.52M ± 1%       2.38M ± 2%   -5.65%  (p=0.000 n=10+10)%0AInsertOverCapacity/50000                                                                       2.52M ± 3%       2.41M ± 1%   -4.22%  (p=0.000 n=10+10)%0AInsertOverCapacity/500000                                                                      2.33M ±11%       2.34M ± 8%     ~     (p=0.796 n=10+10)%0A%0Aname                                                                                      old inserts/ops  new inserts/ops  delta%0Apkg:github.com/timescale/promscale/pkg/clockcache goos:linux goarch:amd64%0AInsertUnderCapacity/500                                                                         1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertUnderCapacity/5000                                                                        1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertUnderCapacity/50000                                                                       1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertUnderCapacity/500000                                                                      1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertOverCapacity/500                                                                          1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertOverCapacity/5000                                                                         1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertOverCapacity/50000                                                                        1.00 ± 0%        1.00 ± 0%     ~     (all equal)%0AInsertOverCapacity/500000                                                                       1.00 ± 0%        1.00 ± 0%     ~     (all equal)
niksajakovljevic commented 1 year ago

It would be interesting to compare memory profiles while running https://github.com/timescale/promscale-benchmark/ . Like you noted before we expect this upgrade to use less memory due to improvements in PGX.