Centralize aggregations into a module

martijnvg commented 2 years ago

This is a meta issue tracks to work to be done for centralising most aggregations in a dedicated module.

The main benefits for centralizing aggregations are:

Building and testing all aggregations is easier and quicker. Server module doesn't need to be build in order to build aggregations. Running something like ./gradlew -p modules/aggregations check will be most of time sufficient to ensure that a change to aggregations is working correctly.
This will make server module more light weight, since it doesn't need to contain many aggregations, but just aggregation infrastructure.

Also aggregations from other modules can be moved to the new aggregation module. But this depends on the license level of these aggregations, whether these aggregations pull in heavy dependencies or require additional security permissions.

When moving an aggregation both production implementation code (builder, aggregator, aggregator factory, bucket representation) should be moved and tests (unit, integration and yaml).

Main tasks

Bucket & metric aggregations

Aggregations that currently exist in server module and can be moved to .modules/aggregations:

[x] Move adjacency_matrix agg to aggregation module. #90294
[x] Move auto_date_histogram agg to aggregation module. #90746
[ ] Move avg agg to aggregation module.
[ ] Move cardinality agg to aggregation module.
[ ] Move composite agg to aggregation module. (#90954)
[ ] Move ctor_cardinality_upper_bound agg to aggregation module.
[ ] Move date_histogram agg to aggregation module.
[ ] Move date_range agg to aggregation module.
[ ] Move diversified_sampler agg to aggregation module.
[ ] Move extended_stats agg to aggregation module.
[ ] Move filter agg to aggregation module.
[ ] Move filters agg to aggregation module.
[ ] Move global agg to aggregation module.
[ ] Move histogram agg to aggregation module.
[ ] Move ip_prefix agg to aggregation module.
[ ] Move ip_range agg to aggregation module.
[ ] Move max agg to aggregation module.
[ ] Move median_absolute_deviation agg to aggregation module. (#92676)
[ ] Move min agg to aggregation module.
[ ] Move missing agg to aggregation module.
[ ] Move nested agg to aggregation module.
[ ] Move percentile_ranks agg to aggregation module.
[ ] Move percentiles agg to aggregation module.
[ ] Move random_sampler agg to aggregation module.
[ ] Move range agg to aggregation module.
[ ] Move rare_terms agg to aggregation module.
[ ] Move reverse_nested agg to aggregation module.
[ ] Move sampler agg to aggregation module.
[ ] Move scripted_metric agg to aggregation module.
[ ] Move significant_terms agg to aggregation module.
[ ] Move significant_text agg to aggregation module.
[ ] Move stats agg to aggregation module.
[ ] Move sum agg to aggregation module.
[ ] Move terms agg to aggregation module.
[x] Move time_series agg to aggregation module. #91356
[ ] Move top_hits agg to aggregation module.
[ ] Move value_count agg to aggregation module.
[ ] Move variable_width_histogram agg to aggregation module.
[ ] Move weighted_avg agg to aggregation module.

Pipeline aggregations

Pipeline aggregations that currently exist in server module and can be moved to .modules/aggregations:

[ ] Move avg_bucket pipeline agg to aggregation module.
[ ] Move bucket_script pipeline agg to aggregation module.
[ ] Move bucket_selector pipeline agg to aggregation module. (#91355)
[x] Move bucket_sort pipeline agg to aggregation module. (#91221)
[ ] Move cumulative_sum pipeline agg to aggregation module.
[x] Move derivative pipeline agg to aggregation module. (#91014)
[ ] Move extended_stats_bucket pipeline agg to aggregation module.
[ ] Move max_bucket pipeline agg to aggregation module.
[ ] Move min_bucket pipeline agg to aggregation module.
[ ] Move moving_avg pipeline agg to aggregation module.
[x] Move moving_fn pipeline agg to aggregation module. (#90836)
[ ] Move percentiles_bucket pipeline agg to aggregation module.
[ ] Move serial_diff pipeline agg to aggregation module.
[ ] Move stats_bucket pipeline agg to aggregation module.
[ ] Move sum_bucket pipeline agg to aggregation module.

Geo aggregations

The geo aggregations should maybe be moved to the existing spatial module or a new module that groups geo aggregations together. This is yet undecided.

[ ] Move geo_bounds agg out of the server module.
[ ] Move geo_centroid agg out of the server module.
[ ] Move geo_distance agg out of the server module.
[ ] Move geohash_grid agg out of the server module.
[ ] Move geotile_grid agg out of the server module.

Other tasks

Other tasks that need completion before closing this meta issue.

[ ] Remove rest-api-spec's dependency on aggs module.

elasticsearchmachine commented 2 years ago

Pinging @elastic/es-analytics-geo (Team:Analytics)

wchaparro commented 1 year ago

@martijnvg Can we update this one / close it out at this time?

elastic / elasticsearch