unify code of GetCounts and Modus functions, and Unique

[MaartenHilferink]

Places where frequency tables are made and used

1. GetCounts (in CalcClassBreaks.h) results contain a ValueCountPairContainer, used in class-break functions and in the MapView for creating a suitable palette for a range of values.
2. ModusTotBySet, ModusTotByTable, ModusPartBySet and ModusPartByTable (in Modus.cpp) and Weighted variants, and formerly: [Weighed]Modus[xxx]ByIndex, used for the modus and similar aggregation functions:
3. GetUniqueValues (in Unique.cpp), very similar to 1, but without counting
4. rlookup -> MakeIndex, also similar, but with keeping first index as a way of 'counting'.

See also #556

Advantages of options 1 and 3 are: very Multithreaded by bottom-up sort and aggregate by merge method Advantage of 2. is: bucket count when appropiate (when pCount * vCount < nCount), but bucket count is much less suitable for multithreading.

[MaartenHilferink]

Improvements at merge:

• GetCounts converts all values to Float64, this is better done after completing the count.
• Modus and similar aggregation functions become parallel
• GetCounts and Unique would use bucket count resp. bitset when vCount <= nCount, similar to pcount and any functions of partitionings
• unique_count can use set or bitset instead of map or count-buckets, similar to what the Unique function uses, as elements don't have to be counted.

Further options:

• make the count type dependent on the domain or nCount and provide specific function overloads with safe increments for expected smaller counts
• revitalize ModusByIndex for strings (now not supported), and share indexing code with rlookup, there use bucket-index (as in invert) and table lookup when vCount <= nCount
• also use results in other aggregations, such as sum, mean, sd, var, but not: min, max, first, last, in cases where bucket-count is preferable (at the expense of pCountvCountT extra memory).

Issues:

• Making bucked count multi threaded increases it's memory pCount vCount and time (nCount) with T pCount vCount where T is the number of concurrent threads; this should only be allowed for T such that T pCount * vCount <= n
• using bucket count for classification requires creating a frequency table afterwards, although this is also done for the map<Value, Count> that results from GetCounts.
• generalizing over map<V, Counter> and set when counts are not required, specifically for Unique, might be a bit confusticated.
• GetCounts results also contain a AbstrValuesRangeDataPtr; this is better not handled where it is not needed. Can it be factored out or its use limited to the classification functions ?

[MaartenHilferink]

ref the first issue: https://chat.openai.com/share/5518acdd-4f17-4811-bb39-df953efa5a64

[jipclaassens]

Use case:

2BURP, to calculate degree of urbanisation it uses 20 iterations per time step: /Analysis/Future/Indicators/Prep/DegreesOfUrbanisation/Y2030/Urban_Centre/Apply_smoothing/smoothing_loop/iter0/NextValue/MedianFiltering

[MaartenHilferink]

/Preprocessing/DegreesOfUrbanisation/Y2020/Urban_Centre/Typology nulmeting: 05:28.

[MaartenHilferink]

01:36 instead of 05:28 = 232 of the 328 seconds faster, i.e. 70% speed-up.

[MaartenHilferink]

Todo's:

• [x] use GetWallCounts in Modus and ModusPart (and all related group count functions)
• [ ] use GetWallCounts in WeightedModus
• [x] Update OperAccUniPart to use up to t threads as long as t * p < n
• [x] Update OperAccUniTot to use up to VCPU threads
• [ ] Update OperAccBinPart, OperAccBinTot to use multiple threads
• [ ] Update ModusByTable to use up to t threads as long as t v p < n
• [ ] use ByTable for GetCounts in Classify
• [x] move existing GetUniqueWallValues to ValuesTable.h
• [ ] use ByTable for GetCounts in JenksFisherClassBreaks
• [ ] use ByTable for GetCounts in Unique
• [ ] use GetWallCounts and GetTable for rlookup and ApplyClassification
• [ ] ValueGetter en WeightGetter aanpassen aan los te laten LazyTiles, conform IndexGetters, en m_Adi helemaal uit valuegetter verwijderen

[jipclaassens]

Test Modus tijd:

/Preprocessing/DegreesOfUrbanisation/y2020/StoreTypology Africa

15.3.0 (2024-06-26 10.00) 2.36 min

15.2.0 (Release) 6.37 min

[MaartenHilferink]

Met parallelle sums: 01:13 i.p.v. 01:36.