Qbeast-io / qbeast-spark

Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!
https://qbeast.io/qbeast-our-tech/
Apache License 2.0
210 stars 19 forks source link

Remove redundant classes and methods #314

Open Jiaweihu08 opened 5 months ago

Jiaweihu08 commented 5 months ago

Classes and methods, and their corresponding tests, are rendered redundant by algorithm changes such as domain-driven double pass and the latest changes introduced by multi-block files.

For instance, NormalizedWeight should not and cannot be merged since their updates come from domain sizes.

BlockWriter is not used - we use IndexFileWriter to write different blocks into a single file.