Allow scalar indices to be used in distinct queries

[wjones127]

Right now, the recomended way to get distinct values of columns is to use duckdb:

rows = duckdb.query("SELECT distinct <my column name> FROM lance_dataset")

However, if we have scalar indices on that column, we could execute the query much more quickly. We likely couldn't do that through the DuckDB integration, but we could do it within a DataFusion query easily.

[wjones127]

To implement this, first we would need to expose an API like this:

import lance

dataset = lance.dataset("test")
lance.sql("SELECT * FROM table", table=dataset).to_table()

That should be straightfoward since we already have a table provider.

Then, we would need:

• [ ] DistinctIndexScan
  • Scans values
  • BTree and bitmap
  • If applicable uses prefilter from deletion files
• [ ] Optimizer Rule
  • Detects distinct queries on columns with Btree/bitmap index
  • Splits indexed and unindexed data
  • Use DistinctIndexScan for indexed data
  • Use normal scan path for unindexed data