dathere / datapusher-plus

A standalone web service that pushes data into the CKAN Datastore fast & reliably. It pushes real good!
GNU Affero General Public License v3.0
27 stars 21 forks source link

Smart auto-indexing #30

Closed jqnatividad closed 1 year ago

jqnatividad commented 2 years ago

Have an env var named QSV_AUTOINDEX.

It will be a string with three positions - PUI:

jqnatividad commented 1 year ago

This is implemented as follows:

# if AUTO_INDEX_THRESHOLD > 0 or AUTO_INDEX_DATES is true
# create indices automatically based on summary statistics
# For columns w/ cardinality = record_count, it's all unique values, create a unique index
# If AUTO_INDEX_DATES is true, index all date columns
# if a column's cardinality <= AUTO_INDEX_THRESHOLD, create an index for that column