search

scicloj / tablecloth.time

Tools for the processing and manipulation of time-series data in Clojure.
Other
18 stars 1 forks source link

added support for metadata derived from columns, with a basic validity check #5

Closed daslu closed 3 years ago

daslu commented 3 years ago

In many dataset transformations, the metadata stays as-is. This means that if we use it to keep some auxiliary structures derived from the data, then we need to make sure they are valid. In other words, we need to make sure that the columns they have been derived from are identical to the current columns by the same names.

This small PR offers just this basic functionality.

In the context of tablecloth.time, one planned use case is adding index structures for efficient slicing / rolling-window / resampling operations.

This will be also relevant for geospatial indexes, summary statistics (e.g. column means), etc.

daslu commented 3 years ago

Thanks.

Yes, I think the scope of this mechanism is not just time indexes, but possibly some dataset statistics, etc. We will need a similar mechanism in tablecloth.geo. Indeed let us discuss how we imagine these different parts organized.