Getting node datasets scales poorly

waggle-sensor / beehive-server

Waggle cloud software for aggregation, storage and analysis of sensor data from Waggle nodes.

13 stars 17 forks source link

Getting node datasets scales poorly #20

Closed seanshahkarami closed 6 years ago

seanshahkarami commented 6 years ago

Currently, getting the dataset dates for an individual node requires pulling dataset dates for all nodes / dates. We probably want a way for this to scale like O(#dates) instead of O(#nodes * #dates).

A reasonable solution to this would be to have the data insertion process also insert an entry into a datasets table with schema (node_id), date.

seanshahkarami commented 6 years ago

This is addressed implicitly by the new method of dataset export and by an improved table used by the new beehive-loader.