search

city-pulse / mskpulse.backend

New media backend
0 stars 0 forks source link

Detector: extreme CPU and memory costs. #43

Closed city-pulse closed 8 years ago

city-pulse commented 8 years ago

Refactoring required.

city-pulse commented 8 years ago

Time in seconds / CPU load in %:

Phase time Median time 3rd Q time Max CPU median
1. Load datapoints from Redis 0.36 0.86 9.8 100
2. Building reference trees 2.95 10 14 23.7
3. Filtering outliers 0.1 0.3 5.3 100
4. DBSCAN 0 0 0.12 0
5. Load previous events 0.2 1.1 35 100 (70)
6. Merge slices to events 0 0.05 12 0
7. Dump events 0.006 0.06 8.8 100 (50)

Previous estimate is in #23.

city-pulse commented 8 years ago

Todo list: resolve #44, #45, #46. Then, if necessary, try resource package.

city-pulse commented 8 years ago

Need to include time.sleep() into inf loop for detector: let's do one loop in 2 minutes. It should give 1 min pause.