Iterations not fully asynchronous

icecube / skymap_scanner

A distributed system that performs a likelihood scan of event directions for IceCube real-time alerts using CPU cluster(s) and queue-based message passing.

5 stars 2 forks source link

Iterations not fully asynchronous #84

Closed tianluyuan closed 1 year ago

tianluyuan commented 2 years ago

The current handling of each iteration isn’t fully asynchronous, as the skymap_scanner waits for the previous nside to complete before starting the next one. The existing followup scanner bot will start up the next iteration once 30% have been scanned, which probably gives a nice speed up as some pixels can take much longer than others.

Related to #79

ric-evans commented 1 year ago

After a conversation with @tianluyuan, @dsschult, and @briedel, we've decided to enable this for realtime via some kind of flag, disabled by default. Perhaps this can be a variable with the predictive percentage (currently 30%)

The tradeoff discussed is:

historical data
- want best resource efficiency -- not highest speed
- so no predictive scanning
- increases in speed can come via largely parallel scans with a shared node cluster to decrease the number of idle nodes
realtime data
- want performance, speed, and ASAP -- not best resource efficiency
- use predictive scanning

ric-evans commented 1 year ago

Historical event scanning is 1000 bicycle commuters. Realtime event scanning is a Ferrari

ric-evans commented 1 year ago

With predictive scanning, nondeterministic scanning is introduced. The practical results are not affected, but re-scans may not result in the same exact skymap, as different regions (local minima) may have been refined before finding the global minimum.