search

BallAerospace / COSMOS

Ball Aerospace COSMOS
https://ballaerospace.github.io/cosmos-website/
Other
360 stars 127 forks source link

DART Reduction unreliable with file imported data #1090

Closed ghost closed 4 years ago

ghost commented 4 years ago

When importing data into DART from files instead of streaming, decommutation can become many minutes out of order when spread across processes. This isn't inherently wrong, but can cause reduction to think that data is ready to be reduced when it is not all available yet. This change will modify:

  1. Make reduction have a 2 minute holdoff to ensure all data is available
  2. Change decom to a work queue design to ensure that packets do not get multiple minutes out of processing order
  3. Improve decommutation speed using transactions
  4. Don't try to switch to bad configs multiple times
ghost commented 4 years ago

@aaronatball I believe this fixes your reduction issues

aaronatball commented 4 years ago

@aaronatball I believe this fixes your reduction issues

Awesome, I will check this out the next time I'm doing DART stuff!