search

kyledewey / aws-parallelizer

Massively parallel file processing framework on AWS.
1 stars 0 forks source link

Investigate Persistent Requests Not Getting Canceled #10

Open kyledewey opened 11 years ago

kyledewey commented 11 years ago

It seems that occasionally persistent request cancel requests aren't received. I'm not 100% sure this is what is happening, but after the meeting with the AWS folks I'm thinking this must be the case. This is an odd one, because most of the time the cancel requests are received correctly, implying a race condition of sorts.

Previously I thought the issue was with the UI, since eventually the requests would go away. However, another possible explanation is a scenario like this:

  1. Persistent request cancel request sent, but not received or processed for whatever reason.
  2. The instance terminates.
  3. Some time later (usually around 30 minutes), another instance is started from the same persistent request that was never cancelled properly.
  4. The instance immediately sees that there is no work, and sends the cancel request again. This time around, the request is received and processed.
  5. The instance terminates, and the persistent request backing it is gone.
kyledewey commented 11 years ago

Possibly related: in testing during issue #12, I noticed that it may take up to a minute for spot requests to show up as being canceled.