The System shutdown should occur after making sure there are no ongoing jobs.

[GoogleCodeExporter]

To keep track of the issue that I raised via email, I just copy and paste the 
email below. 

-------------------------
I would imagine there is a way to know if there are pending jobs from some 
queue or some other container in the CC side?
In addition to what you mentioned above, we also need to pay attention to the 
ongoing flush/merge ops that are asynchronously running in the background, 
which I already keep an eye of them in my local branch.

Sattam

On Tue, Aug 13, 2013 at 11:53 AM, Zachary Heilbron <zheilbron@gmail.com> wrote:
I outlined a proposal below... thoughts? comments? other proposals?

On 8/13/13 11:44 AM, Young-Seok Kim wrote:
Is there any implemented method to check there are no ongoing jobs during 
shutdown?
This question is relevant to the assumption that WE AGREED for the sharp 
checkpoint.
The assumption is that there are no ongoing jobs during the sharp checkpoint is 
being executed.
How can we ensure this assumption? or Do we need to enforce the actor of the 
sharp checkpoint to wait until all ongoing jobs are finished?
A proposal:
1) Upon shutdown, stop all incoming requests by shutting down the REST API 
server and stopping all active feeds. 2) At all times, track active jobs by 
creating an entry in a table each time a job comes in and removing the entry 
when the job leaves.
3) Have the actor of the sharp checkpoint wait till the table is empty before 
performing a sharp checkpoint.

In either way, we should make sure that there are no ongoing jobs during the 
sharp checkpoint is being executed. Otherwise, log records created by the 
ongoing jobs will be removed by the checkpoint. Thus, this removal will be 
wrong.

Two questions.

1. how are we gonna stop feed jobs? This job never ends as far as I know.
Raman implemented a mechanism to stop feeds. You can send a message to a feed 
to tell it to stop.

2. If there are only no-feed jobs, how do we make sure that there are no 
ongoing jobs? is there any flag that we can check so that the checkpoint may 
proceed safely?
See above proposal.

Regards,
Young-Seok.

Original issue reported on code.google.com by kiss...@gmail.com on 14 Aug 2013 at 12:24

[GoogleCodeExporter]

Will draft a plan covering requirements and implementation details and share it 
with the team.

Original comment by RamanGro...@gmail.com on 16 Nov 2013 at 2:42

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Is there further status for this issue? I am curious about its implications for 
the shutdown call I added to the Hyracks client API. Right now I have it 
implemented as a special type of job; so if there were jobs ahead of it in the 
queue they would finish before shutdown. This is all at the Hyracks level 
though, are there things that Asterix should do outside of Hyracks too, to 
safely shut down? 

Original comment by ima...@uci.edu on 1 Aug 2014 at 3:06