Can you fix codes to show progress properly?

hellojinwoo commented 5 years ago

2 issues

There are 2 issues that I am experiencing in all of ipynb files ( fill_in_missing_values, latent_space_embedding, ModelEvaluation, and portfolio_performance)

1. The codes that I grouped as no.2 do not get printed.

I think the solution would be to remove the code

end = "\r"

which I underlined and put 1. After removing the part, following phrases below gets printed in the ' portfolio_performance.ipynb '

340/340 still running. on 4 cores
340/340 still running. on 4 cores
340/340 still running. on 4 cores
340/340 still running. on 4 cores

I don't know why only the same phrases are printed.

2. The while loop does not stop. --> Can you fix codes to show progress properly?

Since the while loop while True: stops when len(rc.result_status(ids)['pending']) == 0. I guess the len(rc.result_status(ids)['pending']) should decrease as code goes by, but as you can see from the question above, the same number keeps repeating. In other words, str(len(rc.result_status(ids)['pending']))does not decrease. I have ran the code for 2 - 3 hours but the while loop does not stop.

I confirmed that the code above in the file 'latent_space_embedding' actually worked. But the printed phrases, showing the progress of codes, seem to be in needs of fix. Following codes are printed with many repetitions as you can see :

It would be nice if you can provide a codes to show proper progress of algorithm!


4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
4/4 still running. on 4 cores
3/4 still running. on 4 cores
3/4 still running. on 4 cores
2/4 still running. on 4 cores
2/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
1/4 still running. on 4 cores
CPU times: user 1.44 s, sys: 238 ms, total: 1.68 s
Wall time: 31min 33s

Background Information

I have ran your codes on Ubuntu, Windows with Jupyter notebook and Jupyter Lab.

hellojinwoo commented 5 years ago

Another question! Why do we need 10 seconds of sleeping in the cell of the picture that I uploaded? Can I decrease it to 1 second or 2 seconds to make codes faster?

rsnirwan commented 5 years ago

After the first for loop (after ids.extend(re.msg_ids)) the code is running in other sessions, not in the notebook session. So, basically the notebook has to wait until the parallel sessions have finished and then has to collect the results via results = [r.get() for r in async_res].

You can do whatever you like in the meantime. I just printed the progress. end="\r" deletes the previous print statement before printing the next one, so that the progress is not printed 1000 or more times. If its not working in your case, you can just delete the 8 line after # display progress or write your own 'display progress statement'.

rsnirwan commented 5 years ago

decreasing the seconds for sleep will not make you code faster, since the code is running in other sessions and the notebook is wait for them to finish anyway. I just typed in 10, because I wanted the progress print to be updated every 10 sec.

hellojinwoo commented 5 years ago

Thank you! Using deque from the library collections, I printed the result only when the current_result is different from previous_result. Your advice was really helpful. I appreciate it.

rsnirwan / GPLVMsInFinance