search

cubewise-code / rushti

Smooth parallelization of TI Processes with TM1py
https://code.cubewise.com/tm1py-help-content/run-processes-in-parallel-using-only-connection
MIT License
9 stars 13 forks source link

Processes run in OPT mode are not immediately running after predecessors have finished #25

Closed lbrowncubewise closed 4 years ago

lbrowncubewise commented 5 years ago

When running RushTI in OPT mode, the script should

  1. Run Cube target version clear
  2. Run Cube source version export
  3. Run cube target version import (After ensuring 1 & 2 are finished)

But what seems to be happening is that the import is not starting until all cube export and clears have finished, which seems more in line with the use of the wait command

Please find attached example of RushTI TXT file and the log output from the execution. RushTI.log SnapshotAllCubes.txt

ichermak commented 4 years ago

Hi @lbrowncubewise,

If you need to launch the bloc you described ( {1, 2} -> {3}) bloc by bloc, you should link each bloc with the next one in the tasks file like this :

id="11" predecessors="" instance="XXfab" process="}bedrock.cube.data.export" id="21" predecessors="" instance="XXfab" process="}bedrock.cube.data.clear" id="31" predecessors="11,21" instance="XXfab" process="}bedrock.cube.data.import" id="12" predecessors="31" instance="XXfab" process="}bedrock.cube.data.export" id="22" predecessors="31" instance="XXfab" process="}bedrock.cube.data.clear" id="32" predecessors="12,22" instance="XXfab" process="}bedrock.cube.data.import" etc...

I hope that answers your need.

lbrowncubewise commented 4 years ago

Hi @ichermak not quite. The issue is more that the sequencing is not working as expected. I.e in the below example Id 31 should not run till 11 and 21 have finished but this fault is because 31 is running before these have finished.

id="11" predecessors="" instance="XXfab" process="}bedrock.cube.data.export" id="21" predecessors="" instance="XXfab" process="}bedrock.cube.data.clear" id="31" predecessors="11,21" instance="XXfab" process="}bedrock.cube.data.import"

MariusWirtz commented 4 years ago

Hi @lbrowncubewise,

sorry for responding soo late on this issue. Is it still an issue? Then I will look into it.

PropPortfolio commented 4 years ago

Hi @MariusWirtz , I am facing similar issue as @lbrowncubewise . Let's consider the example attached in "SnapshotAllCubes.txt" file above.

id="11" predecessors="" instance="XXfab" process="}bedrock.cube.data.export" id="21" predecessors="" instance="XXfab" process="}bedrock.cube.data.clear" id="31" predecessors="11,21" instance="XXfab" process="}bedrock.cube.data.import" id="12" predecessors="" instance="XXfab" process="}bedrock.cube.data.export" id="22" predecessors="" instance="XXfab" process="}bedrock.cube.data.clear" id="32" predecessors="12,22" instance="XXfab" process="}bedrock.cube.data.import" id="13" predecessors="" instance="XXfab" process="}bedrock.cube.data.export" id="23" predecessors="" instance="XXfab" process="}bedrock.cube.data.clear" id="33" predecessors="13,23" instance="XXfab" process="}bedrock.cube.data.import" etc...

My understanding is that, If I am running above in OPT mode with 3 maximum workers,

In First run it should execute, ID # 11, 21 & 12 Lets assume ID # 11 finished first, then It should trigger ID # 22 (while ID # 21 & 12 still going). Next lets assume ID # 21 finished, I would expect ID # 31 to kick off ( as ID #11 & 21 finished), But what it does is go to ID # 13 and so on to ID # 23.

ID # 31, 32 & 33 will wait until all other ID # without predecessors are finished executing, while I was expecting RushTI should have triggered them in between as predecessors IDs are completed.