Scheduler doesn't work right now

[yjlcoder]

The scheduler doesn't work right now. Please fix it as soon as possible. Thanks.

[fno2010]

Attach the following stacktrace:

[2017-11-13 01:44:33,924] [orchestrator] [INFO] Start a scheduling thread
[2017-11-13 01:44:33,925] [orchestrator] [INFO] Start scheduling
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/threads.py", line 426, in run
    result = Scheduler(constraints=self._constraints).schedule()  # type: dict[int, int]
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 29, in schedule
    consStr = self.__getFormedStrFromConstraint(constraint)
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 112, in __getFormedStrFromConstraint
    bijkStr = self.__getBIJKStrFromTerm(term)
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 134, in __getBIJKStrFromTerm
    srcHostId = self._IPPort2HostId(str(srcIP) + ":" + str(srcPort))
TypeError: 'dict' object is not callable

[fno2010]

Modifying scheduler.py#L134 and #L139 should fix it. But the following exception occurs:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/threads.py", line 435, in run
    result = Scheduler(constraints=self._constraints).schedule()  # type: dict[int, int]
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 39, in schedule
    finalResult = secondRun.getResult(firstInputData, firstRunResult)
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/increasemaxflow.py", line 265, in getResult
    self.setGlobalVarsFromStruc(rsaInput, firstRunInput)
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/increasemaxflow.py", line 112, in setGlobalVarsFromStruc
    jobId = str(flowName).split("_")[1]
IndexError: list index out of range

[yjlcoder]

Yes, it is. From the code we could infer that the information is gotten from string format, instead of current models. So it has a long way to go.

[fno2010]

Further information:

[2017-11-13 08:01:15,656] [orchestrator] [INFO] Start scheduling
{'B_0_0_1': B_0_0_1}
{'0': '100000'}
{'0': 1}
{'0': ['B_0_0_1']}
Optimal
B_0_0_1 = 99999601.0
bsmall = 999.99601
[2017-11-13 08:01:15,664] [orchestrator] [DEBUG] first_run_result: B_0_0_1 = 99999601.0
bsmall = 999.99601

[fno2010]

The bsmall is not expected in the firstRunResult object.

Horrible code!!! How can I debug it??? @TonyWang123

[fno2010]

The input of second scheduler and the output of the first scheduler are not consistent. Just checked the code in the archived dir.

[TonyWang123]

fixed the issue: [2017-11-13 01:44:33,924] [orchestrator] [INFO] Start a scheduling thread [2017-11-13 01:44:33,925] [orchestrator] [INFO] Start scheduling Exception in thread Thread-9: Traceback (most recent call last): File "/root/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/threads.py", line 426, in run result = Scheduler(constraints=self._constraints).schedule() # type: dict[int, int] File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 29, in schedule consStr = self.getFormedStrFromConstraint(constraint) File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 112, in getFormedStrFromConstraint bijkStr = self.getBIJKStrFromTerm(term) File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 134, in getBIJKStrFromTerm srcHostId = self._IPPort2HostId(str(srcIP) + ":" + str(srcPort)) TypeError: 'dict' object is not callable

[fno2010]

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/threads.py", line 441, in run
    for flow_id in result.keys():
AttributeError: 'FinalResult' object has no attribute 'keys'

[fno2010]

I think scheduler.py#L41 should return finalResult.getResult().

[fno2010]

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/threads.py", line 442, in run
    flow = FlowDataProvider().get(flow_id)
  File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/flows.py", line 228, in get
    return self._id_flow[identifier]
KeyError: 'B_0_0_1'

[TonyWang123]

I see the reason of: Traceback (most recent call last): File "/root/.pyenv/versions/3.6.3/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/models/threads.py", line 435, in run result = Scheduler(constraints=self._constraints).schedule() # type: dict[int, int] File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/scheduler.py", line 39, in schedule finalResult = secondRun.getResult(firstInputData, firstRunResult) File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/alto_unicorn-0.1-py3.6.egg/alto/unicorn/scheduler/increasemaxflow.py", line 265, in getResult self.setGlobalVarsFromStruc(rsaInput, firstRunInput) File "/root/.pyenv/versions/3.6.3/lib/python3.6/site-packages/altounicorn-0.1-py3.6.egg/alto/unicorn/scheduler/increasemaxflow.py", line 112, in setGlobalVarsFromStruc jobId = str(flowName).split("")[1] IndexError: list index out of range I add "bsmall = 999.99601" by mistake. I will fix that.

[fno2010]

@TonyWang123 thx, I post the patch below:

diff --git a/alto/unicorn/scheduler/scheduler.py b/alto/unicorn/scheduler/scheduler.py
index 7babb63..19e0d0c 100644
--- a/alto/unicorn/scheduler/scheduler.py
+++ b/alto/unicorn/scheduler/scheduler.py
@@ -32,13 +32,14 @@ class Scheduler(object):
         firstInputData.addDataSize(self.__getDataSize())

         firstRunResult = firstRun.getFirstRunResult(firstInputData)
+        logger.debug("first_run_result: %s" % (firstRunResult,))

         secondRun = IncreaseMaxFlowScheduler()

         finalResult = secondRun.getResult(firstInputData, firstRunResult)

-        return finalResult
+        return finalResult.getResult()

diff --git a/alto/unicorn/scheduler/simplifiedscheduler.py b/alto/unicorn/scheduler/simplifiedscheduler.py
index 4f8b09d..b2d38f7 100755
--- a/alto/unicorn/scheduler/simplifiedscheduler.py
+++ b/alto/unicorn/scheduler/simplifiedscheduler.py
@@ -119,6 +119,8 @@ class SimplifiedScheduler:

         for v in prob.variables():
             print(v.name, "=", v.varValue)
+            if v.name == 'bsmall':
+                continue
             result.addFirstRunResult(str(v.name) + " = " + str(v.varValue))

         return result

[TonyWang123]

Push the modification:

1. return finalResult.getResult()
2. if "B" in str(v.name): result.addFirstRunResult(str(v.name) + " = " + str(v.varValue))

[fno2010]

schedulerdataformat.py#L44 provide a flowname to bw mapping, but threads.py#L460 requires a flowid to bw mapping. :(

[fno2010]

I guess current scheduler implementation maintains its own flow name system, and it does not map the flow_id to its own flow_name.

[TonyWang123]

If it is necessary, i will add the mapping from flowname to flowid.

[fno2010]

Sure. Please.

[TonyWang123]

Add getFlowIdFromFlowname method in scheduler. please call this when you want to get flow id.

[fno2010]

This patch should finally fix the scheduler input/output adapting:

diff --git a/alto/unicorn/scheduler/scheduler.py b/alto/unicorn/scheduler/scheduler.py
index e8c161a..ccd8019 100644
--- a/alto/unicorn/scheduler/scheduler.py
+++ b/alto/unicorn/scheduler/scheduler.py
@@ -40,7 +40,7 @@ class Scheduler(object):

         finalResult = secondRun.getResult(firstInputData, firstRunResult)

-        return finalResult.getResult()
+        return {self.getFlowIdFromFlowname(k): v for k, v in finalResult.getResult().items()}

     def getFlowIdFromFlowname(self, flowname):

[TonyWang123]

Pushed the modification.