search

raokrutarth / manufacturing-os

A POC distributed operating system for geo-seperated automotive manufacturing
Creative Commons Zero v1.0 Universal
1 stars 0 forks source link

sc_stage not manufacturing items #109

Closed nishantrai18 closed 4 years ago

nishantrai18 commented 4 years ago

For the cluster, cluste_visual

We do not produce anything except node 0 - which doesn't matter anyway. Production logs look like 0 [(6, ItemReq(Item(type:0), qty:1))] @raokrutarth I haven't looked deep into this, but please ensure it works on master regardless. Maybe it's fixed in your new changes, in that case mark this as invalid.

WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 0
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 1
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 2
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 3
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 4
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 5
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 6
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 7
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 8
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 2
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 9
CRITICAL  19:48:20 MainThread   main.py:145::main                 | All nodes started
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 11
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 10
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 1
WARNING  19:48:20 MainThread   processes.py:92::start                | Starting node 12
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 0
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 3
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 5
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 4
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 8
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 6
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 7
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 9
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 12
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 11
WARNING  19:48:20 MainThread   processes.py:112::start                | Successfully started node 10
0 [(6, ItemReq(Item(type:0), qty:1))]
0 [(6, ItemReq(Item(type:0), qty:1))]
WARNING  19:48:27 production-stage-1 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 1's inbound material queue is empty but needs [ItemReq(Item(type:2), qty:1)]. Skipping manufacturing of one batch of item type 1
WARNING  19:48:27 production-stage-2 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 2's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:27 production-stage-5 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 5's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:27 production-stage-3 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 3's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:27 production-stage-4 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 4's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:27 production-stage-8 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 8's inbound material queue is empty but needs [ItemReq(Item(type:0), qty:1)]. Skipping manufacturing of one batch of item type 3
WARNING  19:48:27 production-stage-6 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 6's inbound material queue is empty but needs [ItemReq(Item(type:0), qty:1)]. Skipping manufacturing of one batch of item type 3
WARNING  19:48:27 production-stage-7 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 7's inbound material queue is empty but needs [ItemReq(Item(type:0), qty:1)]. Skipping manufacturing of one batch of item type 3
WARNING  19:48:27 production-stage-12 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 12's inbound material queue is empty but needs [ItemReq(Item(type:1), qty:1)]. Skipping manufacturing of one batch of item type 5
WARNING  19:48:27 production-stage-11 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 11's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1)]. Skipping manufacturing of one batch of item type 4
WARNING  19:48:27 production-stage-9 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 9's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1)]. Skipping manufacturing of one batch of item type 4
WARNING  19:48:27 production-stage-10 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 10's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1)]. Skipping manufacturing of one batch of item type 4
0 [(6, ItemReq(Item(type:0), qty:1))]
0 [(6, ItemReq(Item(type:0), qty:1))]
0 [(6, ItemReq(Item(type:0), qty:1))]
WARNING  19:48:35 production-stage-1 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 1's inbound material queue is empty but needs [ItemReq(Item(type:2), qty:1)]. Skipping manufacturing of one batch of item type 1
WARNING  19:48:35 production-stage-2 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 2's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:35 production-stage-5 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 5's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:35 production-stage-8 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 8's inbound material queue is empty but needs [ItemReq(Item(type:0), qty:1)]. Skipping manufacturing of one batch of item type 3
WARNING  19:48:35 production-stage-4 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 4's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:35 production-stage-3 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 3's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1), ItemReq(Item(type:4), qty:1)]. Skipping manufacturing of one batch of item type 2
WARNING  19:48:35 production-stage-7 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 7's inbound material queue is empty but needs [ItemReq(Item(type:0), qty:1)]. Skipping manufacturing of one batch of item type 3
WARNING  19:48:35 production-stage-6 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 6's inbound material queue is empty but needs [ItemReq(Item(type:0), qty:1)]. Skipping manufacturing of one batch of item type 3
WARNING  19:48:35 production-stage-11 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 11's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1)]. Skipping manufacturing of one batch of item type 4
WARNING  19:48:35 production-stage-12 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 12's inbound material queue is empty but needs [ItemReq(Item(type:1), qty:1)]. Skipping manufacturing of one batch of item type 5
WARNING  19:48:35 production-stage-10 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 10's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1)]. Skipping manufacturing of one batch of item type 4
WARNING  19:48:35 production-stage-9 sc_stage.py:341::_manufacture_batch_and_enqueue | Node 9's inbound material queue is empty but needs [ItemReq(Item(type:3), qty:1)]. Skipping manufacturing of one batch of item type 4
0 [(6, ItemReq(Item(type:0), qty:1))]
^CException in thread publisher-6:
`
raokrutarth commented 4 years ago

looking into it

raokrutarth commented 4 years ago

image

I can see stages doing the manufacturing. Not sure why the correct callback is not being called

raokrutarth commented 4 years ago

the subscriber doesn't seem to be working. For the highlighted message, there is no "received" from subscriber-2 image

nishantrai18 commented 4 years ago

Reopening to handle a different issue. Sorry didn't want to create a new one.

CRITICAL  23:15:34 production-stage-14 sc_stage.py:392::_attempt_manufacture_cycle | Node 14 successfully manufactured batch ItemReq(Item(type:4, id:4-PZGVE644), qty:1) which is a finale item
CRITICAL  23:15:34 production-stage-9 sc_stage.py:392::_attempt_manufacture_cycle | Node 9 successfully manufactured batch ItemReq(Item(type:3, id:3-YIZCF748), qty:1) which is a finale item
CRITICAL  23:15:34 production-stage-13 sc_stage.py:392::_attempt_manufacture_cycle | Node 13 successfully manufactured batch ItemReq(Item(type:4, id:4-NAGYK249), qty:1) which is a finale item
ERROR   23:15:34 subscriber-15 sc_stage.py:309::process_batch_request_response | Node 15 unable to obtain batch of type 2 from node 5. Request ID: ca802e9e
ERROR   23:15:34 subscriber-5 sc_stage.py:309::process_batch_request_response | Node 5 unable to obtain batch of type 4 from node 12. Request ID: 67f31fef
WARNING  23:15:34 subscriber-0 sc_stage.py:205::process_item_waiting_response | Node 0's outbound log for item ItemReq(Item(type:0, id:0-YMSOO320), qty:1) was at status in-queue, expected in-transit
ERROR   23:15:34 subscriber-5 sc_stage.py:309::process_batch_request_response | Node 5 unable to obtain batch of type 3 from node 8. Request ID: f7aab1a3
CRITICAL  23:15:37 production-stage-10 sc_stage.py:392::_attempt_manufacture_cycle | Node 10 successfully manufactured batch ItemReq(Item(type:3, id:3-UXACM002), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-4 sc_stage.py:392::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:1, id:1-DDFSC142), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-7 sc_stage.py:392::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:2, id:2-LJJLW982), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-2 sc_stage.py:392::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-ZOLWI046), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-3 sc_stage.py:392::_attempt_manufacture_cycle | Node 3 successfully manufactured batch ItemReq(Item(type:1, id:1-STVWJ502), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-6 sc_stage.py:392::_attempt_manufacture_cycle | Node 6 successfully manufactured batch ItemReq(Item(type:2, id:2-JCKQW904), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-14 sc_stage.py:392::_attempt_manufacture_cycle | Node 14 successfully manufactured batch ItemReq(Item(type:4, id:4-RNFYF315), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-13 sc_stage.py:392::_attempt_manufacture_cycle | Node 13 successfully manufactured batch ItemReq(Item(type:4, id:4-JXIQL216), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-11 sc_stage.py:392::_attempt_manufacture_cycle | Node 11 successfully manufactured batch ItemReq(Item(type:3, id:3-LBBXL008), qty:1) which is a finale item
CRITICAL  23:15:37 production-stage-9 sc_stage.py:392::_attempt_manufacture_cycle | Node 9 successfully manufactured batch ItemReq(Item(type:3, id:3-NYNZZ294), qty:1) which is a finale item
ERROR   23:15:38 subscriber-15 sc_stage.py:309::process_batch_request_response | Node 15 unable to obtain batch of type 2 from node 5. Request ID: 66a20ea0
ERROR   23:15:39 subscriber-12 sc_stage.py:309::process_batch_request_response | Node 12 unable to obtain batch of type 1 from node 1. Request ID: 437f2cf4
ERROR   23:15:39 subscriber-5 sc_stage.py:309::process_batch_request_response | Node 5 unable to obtain batch of type 4 from node 12. Request ID: ea9b298f
WARNING  23:15:39 subscriber-1 sc_stage.py:205::process_item_waiting_response | Node 1's outbound log for item ItemReq(Item(type:1, id:1-JTJAE367), qty:1) was at status in-queue, expected in-transit
CRITICAL  23:15:40 production-stage-10 sc_stage.py:392::_attempt_manufacture_cycle | Node 10 successfully manufactured batch ItemReq(Item(type:3, id:3-BHFXV406), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-7 sc_stage.py:392::_attempt_manufacture_cycle | Node 7 successfully manufactured batch ItemReq(Item(type:2, id:2-HMGVH234), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-4 sc_stage.py:392::_attempt_manufacture_cycle | Node 4 successfully manufactured batch ItemReq(Item(type:1, id:1-SOPBE128), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-2 sc_stage.py:392::_attempt_manufacture_cycle | Node 2 successfully manufactured batch ItemReq(Item(type:1, id:1-IMLZU164), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-3 sc_stage.py:392::_attempt_manufacture_cycle | Node 3 successfully manufactured batch ItemReq(Item(type:1, id:1-SUIGU622), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-6 sc_stage.py:392::_attempt_manufacture_cycle | Node 6 successfully manufactured batch ItemReq(Item(type:2, id:2-ODVYF072), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-13 sc_stage.py:392::_attempt_manufacture_cycle | Node 13 successfully manufactured batch ItemReq(Item(type:4, id:4-DRXSR305), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-11 sc_stage.py:392::_attempt_manufacture_cycle | Node 11 successfully manufactured batch ItemReq(Item(type:3, id:3-MUFMA278), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-14 sc_stage.py:392::_attempt_manufacture_cycle | Node 14 successfully manufactured batch ItemReq(Item(type:4, id:4-WTSUH573), qty:1) which is a finale item
CRITICAL  23:15:40 production-stage-9 sc_stage.py:392::_attempt_manufacture_cycle | Node 9 successfully manufactured batch ItemReq(Item(type:3, id:3-UTSZW110), qty:1) which is a finale item
ERROR   23:15:41 subscriber-15 sc_stage.py:309::process_batch_request_response | Node 15 unable to obtain batch of type 2 from node 5. Request ID: 77b62fb3
ERROR   23:15:41 subscriber-5 sc_stage.py:309::process_batch_request_response | Node 5 unable to obtain batch of type 4 from node 12. Request ID: 014e697a
WARNING  23:15:41 subscriber-1 sc_stage.py:205::process_item_waiting_response | Node 1's outbound log for item ItemReq(Item(type:1, id:1-LZXLK186), qty:1) was at status in-queue, expected in-transit
ERROR   23:15:42 subscriber-8 sc_stage.py:309::process_batch_request_response | Node 8 unable to obtain batch of type 1 from node 1. Request ID: dbe27a88
ERROR   23:15:42 subscriber-5 sc_stage.py:309::process_batch_request_response | Node 5 unable to obtain batch of type 3 from node 8. Request ID: 87fcb983
ERROR   23:15:42 subscriber-5 sc_stage.py:309::process_batch_request_response | Node 5 unable to obtain batch of type 3 from node 8. Re

Could it be due to your new changes? Even my IDE complains of seeing some inconsistent types in sc_stage.py.

raokrutarth commented 4 years ago

what is the issue?