DUNE-DAQ / drunc

Dune RUN Control (DRUNC) is the run control for the DUNE experiment
1 stars 1 forks source link

`exclude` and `include` issue #197

Open PawelPlesniak opened 3 months ago

PawelPlesniak commented 3 months ago

I ran a test-session, connected to the ru-controller with a drunc-controller-shell, excluded it, and then there were errors as

drunc-unified-shell > status
                     root-controller status                      
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Name               ┃ State   ┃ Substate ┃ In error ┃ Included ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ root-controller    │ initial │ initial  │    No    │   Yes    │
│ ├── ru-controller  │ initial │ initial  │    No    │    No    │
│ ├── df-controller  │ initial │ initial  │    No    │   Yes    │
│ ├── trg-controller │ initial │ initial  │    No    │   Yes    │
│ └── hsi-controller │ initial │ initial  │    No    │   Yes    │
└────────────────────┴─────────┴──────────┴──────────┴──────────┘
drunc-unified-shell > conf
Command not found: conf
drunc-unified-shell > fsm conf
Sending conf.
                    conf execution report                    
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Name                 ┃ Command execution ┃ FSM transition ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ root-controller      │ success           │ success        │
│   ru-controller      │ success           │ failed         │
│   hsi-controller     │ success           │ success        │
│     hsi-01           │ success           │ success        │
│   df-controller      │ success           │ success        │
│     dfo-01           │ success           │ success        │
│     tp-stream-writer │ success           │ success        │
│     df-02            │ success           │ success        │
│     df-01            │ success           │ success        │
│   trg-controller     │ success           │ success        │
│     tc-maker-1       │ success           │ success        │
│     mlt              │ success           │ success        │
│     hsi-to-tc-app    │ success           │ success        │
└──────────────────────┴───────────────────┴────────────────┘

conf executed successfully.
Current FSM status is configured. Available transitions are start scrap
drunc-unified-shell > status
                        root-controller status                        
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Name               ┃ State      ┃ Substate   ┃ In error ┃ Included ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ root-controller    │ configured │ configured │    No    │   Yes    │
│ ├── ru-controller  │ initial    │ initial    │    No    │    No    │
│ ├── df-controller  │ configured │ configured │    No    │   Yes    │
│ ├── trg-controller │ configured │ configured │    No    │   Yes    │
│ └── hsi-controller │ configured │ configured │    No    │   Yes    │
└────────────────────┴────────────┴────────────┴──────────┴──────────┘
drunc-unified-shell > include
[11:47:17] ERROR    "controller_driver": Command 'include' failed on 'root-controller' (response flag 'DRUNC_EXCEPTION_THROWN')                                          shell_utils.py:138
drunc.exceptions.DruncServerSideError: Stacktrace [bold red]on remote server![/bold red]
Traceback (most recent call last):
  File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/broadcast/server/decorators.py", line 30, in wrap
    ret = cmd(obj, request) # we strip the context here, no need for that anymore
  File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/authoriser/decorators.py", line 34, in check_token
    ret = cmd(obj, request)
  File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/decorators.py", line 20, in wrap
    return cmd(obj, request)
  File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/utils/grpc_utils.py", line 65, in unpack_request
    ret = cmd(obj, **kwargs)
  File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/controller.py", line 615, in include
    self.stateful_node.include_node()
  File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/stateful_node.py", line 135, in include_node
    raise CannotInclude()
drunc.controller.stateful_node.CannotInclude: Cannot include node (most likely, it is already included)

drunc.controller.stateful_node.CannotInclude: Cannot include node (most likely, it is already included)
 root-controller -> 5

           WARNING  "click_shell.core": Traceback (most recent call last):                                                                                                       core.py:50
                      File "/nfs/home/pplesnia/nightlyDev/240805/.venv/lib/python3.10/site-packages/click_shell/core.py", line 34, in invoke_                                              
                        command.main(args=shlex.split(arg),                                                                                                                                
                      File "/nfs/home/pplesnia/nightlyDev/240805/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main                                                     
                        rv = self.invoke(ctx)                                                                                                                                              
                      File "/nfs/home/pplesnia/nightlyDev/240805/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke                                                   
                        return ctx.invoke(self.callback, **ctx.params)                                                                                                                     
                      File "/nfs/home/pplesnia/nightlyDev/240805/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke                                                    
                        return __callback(*args, **kwargs)                                                                                                                                 
                      File "/nfs/home/pplesnia/nightlyDev/240805/.venv/lib/python3.10/site-packages/click/decorators.py", line 38, in new_func                                             
                        return f(get_current_context().obj, *args, **kwargs)                                                                                                               
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/interface/commands.py", line 309, in include                                        
                        result = obj.get_driver('controller').include(arguments=data).data                                                                                                 
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/controller_driver.py", line 53, in include                                          
                        return self.send_command('include', data = arguments, outformat = PlainText)                                                                                       
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/utils/shell_utils.py", line 191, in send_command                                               
                        return self.handle_response(response, command, outformat)                                                                                                          
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/utils/shell_utils.py", line 175, in handle_response                                            
                        raise DruncServerSideError(error_txt, stack_txt, server_response=dr)                                                                                               
                    drunc.exceptions.DruncServerSideError: Stacktrace [bold red]on remote server![/bold red]                                                                               
                    Traceback (most recent call last):                                                                                                                                     
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/broadcast/server/decorators.py", line 30, in wrap                                              
                        ret = cmd(obj, request) # we strip the context here, no need for that anymore                                                                                      
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/authoriser/decorators.py", line 34, in check_token                                             
                        ret = cmd(obj, request)                                                                                                                                            
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/decorators.py", line 20, in wrap                                                    
                        return cmd(obj, request)                                                                                                                                           
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/utils/grpc_utils.py", line 65, in unpack_request                                               
                        ret = cmd(obj, **kwargs)                                                                                                                                           
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/controller.py", line 615, in include                                                
                        self.stateful_node.include_node()                                                                                                                                  
                      File "/nfs/home/pplesnia/nightlyDev/240805/sourcecode/drunc/src/drunc/controller/stateful_node.py", line 135, in include_node                                        
                        raise CannotInclude()                                                                                                                                              
                    drunc.controller.stateful_node.CannotInclude: Cannot include node (most likely, it is already included)                                                                

                    drunc.controller.stateful_node.CannotInclude: Cannot include node (most likely, it is already included)                                                                
                     root-controller -> 5                                                                                                                                                  

drunc-unified-shell > status
                        root-controller status                        
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Name               ┃ State      ┃ Substate   ┃ In error ┃ Included ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ root-controller    │ configured │ configured │    No    │   Yes    │
│ ├── ru-controller  │ initial    │ initial    │    No    │   Yes    │
│ ├── df-controller  │ configured │ configured │    No    │   Yes    │
│ ├── trg-controller │ configured │ configured │    No    │   Yes    │
│ └── hsi-controller │ configured │ configured │    No    │   Yes    │
└────────────────────┴────────────┴────────────┴──────────┴──────────┘