Open MachinesJesus opened 3 years ago
Really I just need a 'remove all agents' that works reliably. For my system tests I re-build all the agents each time (only take ~20 secs).
@MachinesJesus can you tell me how many agents you are targeting? From what I see you have 4. I have a feeling there is some timeouts going on that are happening.
Using the scripts/install-agent.py --force script you can rebuild/install without replacing the payloads or changing vip identities.
I will take a look and see if I can reproduce on my system and go from there as far as figuring out the issue.
Yeah I think you may have said add time delays on Slack. So I have 1 sec between every remove.
I guess the questions is: how does the agent get created, or half deleted and end up tagless.
I believe the tagless nature of the ghost agent may indicate where the code needs to be more robust.
that is truly a question...one thing that I do know using the normal time.sleep in python that's probably not going to help because of the coroutine (I mean it may)...but using gevent.sleep allows switching of context until the command is done. I will see why this is not being atomic in nature...because I believe that is truly what's going on here.
Here is the data: Starting VOLTTRON verbosely in the background with VOLTTRON_HOME=/home/ubuntu/.volttron Waiting for VOLTTRON to startup.. VOLTTRON startup complete
Before Remove:
8 awspagent-0.1 awspagent-0.1_1 awsp1
c awssagent-0.1 awssagent-0.1_1 awss1
a ec2marketagent-0.1 ec2marketagent-0.1_1 aws_ec2
AGENT IDENTITY TAG PRI
Removing agents...
remove: error: agent not found: bat remove: error: agent not found: listener remove: error: agent not found: zig remove: error: agent not found: plc Removing c9b7df8b-16e5-4a27-a2b2-8afa2f31f554 awssagent-0.1 Removing a4e8e681-02cc-4683-93fb-87ccb2b6edb6 ec2marketagent-0.1 Removing 89a6e12c-cfe4-42b4-b4bd-eaea6017a3a9 awspagent-0.1 remove: error: agent not found: test remove: error: agent not found: stupid
After Remove: (After full remove should be empty below) No installed Agents found
Re-installing...
{
"agent_uuid": "99f74afc-d5f9-4c4c-a29b-21eed9a70ee7"
}
ERROR:install-agent.py:
Error installing agent: Below Command failed with non zero exit code.
Command:['/home/ubuntu/te/vtrn/env/bin/volttron-ctl', 'install', '/home/ubuntu/.volttron/packaged/awspagent-0.1-py3-none-any.whl', '--tag', 'awsp1']
Stderr:
b'2020-11-11 10:41:39,775 () volttron.platform.vip.agent.core ERROR: No response to hello message after 10 seconds.\n2020-11-11 10:41:39,776 () volttron.platform.vip.agent.core ERROR: Type of message bus used zmq\n2020-11-11 10:41:39,776 () volttron.platform.vip.agent.core ERROR: A common reason for this is a conflicting VIP IDENTITY.\n2020-11-11 10:41:39,776 () volttron.platform.vip.agent.core ERROR: Another common reason is not having an auth entry onthe target instance.\n2020-11-11 10:41:39,776 () volttron.platform.vip.agent.core ERROR: Shutting down agent.\n2020-11-11 10:41:39,776 () volttron.platform.vip.agent.core ERROR: Possible conflicting identity is: control.connection\ninstall: operation timed out\n'
NoneType: None
Traceback (most recent call last):
File "scripts/install-agent.py", line 387, in
It always seems to be on the second agent install FYI.
Can I alter the 10 seconds? "No response to hello message after 10 seconds." Can I make that 2 seconds?
https://github.com/VOLTTRON/volttron/blob/125d7a0a47d0f2a1eaf130016a5197a787f52ceb/volttron/platform/vip/agent/core.py#L588 is where that wait is located.
This is in the main branch, but should be very close on the others if you want to modify it.
Note the other thing you might try is --skip-requirements. Assuming your agent has a requirements.txt file it will attempt to install those each time an install. It would be good to skip during a force install, but that's not the way the code is written currently.
Hey @craig8, Gave that a go (above) and also changed the timeout=10.0 to 2.0 on line 600. Rebuilt and installed. Weirdly I still think it took 10 secs. Also, I got 2 in a row for the first time. I'm gonna try a few more tests se if I can pinpoint the phenomena.
Yeah appears to always be the second agent install.
Description of Issue
System Test leads to unlabelled agent that then must be manually removed (non-auto)
Affected Version
7.0 but can test any.
Screenshots
Expected
Actual
About 20% of the time this fails. Then I have to go and manually remove the agent.
Steps to Reproduce
Use: os.system('vctl remove --tag aws* --force')
Change aws* to your agent prefix. Repeat the system remove, re-build, run cycle. 20% of the time it will fail in the removal process, requiring intervention.
System should have min 3 agents that are auto-remove and rebuilt.
Additional Details
(volttron) ubuntu@ubuntu:~/te/vtrn$ vctl status AGENT IDENTITY TAG STATUS HEALTH 0 awspagent-0.1 awspagent-0.1_1 awsp1
1 awssagent-0.1 awssagent-0.1_2 awss1
e awssagent-0.1 awssagent-0.1_1
9 ec2marketagent-0.1 ec2marketagent-0.1_1 aws_ec2
This shows me manually removing...
(volttron) ubuntu@ubuntu:~/te/vtrn$ vctl remove e Removing ebacd847-6e6a-4c8e-aad3-b6701dc96055 awssagent-0.1