dkindlund
commented
14 years ago Author: kindlund In general, running all the VMs required on a Linux server with only 1GB of RAM will likely lead to resource starvation issues... that's partly why the UserGuide recommends the host have at least 1.5GB free (we assume the host OS will use at least 512MB).
The sporadic RPC Communications Failures suggest that either the MySQL database or the Drone rails webservice is slowing down such that the SOAP calls are timing out. To trace down exactly why this slowdown is happening, I'd recommend you look at the drone/log/development.log or drone/log/production.log files to see if there are any errors recorded in those files around the time you start experiencing the timeouts.
More than likely, I'm guessing the MySQL database may have encountered some type of deadlock (due to resource exhaustion issues), which caused the Drone webservice to partially fail. To troubleshoot further, I'd need to know the corresponding drone/log/*.log error messages, along with your copy of the drone/app/controllers/hc_database_controller.rb, since that entire logic has been rewritten numerous times since I last looked at it.
Frankly, we're in the midst of an infrastructure overhaul, which is why we haven't released new versions of the code. Once stable, the latest version has: ESX support, REST integration with Drone, and uses message queues rather than make expensive database calls in order to obtain more work.
-- Darien
Hello,
I'm using a non-modified version of the latest "packages" available (plus corrections so that the Firewall doesn't connect to svn). I'm using the Drone service and it works (I can access tho the webui), but sometimes it gets unresponsive.
Although I wouldn't care if only the webui had problems (I'm confortable with MySQL query browser), adding URLs through command line often gives me that :
ERRORHoneyClient::Manager::Database::_AUTOLOAD - Error :RPC communications failure, and then I can see in the database that only part of the urls sent have been received (how many varies).
I can't add the URLs "by hand" as I want to add a relatively large queue (roughly 1000 urls to open), so I tried 2 solutions : _ modificating insert_queueurls.pl writing my own script to read the urls from an external file then send them using the same method than insert_queue_urls.pl.
I'm not trying to send my 1000 urls in one request (my script automatically sends a fixed, relatively small (<20, currently I'm trying using 4) number of links per request then sleeps x seconds (5).
This is strange because once I was able to send roughly 280 by packets of 20 before getting the error. Restarting drone doesn't always/rarely solve the problem. My computer has less ram (only 1GB - I can't do anything against that) but there's still free ram, no swap, and nothing heavy is running appart from what's required (ie : just firefox with a few, static pages, vmwareserver and sometimes its console, drone, 2 to 5 terminals and one or 2 distant terminals, sometimes gedit, and sometimes the Honeyclient manager is running). I'm using the latest Ubuntu (up to date) and VMware Server 1.0.9 and appart from that (and VM not unregistering automatically when suspicious), everything's fine.
_ At last : I tried adding the URLs directly to the database (through a python script & the mysql library). The honeyclient visited the links, but they weren't deleted from the queue, so when asking for more urls, it was getting always the same 20 firsts. I was only setting the priority and url. Any cue what happened ?