search

dkindlund / honeyclient

MITRE HoneyClient Project
http://www.honeyclient.org
GNU General Public License v2.0
8 stars 4 forks source link

VM Snapshots #197

Closed dkindlund closed 14 years ago

dkindlund commented 14 years ago

Hello,

I have been running the Honeyclient for some time without issue. However, it seems as though I have stumbled upon a new problem this morning. For whatever reason, the VM can no longer successfully take a snapshot. I get an error message and the Honeyclient closes. However, I also still get a snapshot on my hard drive. I will attach the error message to this submission. Any help would be greatly appreciated. Thank you,

dkindlund commented 14 years ago

Author: kindlund Hi Jason,

I've come across this particular issue also, and it stems from the following use cases:

Let me know if either of these scenarios apply to your situation. If you do have an NFS share, check the syslog to see if (perhaps) the NFS share temporarily disconnected... you should see something like "NFS: timed out".

-- Darien

dkindlund commented 14 years ago

Author: jdortiz

Hi Darien,

Well, the /vm/snapshots directory has 49GB of free space and it is on the local machine and is not mounted on any NFS share.

Could there be a problem in the setup? I have changed some things as the experiment has continued forward. What files define the standards for taking a snapshot?

Thanks,

Jason

dkindlund commented 14 years ago

Author: anonymous Hi Jason,

Okay, so based on the error message you've attached, it looks like the problem is entirely contained within the snapshotVM() calling library. Eventually, this call creates a child thread that handles the heavy lifting of copying the VM files. If the child thread fails, then the overall program eventually shuts down.

So, if we step through the errors, here's what we know:

{{{ Use of uninitialized value $pwd in chdir at lib/HoneyClient/Manager/VM.pm line 4473. Use of chdir('') or chdir(undef) as chdir() is deprecated at lib/HoneyClient/Manager/VM.pm line 4473. }}}

This seems to be more of a warning. It's caused by the chdir() call on line 4473 within the VM.pm file. Essentially, the idea is that before the child thread performs all the necessary file manipulations, it has to temporarily change the current working directory -- this is so that when the VM gets archived, the archive processes files based upon the relative directory to /vm/clones. Ultimately, this makes the .tar.gz archive easier to read/handle.

Anyway, before the child thread changes directories, it tries to figure out what the current directory is that the script is running within. (This is from looking at $ENV{PWD}.) So, if you're running the perl code from within some sort of locked down shell, where PWD isn't defined, then this warning would probably ensue.

Bottom line: I don't think this warning is causing your problems, but to be sure, you may want to run the StartManager.pl code from within a bash shell that actually has the "PWD" environment variable set (just type env | grep -i pwd to confirm within the shell).

{{{ /bin/tar: 1aed5cda5e212e278b75483200/master.vmem: file changed as we read it }}}

This error is your actual culprit. It basically says that it tried to archive the VM, but it appears that the master.vmem file was changing as tar was running -- which is bad. This means that the VM wasn't properly suspended by the time that tar was told to archive the file.

As such, there's something going on where VM unregister and suspend operations aren't happening properly. In this case, are you sure that "1aed5cda5e212e278b75483200" was both properly SUSPENDED and UNREGISTERED by the time the tar operation starts?

When you run your test again, you can verify if this is happening.

Lastly, the fact that this just started occurring and you have not changed anything within the Honeyclient system tells me that there was probably something that changed OUTSIDE the Honeyclient system (within the host) that may have caused this behavior.

When simple operations like suspend and unregister stop working, then it's probable that simple operations like using the VMware Remote Console will also stop working. In which case, VMware Server is having fundamental issues.

Most likely culprit (assuming you haven't upgraded/changed VMware Server recently) is that some other process is on the host system (that's not Honeyclient related) is eating up CPU resources, causing Honeyclient-related operations to fail.

Also, as an aside note, if you did upgrade to VMware Server v2.x, the Honeyclient code will not support that version -- only v1.x. Though, I suspect you did not, because we would have seen different error messages.

Hope this helps,

-- Darien

dkindlund commented 14 years ago

Author: psi Hi,

I have just started to use Honeyclient and I have the same problem. The only difference from the above is that I do not get the

{{{ Use of uninitialized value $pwd in chdir at lib/HoneyClient/Manager/VM.pm line 4473. Use of chdir('') or chdir(undef) as chdir() is deprecated at lib/HoneyClient/Manager/VM.pm line 4473. }}}

messages. Just the error regarding snapshot processing. In fact the snapshot is created. After I 'tar zxvf' it I can add it to VMware server console and power it up. My main problem is that after this error Honeyclient Manager exits and I have to run it again manually. I attach output from both manager and agent (this one captured after I power up the untared snapshot). Thanks in advance for any suggestions.

dkindlund commented 14 years ago

Author: kindlund Hi psi,

Okay, here's your clue as to what is going wrong: {{{ /bin/tar: dbdd02e66b01f03f86dda87834/master.vmem: file changed as we read it }}}

This essentially means that the tar process was tried to create a .tar.gz archive of the contents of the clone VM, but somehow, the clone VM's .vmem file has changed. This can happen in the following scenarios:

So, to rule out scenario 2, can you please upload the following files as attachments to this ticket?

-- Darien

dkindlund commented 14 years ago

Author: anonymous hi,Darien. I met the same problem you are talking about in #197.

I upload the clone.vmx and master.vmx you just mentioned.

dkindlund commented 14 years ago

Author: anonymous clone.vmx

!/usr/bin/vmware

config.version = "7" virtualHW.version = "4" scsi0.present = "TRUE" memsize = "384" ide0:0.present = "TRUE" ide0:0.fileName = "/vm/master/master.vmdk" ide0:0.mode = "independent-persistent" ide0:0.writeThrough = "TRUE" ide1:0.present = "TRUE" ide1:0.fileName = "auto detect" ide1:0.deviceType = "atapi-cdrom" floppy0.startConnected = "FALSE" floppy0.fileName = "/dev/fd0" Ethernet0.present = "TRUE" Ethernet0.connectionType = "hostonly" displayName = "600a1c1eac6ea8709b797a4270" guestOS = "winxppro" priority.grabbed = "normal" priority.ungrabbed = "normal" powerType.powerOff = "hard" powerType.powerOn = "hard" powerType.suspend = "hard" powerType.reset = "hard"

ide0:0.redo = "" ethernet0.addressType = "generated" uuid.location = "56 4d 1a 5f 2c 3a d1 93-c2 f9 9e 5c e7 55 5a 33" uuid.bios = "56 4d 1a 5f 2c 3a d1 93-c2 f9 9e 5c e7 55 5a 33" ethernet0.generatedAddress = "00:0c:29:55:5a:33" ethernet0.generatedAddressOffset = "0"

ide1:0.startConnected = "FALSE" floppy0.present = "FALSE"

tools.syncTime = "TRUE"

checkpoint.vmState = ""

ide0:0.deviceType = "ata-hardDisk"

usb.present = "TRUE"

ide1:0.autodetect = "TRUE"

dkindlund commented 14 years ago

Author: anonymous master.vmx

!/usr/bin/vmware

config.version = "7"

virtualHW.version = "4"

scsi0.present = "TRUE"

memsize = "384"

ide0:0.present = "TRUE"

ide0:0.fileName = "/vm/master/master.vmdk"

ide0:0.mode = "independent-persistent"

ide0:0.writeThrough = "TRUE"

ide1:0.present = "TRUE"

ide1:0.fileName = "auto detect"

ide1:0.deviceType = "atapi-cdrom"

floppy0.startConnected = "FALSE"

floppy0.fileName = "/dev/fd0"

Ethernet0.present = "TRUE"

Ethernet0.connectionType = "hostonly"

displayName = "master"

guestOS = "winxppro"

priority.grabbed = "normal"

priority.ungrabbed = "normal"

powerType.powerOff = "hard"

powerType.powerOn = "hard"

powerType.suspend = "hard"

powerType.reset = "hard"

ide0:0.redo = ""

ethernet0.addressType = "generated"

uuid.location = "56 4d 3e f2 45 3a 80 e1-6b e6 bc 23 ea 9f 8e 22"

uuid.bios = "56 4d 3e f2 45 3a 80 e1-6b e6 bc 23 ea 9f 8e 22"

ethernet0.generatedAddress = "00:0c:29:9f:8e:22"

ethernet0.generatedAddressOffset = "0"

ide1:0.startConnected = "FALSE"

floppy0.present = "FALSE"

tools.syncTime = "TRUE"

checkpoint.vmState = ""

ide0:0.deviceType = "ata-hardDisk"

usb.present = "TRUE"

ide1:0.autodetect = "TRUE"

dkindlund commented 14 years ago

Author: kindlund Anonymous: There is a problem with your master.vmx configuration; see this ticket for more information:

http://www.honeyclient.org/trac/ticket/148#comment:9

Specifically, this line in your .vmx: {{{ ide0:0.mode = "independent-persistent" }}}

That should be: {{{ ide0:0.mode = "persistent" }}}