Weekly CI Builds Failing on configure-ou.ps1

clong commented 5 years ago

Operating System Version: Ubuntu 16.04
Provider (VirtualBox/VMWare): VirtualBox
Vagrant Version: 2.2.4
Packer Version: n/a
Are you using stock boxes (downloaded) or were they built from scratch using Packer? Stock
Is the issue reproducible or intermittent? Reproducible

Description of the issue:

    dc: Running: scripts/configure-ou.ps1 as c:\tmp\vagrant-shell.ps1
    dc: Creating Server and Workstation OUs...
    dc: Creating Servers OU...
    dc: powershell.exe : Exception calling "Exists" with "1" argument(s): "The specified domain either does not exist or 
    dc:     + CategoryInfo          : NotSpecified: (Exception calli...s not exist or :String) [], RemoteException
    dc:     + FullyQualifiedErrorId : NativeCommandError
    dc: could not be contacted.
    dc: "
    dc: At C:\tmp\vagrant-shell.ps1:4 char:5
    dc: + if (!([ADSI]::Exists("LDAP://OU=Servers,DC=windomain,DC=local")))
    dc: +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    dc:     + FullyQualifiedErrorId : COMException
    dc:  
    dc: Creating Workstations OU
    dc: Exception calling "Exists" with "1" argument(s): "The specified domain either does not exist or 
    dc: could not be contacted.
    dc: "
    dc: At C:\tmp\vagrant-shell.ps1:13 char:5
    dc: + if (!([ADSI]::Exists("LDAP://OU=Workstations,DC=windomain,DC=local")) ...
    dc: +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    dc:     + FullyQualifiedErrorId : COMException
    dc:  
The following WinRM command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

clong commented 5 years ago

This is really bizarre. If I use the exact same build script on the exact same infrastructure, I'm unable to reproduce this problem:

Attempting to bring up the logger host using Vagrant
Good news! logger was built successfully!
Attempting to bring up the dc host using Vagrant
Good news! dc was built successfully!
Attempting to bring up the wef host using Vagrant
Good news! wef was built successfully!
Attempting to bring up the win10 host using Vagrant
Good news! win10 was built successfully!

Seems like some sort of intermittent timing issue or something. I'm going to try re-shuffling the order of scripts in the Vagrantfile and see if that makes a difference.

jsecurity101 commented 5 years ago

Hi @clong , while attempting a clean build, all of the boxes are failing to join the domain, except the logger. The DC will join after a reload, but the WIN10 and WEF will not. Here example of the errors from the WEF:

clong commented 5 years ago

@jsecurity101 whoa, no bueno. Can you fill this out?

Operating System Version: 
Provider (VirtualBox/VMWare):
Vagrant Version: 
Packer Version: 
Are you using stock boxes (downloaded) or were they built from scratch using Packer?

jsecurity101 commented 5 years ago

Operating System Version: Mac OS Mojave 10.14.3 Provider (VirtualBox/VMWare): Virtualbox Vagrant Version: 2.2.4 Packer Version: 1.2.5 Are you using stock boxes (downloaded) or were they built from scratch using Packer? stock

jsecurity101 commented 5 years ago

Hi @clong I was wondering if you needed any more information for this?

Again, I cannot express how thankful I am for the work you put into this man. Please let me know if there is anything I can do to help!!

clong commented 5 years ago

Hey @jsecurity101 to be completely honest I have no idea why this is failing. I actually can't get it to reproduce on the build servers when I kick off the build manually and can't imagine why this part of the process is error-ing out.

I tried forcing the OU creation (https://github.com/clong/DetectionLab/pull/222/files) and ended up with this error:

  dc: Running: scripts/configure-ou.ps1 as c:\tmp\vagrant-shell.ps1
    dc: Creating Server and Workstation OUs...
    dc: Creating Servers OU...
    dc: DEBUG: DC.WINDOMAIN.LOCAL
    dc: powershell.exe : New-ADOrganizationalUnit : An attempt was made to add an object to the directory with a name that 
    dc:     + CategoryInfo          : NotSpecified: (New-ADOrganizat...th a name that :String) [], RemoteException
    dc:     + FullyQualifiedErrorId : NativeCommandError
    dc: is already in use
    dc: At C:\tmp\vagrant-shell.ps1:15 char:3
    dc: +   New-ADOrganizationalUnit -Name "Servers" -Server "dc.windomain.loca ...
    dc: +   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (OU=Servers,DC=windomain,DC=local:String) [New-ADOrga 
    dc:    nizationalUnit], ADException
    dc:     + FullyQualifiedErrorId : ActiveDirectoryServer:8305,Microsoft.ActiveDirectory.Management.Com 
    dc:    mands.NewADOrganizationalUnit
    dc:  
    dc: Creating Workstations OU
    dc: New-ADOrganizationalUnit : An attempt was made to add an object to the directory with a name that 
    dc: is already in use
    dc: At C:\tmp\vagrant-shell.ps1:29 char:3
    dc: +   New-ADOrganizationalUnit -Name "Workstations" -Server "dc.windomain ...
    dc: +   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (OU=Workstations,DC=windomain,DC=local:String) [New-A 
    dc:    DOrganizationalUnit], ADException
    dc:     + FullyQualifiedErrorId : ActiveDirectoryServer:8305,Microsoft.ActiveDirectory.Management.Com 
    dc:    mands.NewADOrganizationalUnit
    dc:  
The following WinRM command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

Without a way to easily reproduce it (aside from automated CI jobs), I'm having a hard time figuring out what the problem is.

jsecurity101 commented 5 years ago

Hey @clong , my friend ran everything on VMware using the stock boxes. Everything is checking out, and he had no problems on a clean build. However I am using VirtualBox, I am not sure why I was getting the errors thrown above. I wanted to update you on what we were seeing on our end. Thank you for your help/work. ` ==> dc: Running provisioner: shell...

dc: Running: scripts/install-sysinternals.ps1 as c:\tmp\vagrant-shell.ps1

dc:     Directory: C:\Tools

dc: Mode                LastWriteTime         Length Name                                             

dc: ----                -------------         ------ ----                                             

dc: d-----        3/15/2019   4:16 PM                Sysinternals                                     

dc:     Directory: C:\ProgramData

dc: Mode                LastWriteTime         Length Name                                             

dc: ----                -------------         ------ ----                                             

dc: d-----        3/15/2019   4:16 PM                Sysmon                                           

dc: Downloading Autoruns64.exe...

dc: Downloading Procmon.exe...

dc: Downloading PsExec64.exe...

dc: Downloading procexp64.exe...

dc: Downloading Sysmon64.exe...

dc: Downloading Tcpview.exe...

dc: Downloading Olaf Hartong's Sysmon config...

dc: Starting Sysmon...

dc: Verifying that the Sysmon service is running...

==> dc: Running provisioner: shell...

dc: Running: scripts/configure-ou.ps1 as c:\tmp\vagrant-shell.ps1

dc: Creating Server and Workstation OUs...

dc: Creating Servers OU...

dc: Creating Workstations OU

==> dc: Running provisioner: shell...

dc: Running: scripts/configure-wef-gpo.ps1 as c:\tmp\vagrant-shell.ps1

dc: Importing the GPO to specify the WEF collector

dc: DisplayName      : Windows Event Forwarding Server

dc: DomainName       : windomain.local

dc: Owner            : WINDOMAIN\vagrant

dc: Id               : 0f14cc0c-0e2d-4c2a-9eaa-6eef5fba8d3d

dc: GpoStatus        : AllSettingsEnabled

dc: Description      : 

dc: CreationTime     : 3/15/2019 4:16:33 PM

dc: ModificationTime : 3/15/2019 4:16:34 PM

dc: UserVersion      : AD Version: 1, SysVol Version: 1

dc: ComputerVersion  : AD Version: 1, SysVol Version: 1

dc: WmiFilter        : 

dc: DisplayName   : Windows Event Forwarding Server

dc: GpoId         : 0f14cc0c-0e2d-4c2a-9eaa-6eef5fba8d3d

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 1

dc: Target        : OU=Servers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: DisplayName   : Windows Event Forwarding Server

dc: GpoId         : 0f14cc0c-0e2d-4c2a-9eaa-6eef5fba8d3d

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 2

dc: Target        : OU=Domain Controllers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: DisplayName   : Windows Event Forwarding Server

dc: GpoId         : 0f14cc0c-0e2d-4c2a-9eaa-6eef5fba8d3d

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 1

dc: Target        : OU=Workstations,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: Importing the GPO to modify ACLs on Custom Event Channels

dc: DisplayName      : Custom Event Channel Permissions

dc: DomainName       : windomain.local

dc: Owner            : WINDOMAIN\vagrant

dc: Id               : e6868932-b25d-42b2-ba6a-371930fe17c0

dc: GpoStatus        : AllSettingsEnabled

dc: Description      : 

dc: CreationTime     : 3/15/2019 4:16:34 PM

dc: ModificationTime : 3/15/2019 4:16:34 PM

dc: UserVersion      : AD Version: 1, SysVol Version: 1

dc: ComputerVersion  : AD Version: 1, SysVol Version: 1

dc: WmiFilter        : 

dc: DisplayName   : Custom Event Channel Permissions

dc: GpoId         : e6868932-b25d-42b2-ba6a-371930fe17c0

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 2

dc: Target        : OU=Servers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: DisplayName   : Custom Event Channel Permissions

dc: GpoId         : e6868932-b25d-42b2-ba6a-371930fe17c0

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 3

dc: Target        : OU=Domain Controllers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: DisplayName   : Custom Event Channel Permissions

dc: GpoId         : e6868932-b25d-42b2-ba6a-371930fe17c0

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 2

dc: Target        : OU=Workstations,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: Updating policy...

dc: Computer Policy update has completed successfully.

dc: User Policy update has completed successfully.

==> dc: Running provisioner: shell...

dc: Running: scripts/configure-powershelllogging.ps1 as c:\tmp\vagrant-shell.ps1

dc: Importing the GPO to enable Powershell Module, ScriptBlock and Transcript logging...

dc: DisplayName      : Powershell Logging

dc: DomainName       : windomain.local

dc: Owner            : WINDOMAIN\vagrant

dc: Id               : 7463f416-03d3-45fe-80b0-6f433a8441b4

dc: GpoStatus        : AllSettingsEnabled

dc: Description      : 

dc: CreationTime     : 3/15/2019 4:16:50 PM

dc: ModificationTime : 3/15/2019 4:16:50 PM

dc: UserVersion      : AD Version: 1, SysVol Version: 1

dc: ComputerVersion  : AD Version: 1, SysVol Version: 1

dc: WmiFilter        : 

dc: DisplayName   : Powershell Logging

dc: GpoId         : 7463f416-03d3-45fe-80b0-6f433a8441b4

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 3

dc: Target        : OU=Workstations,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: DisplayName   : Powershell Logging

dc: GpoId         : 7463f416-03d3-45fe-80b0-6f433a8441b4

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 3

dc: Target        : OU=Servers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: DisplayName   : Powershell Logging

dc: GpoId         : 7463f416-03d3-45fe-80b0-6f433a8441b4

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 4

dc: Target        : OU=Domain Controllers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: Updating policy...

dc: Computer Policy update has completed successfully.

dc: User Policy update has completed successfully.

==> dc: Running provisioner: shell...

dc: Running: scripts/configure-AuditingPolicyGPOs.ps1 as c:\tmp\vagrant-shell.ps1

dc: Configuring auditing policy GPOS...

dc: Importing Domain Controllers Enhanced Auditing Policy...

dc: DisplayName      : Domain Controllers Enhanced Auditing Policy

dc: DomainName       : windomain.local

dc: Owner            : WINDOMAIN\vagrant

dc: Id               : ed9e2b4a-c6b7-4975-9d8b-d5fefb1cc98f

dc: GpoStatus        : UserSettingsDisabled

dc: Description      : 

dc: CreationTime     : 3/15/2019 4:17:05 PM

dc: ModificationTime : 3/15/2019 4:17:05 PM

dc: UserVersion      : AD Version: 1, SysVol Version: 1

dc: ComputerVersion  : AD Version: 1, SysVol Version: 1

dc: WmiFilter        : 

dc: DisplayName   : Domain Controllers Enhanced Auditing Policy

dc: GpoId         : ed9e2b4a-c6b7-4975-9d8b-d5fefb1cc98f

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 5

dc: Target        : OU=Domain Controllers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: Importing Servers Enhanced Auditing Policy...

dc: DisplayName      : Servers Enhanced Auditing Policy

dc: DomainName       : windomain.local

dc: Owner            : WINDOMAIN\vagrant

dc: Id               : 691e630b-f985-4c32-abb8-f3c2577c1741

dc: GpoStatus        : UserSettingsDisabled

dc: Description      : 

dc: CreationTime     : 3/15/2019 4:17:06 PM

dc: ModificationTime : 3/15/2019 4:17:06 PM

dc: UserVersion      : AD Version: 1, SysVol Version: 1

dc: ComputerVersion  : AD Version: 1, SysVol Version: 1

dc: WmiFilter        : 

dc: DisplayName   : Servers Enhanced Auditing Policy

dc: GpoId         : 691e630b-f985-4c32-abb8-f3c2577c1741

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 4

dc: Target        : OU=Servers,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

dc: Importing Workstations Enhanced Auditing Policy...

dc: DisplayName      : Workstations Enhanced Auditing Policy

dc: DomainName       : windomain.local

dc: Owner            : WINDOMAIN\vagrant

dc: Id               : c962f2cc-1e3a-457c-baac-5fd356581f04

dc: GpoStatus        : UserSettingsDisabled

dc: Description      : 

dc: CreationTime     : 3/15/2019 4:17:06 PM

dc: ModificationTime : 3/15/2019 4:17:07 PM

dc: UserVersion      : AD Version: 1, SysVol Version: 1

dc: ComputerVersion  : AD Version: 1, SysVol Version: 1

dc: WmiFilter        : 

dc: DisplayName   : Workstations Enhanced Auditing Policy

dc: GpoId         : c962f2cc-1e3a-457c-baac-5fd356581f04

dc: Enabled       : True

dc: Enforced      : True

dc: Order         : 4

dc: Target        : OU=Workstations,DC=windomain,DC=local

dc: GpoDomainName : windomain.local

==> dc: Running provisioner: shell...`

clong commented 5 years ago

This is really a bizarre problem. During the most recent CI build, this script initially failed and then was successful after the host was rebooted: https://207-86134528-gh.circle-artifacts.com/0/tmp/artifacts/vagrant_up_dc.log

I can't for the life of me figure out why the DC would fail to find the domain if it's the domain controller.

==> dc: Running provisioner: shell...
    dc: Running: scripts/configure-ou.ps1 as c:\tmp\vagrant-shell.ps1
    dc: Creating Server and Workstation OUs...
    dc: Creating Servers OU...
    dc: powershell.exe : Exception calling "Exists" with "1" argument(s): "The specified domain either does not exist or 
    dc:     + CategoryInfo          : NotSpecified: (Exception calli...s not exist or :String) [], RemoteException
    dc:     + FullyQualifiedErrorId : NativeCommandError
    dc: could not be contacted.
    dc: "
    dc: At C:\tmp\vagrant-shell.ps1:4 char:5
    dc: + if (!([ADSI]::Exists("LDAP://OU=Servers,DC=windomain,DC=local")))
    dc: +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    dc:     + FullyQualifiedErrorId : COMException
    dc:  
    dc: Creating Workstations OU
    dc: Exception calling "Exists" with "1" argument(s): "The specified domain either does not exist or 
    dc: could not be contacted.
    dc: "
    dc: At C:\tmp\vagrant-shell.ps1:13 char:5
    dc: + if (!([ADSI]::Exists("LDAP://OU=Workstations,DC=windomain,DC=local")) ...
    dc: +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    dc:     + FullyQualifiedErrorId : COMException
    dc:  
The following WinRM command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

<snip - reload happens here>

==> dc: Running provisioner: shell...
    dc: Running: scripts/configure-ou.ps1 as c:\tmp\vagrant-shell.ps1
    dc: Creating Server and Workstation OUs...
    dc: Creating Servers OU...
    dc: Creating Workstations OU
==> dc: Running provisioner: shell...
    dc: Running: scripts/configure-wef-gpo.ps1 as c:\tmp\vagrant-shell.ps1
    dc: Importing the GPO to specify the WEF collector
    dc: DisplayName      : Windows Event Forwarding Server
    dc: DomainName       : windomain.local
    dc: Owner            : WINDOMAIN\vagrant
    dc: Id               : fc131d64-ee90-4aba-9e85-042f8ba71509

clong commented 5 years ago

After doing a bit of reading, I'm guessing this may be a DNS problem. I'm going to try working around this by adding windomain.local to the hosts file on the DC. Not sure if that's non-sensical, but it's the only idea I have at the moment.

clong commented 5 years ago

I’m testing a fix in this PR: https://github.com/clong/DetectionLab/pull/222

The most recent build passed without any issues, but I’m going to run it one more time to make sure it wasn’t a fluke.

clong commented 5 years ago

Fixed in #222

clong commented 5 years ago

This is still an issue as referenced in Build #222: https://222-86134528-gh.circle-artifacts.com/0/tmp/artifacts/vagrant_up_dc.log

I have no idea how it's possible for the DC to ping the domain and itself successfully and then be unable to reach the domain:

    dc: Running: scripts/configure-ou.ps1 as c:\tmp\vagrant-shell.ps1
    dc: Checking AD services status...
    dc: Pinging dc.windomain.local [fe80::602d:6903:a8c4:d92e%6] with 32 bytes of data:
    dc: Reply from fe80::602d:6903:a8c4:d92e%6: time<1ms 
    dc: Ping statistics for fe80::602d:6903:a8c4:d92e%6:
    dc:     Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
    dc: Approximate round trip times in milli-seconds:
    dc:     Minimum = 0ms, Maximum = 0ms, Average = 0ms
    dc: Pinging windomain.local [192.168.38.102] with 32 bytes of data:
    dc: Reply from 192.168.38.102: bytes=32 time<1ms TTL=128
    dc: Ping statistics for 192.168.38.102:
    dc:     Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
    dc: Approximate round trip times in milli-seconds:
    dc:     Minimum = 0ms, Maximum = 0ms, Average = 0ms
    dc: Creating Server and Workstation OUs...
    dc: Creating Servers OU...
    dc: Creating Workstations OU
    dc: MachineName Name      Status
    dc: ----------- ----      ------
    dc: localhost   adws     Running
    dc: localhost   dns      Running
    dc: localhost   kdc      Running
    dc: localhost   Netlogon Running
    dc: powershell.exe : Exception calling "Exists" with "1" argument(s): "The specified domain either does not exist or 
    dc:     + CategoryInfo          : NotSpecified: (Exception calli...s not exist or :String) [], RemoteException
    dc:     + FullyQualifiedErrorId : NativeCommandError
    dc: could not be contacted.
    dc: "
    dc: At C:\tmp\vagrant-shell.ps1:17 char:5
    dc: + if (!([ADSI]::Exists("LDAP://OU=Servers,DC=windomain,DC=local")))
    dc: +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    dc:     + FullyQualifiedErrorId : COMException
    dc:  
    dc: Exception calling "Exists" with "1" argument(s): "The specified domain either does not exist or 
    dc: could not be contacted.
    dc: "
    dc: At C:\tmp\vagrant-shell.ps1:27 char:5
    dc: + if (!([ADSI]::Exists("LDAP://OU=Workstations,DC=windomain,DC=local")) ...
    dc: +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    dc:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    dc:     + FullyQualifiedErrorId : COMException
    dc:

Totally stuck on this one.

clong commented 5 years ago

Haven't seen this happen in over a month. Closing.

clong commented 4 years ago

🎵This is the bug that never ends...🎵

Re-appeared today: https://375-86134528-gh.circle-artifacts.com/0/tmp/artifacts/vagrant_up_dc.log

clong commented 4 years ago

Hopefully fixed once and for all: https://github.com/clong/DetectionLab/commit/0393d627ad5d02089b283467c1a9c6be8370cf29#diff-332e6a941dcb94459531d80e868b2c45

clong / DetectionLab

Weekly CI Builds Failing on configure-ou.ps1 #221

Description of the issue: