Open thomsonjames opened 7 years ago
thomsonjames
commented
7 years ago I did another test running tcpdump on the BIG-IP after it boots up
[root@host-10:NO LICENSE:] config # tcpdump -i 1.1 -nnvvvNXs 512 tcpdump: listening on 1.1, link-type EN10MB (Ethernet), capture size 512 bytes
I confirmed no DHCP requests leave the 1.1 interface or the 0.0 interface
thomsonjames
commented
7 years ago My last comment about 0.0 and 1.1 is not totally valid. When it works, those Tcpdump commands don't show any DHCP requests either. I can't tcpdump on eth1 so I'm not sure how to capture it.
acme58
commented
7 years ago I have seen something similar but now I can't reproduce it in my test environment. When I havd the problem it looked as if TMM dropped the interfaces for a short time just when the DHCP request was sent. I could see this in /var/log/ltm
Testing on Mitaka. I've successfully patched a BIG-IP image and loaded into Glance. I'm using this userdata.json
{ "bigip": { "system_cmds": [ "echo 'this command runs first!' > /tmp/test.txt", "ls -l / > /tmp/ls-of-root-fs.txt", "echo 'run me last'" ], "change_passwords": "true", "admin_password": "f5test", "root_password": "f5test", "modules": { "auto_provision": "false", "ltm": "nominal", "gtm": "none", "avr": "none", "apm": "none", "afm": "minimum", "asm": "none" }, "network": { "dhcp": "true", "selfip_prefix": "selfip-dhcp-abc-", "vlan_prefix": "network-abc-", "interfaces": { "1.1": { "selfip_allow_service": "default", "selfip_name": "selfip.apps", "vlan_name": "vlan.main" } } } } }On first boot, it does not get DHCP from TMM client. If I reboot it 3-4 more times, I will eventually get a DHCP address.
Log messages:
[root@host-10:NO LICENSE:Standalone] config # cat /var/log/messages Feb 7 06:45:27 localhost notice syslog-ng[1792]: syslog-ng starting up; version=\'2.1.4\' Feb 7 06:47:12 localhost notice syslog-ng[1792]: Configuration reload request received, reloading configuration; Feb 7 06:47:21 localhost warning vmsvc[4229]: [ warning] [vmtoolsd] The vmsvc service needs to run inside a virtual machine. Feb 7 06:47:25 localhost notice openstack-init: Starting OpenStack auto-configuration... Feb 7 06:47:25 localhost notice openstack-init: Forcing /PLATFORM... Feb 7 06:47:35 localhost notice openstack-init: Setting Management interface MTU to default Feb 7 06:47:35 localhost notice openstack-init: Running OpenStack auto-configuration... Feb 7 06:47:35 localhost notice openstack-init: Could not connect to mcpd (attempt 1/20), retrying in 10 seconds... Feb 7 06:47:45 localhost notice openstack-init: Could not connect to mcpd (attempt 2/20), retrying in 10 seconds... Feb 7 06:48:17 localhost notice openstack-init: Could not connect to mcpd (attempt 3/20), retrying in 10 seconds... Feb 7 06:48:33 localhost notice syslog-ng[9363]: syslog-ng starting up; version='2.1.4' Feb 7 06:48:34 localhost notice openstack-init: Could not connect to mcpd (attempt 4/20), retrying in 10 seconds... Feb 7 06:48:49 localhost notice openstack-init: Could not connect to mcpd (attempt 5/20), retrying in 10 seconds... Feb 7 06:48:59 host-10-100-0-103 notice syslog-ng[10514]: syslog-ng starting up; version='2.1.4' Feb 7 06:49:00 host-10-100-0-103 notice openstack-init: Successfully connected to mcpd... Feb 7 06:49:23 host-10 notice syslog-ng[12343]: syslog-ng starting up; version='2.1.4' Feb 7 06:49:25 host-10 notice openstack-init: Saving running configuration... Feb 7 06:49:25 host-10 notice openstack-init: /config/bigip.conf Feb 7 06:49:25 host-10 notice openstack-init: /config/bigip_base.conf Feb 7 06:49:25 host-10 notice openstack-init: /config/bigip_user.conf Feb 7 06:49:25 host-10 notice openstack-init: Saving Ethernet mapping...done Feb 7 06:49:25 host-10 notice openstack-init: Retrieving user-data from config drive... Feb 7 06:49:26 host-10 notice openstack-init: No config drive found Feb 7 06:49:41 host-10 notice openstack-init: Retrieving user-data from http://169.254.169.254... Feb 7 06:49:43 host-10 notice openstack-init: Successfully retrieved user-data from instance metadata service... Feb 7 06:49:50 host-10 notice openstack-init: Retrieving SSH public key from http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key... Feb 7 06:49:52 host-10 notice openstack-init: Could not retrieve SSH public key after 5 attempts, quitting... Feb 7 06:49:52 host-10 notice openstack-init: Found plain text password and (against my better judgment) successfully changed root password... Feb 7 06:49:52 host-10 notice openstack-init: Found plain text password and (against my better judgment) successfully changed admin password... Feb 7 06:49:52 host-10 notice openstack-init: Loading configuration... Feb 7 06:49:52 host-10 notice openstack-init: /config/bigip_user.conf Feb 7 06:49:55 host-10 notice openstack-init: Stopping DHCP client for management interface... Feb 7 06:49:56 host-10 notice openstack-init: Setup VLAN on interface 1.1... Feb 7 06:49:56 host-10 notice openstack-init: Configuring VLAN vlan.main on interface 1.1... Feb 7 06:49:56 host-10 notice openstack-init: tmsh create net vlan vlan.main interfaces add { 1.1 } description "auto-added by openstack-init" mtu 1500 Feb 7 06:49:57 host-10 notice openstack-init: Configuring self IP for interface 1.1... Feb 7 06:49:57 host-10 notice openstack-init: Issuing DHCP request on interface 1.1... Feb 7 06:50:29 host-10 notice setsebool: The allow_dynad policy boolean was changed to on by root Feb 7 06:50:35 host-10 notice openstack-init: Saving after configuring interfaces Feb 7 06:50:38 host-10 notice openstack-init: Saving running configuration... Feb 7 06:50:38 host-10 notice openstack-init: /config/bigip.conf Feb 7 06:50:38 host-10 notice openstack-init: /config/bigip_base.conf Feb 7 06:50:38 host-10 notice openstack-init: /config/bigip_user.conf Feb 7 06:50:38 host-10 notice openstack-init: Saving Ethernet mapping...done Feb 7 06:50:38 host-10 notice openstack-init: No BIG-IP license key found, skipping license activation... Feb 7 06:50:38 host-10 notice openstack-init: Could not locate valid BIG-IP license file, no licensed modules found... Feb 7 06:52:21 host-10 notice openstack-init: Successfully connected to mcpd... Feb 7 06:52:22 host-10 notice openstack-init: Executing system command: echo 'this command runs first!' > /tmp/test.txt... Feb 7 06:52:25 host-10 notice openstack-init: echo 'this command runs first!' > /tmp/test.txt exited properly... Feb 7 06:52:32 host-10 notice openstack-init: Executing system command: ls -l / > /tmp/ls-of-root-fs.txt... Feb 7 06:52:35 host-10 notice openstack-init: ls -l / > /tmp/ls-of-root-fs.txt exited properly... Feb 7 06:52:35 host-10 notice openstack-init: Executing system command: echo 'run me last'... Feb 7 06:52:35 host-10 notice openstack-init: run me last Feb 7 06:52:35 host-10 notice openstack-init: echo 'run me last' exited properly... Feb 7 06:52:35 host-10 notice openstack-init: Successfully connected to mcpd... Feb 7 06:52:35 host-10 notice openstack-init: detected tmm started Feb 7 06:52:44 host-10 notice openstack-init: Saving running configuration... Feb 7 06:52:44 host-10 notice openstack-init: /config/bigip.conf Feb 7 06:52:44 host-10 notice openstack-init: /config/bigip_base.conf Feb 7 06:52:44 host-10 notice openstack-init: /config/bigip_user.conf Feb 7 06:52:55 host-10 notice openstack-init: Saving Ethernet mapping...done Feb 7 06:53:08 host-10 notice openstack-init: Completed OpenStack auto-configuration in 343 seconds...
I started a tcpdump on the dhcp interface in netns space
ip netns exec qdhcp-4c35a314-a986-48ac-9191-ed86edf656eb tcpdump -i tape90b6bfd-3c -nnvvvNXs 512
No DHCP messages show up on this interface. I verified with the same command that when it succeeds a few reboots later, there is a request and response as seen here:
09:28:03.158426 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:0c:b0:a3, length 300, xid 0xb05a023f, Flags [none] (0x0000) Client-Ethernet-Address fa:16:3e:0c:b0:a3 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Discover Parameter-Request Option 55, length 4: Subnet-Mask, BR, MTU, Default-Gateway END Option 255, length 0 PAD Option 0, length 0, occurs 50 0x0000: 4510 0148 0000 0000 8011 3996 0000 0000 E..H......9..... 0x0010: ffff ffff 0044 0043 0134 4966 0101 0600 .....D.C.4If.... 0x0020: b05a 023f 0000 0000 0000 0000 0000 0000 .Z.?............ 0x0030: 0000 0000 0000 0000 fa16 3e0c b0a3 0000 ..........>..... 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0100: 0000 0000 0000 0000 6382 5363 3501 0137 ........c.Sc5..7 0x0110: 0401 1c1a 03ff 0000 0000 0000 0000 0000 ................ 0x0120: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0130: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0140: 0000 0000 0000 0000 ........ 09:28:03.216565 IP (tos 0xc0, ttl 64, id 63949, offset 0, flags [none], proto UDP (17), length 328) 10.200.0.100.67 > 10.200.0.101.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0xb05a023f, Flags [none] (0x0000) Your-IP 10.200.0.101 Server-IP 10.200.0.100 Client-Ethernet-Address fa:16:3e:0c:b0:a3 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Offer Server-ID Option 54, length 4: 10.200.0.100 Lease-Time Option 51, length 4: 86400 RN Option 58, length 4: 43200 RB Option 59, length 4: 75600 Subnet-Mask Option 1, length 4: 255.255.255.0 BR Option 28, length 4: 10.200.0.255 Default-Gateway Option 3, length 4: 10.200.0.1 MTU Option 26, length 2: 1450 END Option 255, length 0 PAD Option 0, length 0, occurs 10 0x0000: 45c0 0148 f9cd 0000 4011 68bf 0ac8 0064 E..H....@.h....d 0x0010: 0ac8 0065 0043 0044 0134 24a6 0201 0600 ...e.C.D.4$..... 0x0020: b05a 023f 0000 0000 0000 0000 0ac8 0065 .Z.?...........e 0x0030: 0ac8 0064 0000 0000 fa16 3e0c b0a3 0000 ...d......>..... 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0100: 0000 0000 0000 0000 6382 5363 3501 0236 ........c.Sc5..6 0x0110: 040a c800 6433 0400 0151 803a 0400 00a8 ....d3...Q.:.... 0x0120: c03b 0400 0127 5001 04ff ffff 001c 040a .;...'P......... 0x0130: c800 ff03 040a c800 011a 0205 aaff 0000 ................ 0x0140: 0000 0000 0000 0000 ........ 09:28:03.596439 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from fa:16:3e:0c:b0:a3, length 300, xid 0xb05a023f, Flags [none] (0x0000) Client-Ethernet-Address fa:16:3e:0c:b0:a3 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: Request Server-ID Option 54, length 4: 10.200.0.100 Requested-IP Option 50, length 4: 10.200.0.101 Parameter-Request Option 55, length 4: Subnet-Mask, BR, MTU, Default-Gateway END Option 255, length 0 PAD Option 0, length 0, occurs 38 0x0000: 4510 0148 0000 0000 8011 3996 0000 0000 E..H......9..... 0x0010: ffff ffff 0044 0043 0134 e5e7 0101 0600 .....D.C.4...... 0x0020: b05a 023f 0000 0000 0000 0000 0000 0000 .Z.?............ 0x0030: 0000 0000 0000 0000 fa16 3e0c b0a3 0000 ..........>..... 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0100: 0000 0000 0000 0000 6382 5363 3501 0336 ........c.Sc5..6 0x0110: 040a c800 6432 040a c800 6537 0401 1c1a ....d2....e7.... 0x0120: 03ff 0000 0000 0000 0000 0000 0000 0000 ................ 0x0130: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0140: 0000 0000 0000 0000 ........ 09:28:03.718176 IP (tos 0xc0, ttl 64, id 64071, offset 0, flags [none], proto UDP (17), length 328) 10.200.0.100.67 > 10.200.0.101.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0xb05a023f, Flags [none] (0x0000) Your-IP 10.200.0.101 Server-IP 10.200.0.100 Client-Ethernet-Address fa:16:3e:0c:b0:a3 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: ACK Server-ID Option 54, length 4: 10.200.0.100 Lease-Time Option 51, length 4: 86400 RN Option 58, length 4: 43200 RB Option 59, length 4: 75600 Subnet-Mask Option 1, length 4: 255.255.255.0 BR Option 28, length 4: 10.200.0.255 Default-Gateway Option 3, length 4: 10.200.0.1 MTU Option 26, length 2: 1450 END Option 255, length 0 PAD Option 0, length 0, occurs 10 0x0000: 45c0 0148 fa47 0000 4011 6845 0ac8 0064 E..H.G..@.hE...d 0x0010: 0ac8 0065 0043 0044 0134 21a6 0201 0600 ...e.C.D.4!..... 0x0020: b05a 023f 0000 0000 0000 0000 0ac8 0065 .Z.?...........e 0x0030: 0ac8 0064 0000 0000 fa16 3e0c b0a3 0000 ...d......>..... 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0090: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00f0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0100: 0000 0000 0000 0000 6382 5363 3501 0536 ........c.Sc5..6 0x0110: 040a c800 6433 0400 0151 803a 0400 00a8 ....d3...Q.:.... 0x0120: c03b 0400 0127 5001 04ff ffff 001c 040a .;...'P......... 0x0130: c800 ff03 040a c800 011a 0205 aaff 0000 ................ 0x0140: 0000 0000 0000 0000 ........
09:28:07.508508 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.101 (ff:ff:ff:ff:ff:ff) tell 10.200.0.101, length 28 0x0000: 0001 0800 0604 0001 fa16 3e0c b0a3 0ac8 ..........>..... 0x0010: 0065 ffff ffff ffff 0ac8 0065 .e.........e 09:28:07.516876 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.101 (ff:ff:ff:ff:ff:ff) tell 10.200.0.101, length 28 0x0000: 0001 0800 0604 0001 fa16 3e0c b0a3 0ac8 ..........>..... 0x0010: 0065 ffff ffff ffff 0ac8 0065 .e.........e 09:28:07.527358 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.101 (ff:ff:ff:ff:ff:ff) tell 10.200.0.101, length 28 0x0000: 0001 0800 0604 0001 fa16 3e0c b0a3 0ac8 ..........>..... 0x0010: 0065 ffff ffff ffff 0ac8 0065 .e.........e 09:28:07.536892 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.101 (ff:ff:ff:ff:ff:ff) tell 10.200.0.101, length 28 0x0000: 0001 0800 0604 0001 fa16 3e0c b0a3 0ac8 ..........>..... 0x0010: 0065 ffff ffff ffff 0ac8 0065 .e.........e 09:28:07.546915 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.101 (ff:ff:ff:ff:ff:ff) tell 10.200.0.101, length 28 0x0000: 0001 0800 0604 0001 fa16 3e0c b0a3 0ac8 ..........>..... 0x0010: 0065 ffff ffff ffff 0ac8 0065 .e.........e
I also validated that the dnsmasq hosts file has the mac address of the instance.
[root@mitaka1 tmp(keystone_admin)]# ps -efa | grep tape90b6bfd-3c nobody 11028 1 0 Feb06 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/var/lib/neutron/dhcp/4c35a314-a986-48ac-9191-ed86edf656eb/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/4c35a314-a986-48ac-9191-ed86edf656eb/host --addn-hosts=/var/lib/neutron/dhcp/4c35a314-a986-48ac-9191-ed86edf656eb/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/4c35a314-a986-48ac-9191-ed86edf656eb/opts --dhcp-leasefile=/var/lib/neutron/dhcp/4c35a314-a986-48ac-9191-ed86edf656eb/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tape90b6bfd-3c --dhcp-range=set:tag0,10.200.0.0,static,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
[root@mitaka1 tmp(keystone_admin)]# cat /var/lib/neutron/dhcp/4c35a314-a986-48ac-9191-ed86edf656eb/host fa:16:3e:3d:4f:05,host-10-200-0-100.openstacklocal,10.200.0.100 fa:16:3e:df:05:ee,host-10-200-0-102.openstacklocal,10.200.0.102
My instance is 10.1.0.102
[root@mitaka1 tmp(keystone_admin)]# nova show 6354eaf0-4502-40ef-9c46-90cacf7a5743 +--------------------------------------+-----------------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------------+ | OS-DCF:diskConfig | AUTO | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | mitaka1.jthomsonf5.com | | OS-EXT-SRV-ATTR:hypervisor_hostname | mitaka1.jthomsonf5.com | | OS-EXT-SRV-ATTR:instance_name | instance-00000004 | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2017-02-07T14:41:00.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2017-02-07T14:40:42Z | | flavor | m1.bigip.large-4x8 (dd902c27-ac9a-459f-95f8-92f71e9e92e3) | | hostId | c6b803be37ae121e52b89856ec4d430e8fdc4ae8967d114e6b6eb925 | | id | 6354eaf0-4502-40ef-9c46-90cacf7a5743 | | image | BIG-IP 12.1.2-All (2bf3171d-2dc4-4ffc-99ac-94dadb0e9246) | | key_name | - | | metadata | {} | | mynet1 network | 10.100.0.103, 10.1.0.173 | | mynet2 network | 10.200.0.102 | | name | s2 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | james-allow | | status | ACTIVE | | tenant_id | 31e64a9a4ee84c22a88fa4ac50d128e0 | | updated | 2017-02-07T14:43:57Z | | user_id | 8ea7a15a05ea4309b654bf8f3807e1b5 | +--------------------------------------+-----------------------------------------------------------+
Maybe the DHCP request needs some sort of a retry if it doesn't get a response? Let me know what else you need. I can pretty easily replicate it.