Lock up on campaign launch

[alecisec]

Currently running 7,1 release, all seems OK until I launch the campaign, I did have to manually create the users table in the DB and suspect it may be related to that. This is using AWS 'serverless' RDS (MySQL compatible)

172.17.0.4 - - [01/Aug/2019:03:42:54 +0000] "POST /api/campaigns/?api_key=1976819f2281a5f5e54e8e74718d9c62c301fee14e5a9c30a49b4efba488a3 HTTP/1.0" 201 1310 "https://gophish.t2sjthqs75.ap-southeast-2.elasticbeanstalk.com:3333/campaigns" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"

03:42:54 panic: runtime error: index out of range

03:42:54 goroutine 1165 [running]:

03:42:54 github.com/gophish/gophish/mailer.(*MailWorker).Start.func1(0xb5a180, 0xc420210140, 0x107d5d0, 0x0, 0x0)

03:42:54 /go/src/github.com/gophish/gophish/mailer/mailer.go:86 +0xe3

03:42:54 created by github.com/gophish/gophish/mailer.(*MailWorker).Start

03:42:54 /go/src/github.com/gophish/gophish/mailer/mailer.go:85 +0x81

[alecisec]

The users table looks like this:

mysql> describe users; +----------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | username | varchar(100) | NO | UNI | NULL | | | hash | varchar(100) | YES | | NULL | | | api_key | varchar(100) | NO | UNI | NULL | | +----------+--------------+------+-----+---------+----------------+ 4 rows in set (0.00 sec)

[jordan-wright]

I did have to manually create the users table in the DB and suspect it may be related to that.

That indicates to me that something is wrong with the database connection being used. The error you posted roughly suggests that the mailer received a campaign that had no recipients, which is very odd indeed.

My best recommendation at this point would be to see if the problem occurs with a traditional MySQL instance. If it does, then we can get more information about the campaign your launching to continue troubleshooting.

[alecisec]

Works fine on a normal MySQL DB

This is now the AWS DB looks

mysql> show tables; +-------------------+ | Tables_in_gophish | +-------------------+ | attachments | | campaigns | | email_requests | | events | | goose_db_version | | group_targets | | groups | | headers | | mail_logs | | pages | | results | | smtp | | targets | | templates | | users | +-------------------+ 15 rows in set (0.00 sec)

mysql> describe attachments; +-------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | template_id | bigint(20) | YES | | NULL | | | content | longtext | YES | | NULL | | | type | varchar(255) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | +-------------+--------------+------+-----+---------+----------------+ 5 rows in set (0.00 sec)

mysql> describe campaigns; +----------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user_id | bigint(20) | YES | | NULL | | | name | varchar(255) | NO | | NULL | | | created_date | datetime | YES | | NULL | | | completed_date | datetime | YES | | NULL | | | template_id | bigint(20) | YES | | NULL | | | page_id | bigint(20) | YES | | NULL | | | status | varchar(255) | YES | | NULL | | | url | varchar(255) | YES | | NULL | | | smtp_id | bigint(20) | YES | | NULL | | | launch_date | datetime | YES | | NULL | | | send_by_date | datetime | YES | | NULL | | +----------------+--------------+------+-----+---------+----------------+ 12 rows in set (0.00 sec)

mysql> describe email_requests; +--------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user_id | int(11) | YES | | NULL | | | template_id | int(11) | YES | | NULL | | | page_id | int(11) | YES | | NULL | | | first_name | varchar(255) | YES | | NULL | | | last_name | varchar(255) | YES | | NULL | | | email | varchar(255) | YES | | NULL | | | position | varchar(255) | YES | | NULL | | | url | varchar(255) | YES | | NULL | | | r_id | varchar(255) | YES | | NULL | | | from_address | varchar(255) | YES | | NULL | | +--------------+--------------+------+-----+---------+----------------+ 11 rows in set (0.00 sec)

mysql> describe events; +-------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | campaign_id | bigint(20) | YES | | NULL | | | email | varchar(255) | YES | | NULL | | | time | datetime | YES | | NULL | | | message | varchar(255) | YES | | NULL | | | details | blob | YES | | NULL | | +-------------+--------------+------+-----+---------+----------------+ 6 rows in set (0.00 sec)

mysql> describe goose_db_version; +------------+---------------------+------+-----+-------------------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------+---------------------+------+-----+-------------------+----------------+ | id | bigint(20) unsigned | NO | PRI | NULL | auto_increment | | version_id | bigint(20) | NO | | NULL | | | is_applied | tinyint(1) | NO | | NULL | | | tstamp | timestamp | YES | | CURRENT_TIMESTAMP | | +------------+---------------------+------+-----+-------------------+----------------+ 4 rows in set (0.00 sec)

mysql> describe group_targets ; +-----------+------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-----------+------------+------+-----+---------+-------+ | group_id | bigint(20) | YES | | NULL | | | target_id | bigint(20) | YES | | NULL | | +-----------+------------+------+-----+---------+-------+ 2 rows in set (0.03 sec)

mysql> describe groups; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user_id | bigint(20) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | modified_date | datetime | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ 4 rows in set (0.00 sec)

mysql> describe headers; +---------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | key | varchar(255) | YES | | NULL | | | value | varchar(255) | YES | | NULL | | | smtp_id | bigint(20) | YES | | NULL | | +---------+--------------+------+-----+---------+----------------+ 4 rows in set (0.00 sec)

mysql> describe mail_logs; +--------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | campaign_id | int(11) | YES | | NULL | | | user_id | int(11) | YES | | NULL | | | send_date | datetime | YES | | NULL | | | send_attempt | int(11) | YES | | NULL | | | r_id | varchar(255) | YES | | NULL | | | processing | tinyint(1) | YES | | NULL | | +--------------+--------------+------+-----+---------+----------------+ 7 rows in set (0.00 sec)

mysql> describe pages; +---------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user_id | bigint(20) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | html | mediumtext | YES | | NULL | | | modified_date | datetime | YES | | NULL | | | capture_credentials | tinyint(1) | YES | | NULL | | | capture_passwords | tinyint(1) | YES | | NULL | | | redirect_url | text | YES | | NULL | | +---------------------+--------------+------+-----+---------+----------------+ 8 rows in set (0.00 sec)

mysql> describe results; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | campaign_id | bigint(20) | YES | | NULL | | | user_id | bigint(20) | YES | | NULL | | | r_id | varchar(255) | YES | | NULL | | | email | varchar(255) | YES | | NULL | | | first_name | varchar(255) | YES | | NULL | | | last_name | varchar(255) | YES | | NULL | | | status | varchar(255) | NO | | NULL | | | ip | varchar(255) | YES | | NULL | | | latitude | double | YES | | NULL | | | longitude | double | YES | | NULL | | | modified_date | datetime | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ 12 rows in set (0.00 sec)

mysql> describe smtp; +--------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user_id | bigint(20) | YES | | NULL | | | interface_type | varchar(255) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | host | varchar(255) | YES | | NULL | | | username | varchar(255) | YES | | NULL | | | password | varchar(255) | YES | | NULL | | | from_address | varchar(255) | YES | | NULL | | | modified_date | datetime | YES | | NULL | | | ignore_cert_errors | tinyint(1) | YES | | NULL | | +--------------------+--------------+------+-----+---------+----------------+ 10 rows in set (0.00 sec)

mysql> describe targets; +------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | first_name | varchar(255) | YES | | NULL | | | last_name | varchar(255) | YES | | NULL | | | email | varchar(255) | YES | | NULL | | | position | varchar(255) | YES | | NULL | | +------------+--------------+------+-----+---------+----------------+ 5 rows in set (0.01 sec)

mysql> describe templates; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user_id | bigint(20) | YES | | NULL | | | name | varchar(255) | YES | | NULL | | | subject | varchar(255) | YES | | NULL | | | text | text | YES | | NULL | | | html | mediumtext | YES | | NULL | | | modified_date | datetime | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ 7 rows in set (0.00 sec)

mysql> describe users; +----------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | username | varchar(100) | NO | UNI | NULL | | | hash | varchar(100) | YES | | NULL | | | api_key | varchar(100) | NO | UNI | NULL | | +----------+--------------+------+-----+---------+----------------+ 4 rows in set (0.00 sec)

[alecisec]

I've tried starting from scratch with a new database and the 7.1 release - I'm now getting an error

time="2019-08-05T06:04:49Z" level=warning msg="No contact address has been configured." time="2019-08-05T06:04:49Z" level=warning msg="Please consider adding a contact_address entry in your config.json" time="2019-08-05T06:04:49Z" level=info msg="Background Worker Started Successfully - Waiting for Campaigns" goose: migrating db environment 'production', current version: 0, target: 20180830215615 2019/08/05 06:04:49 FAIL 20160118194630_init.sql (Error 1071: Specified key was too long; max key length is 767 bytes), quitting migration.

Is gophish not compatible with mysql 5.6.x ?

[toniopelo]

I do have the same problem, and I am using AWS Aurora as well (but compatible with mysql 8.0.23)

gophish_1          | panic: runtime error: index out of range [0] with length 0
gophish_1          | 
gophish_1          | goroutine 206 [running]:
gophish_1          | github.com/gophish/gophish/mailer.(*MailWorker).Start.func1(0xcdc820, 0xc0000aa010, 0x117d7d8, 0x0, 0x0)
gophish_1          |    /go/src/github.com/gophish/gophish/mailer/mailer.go:85 +0xe5
gophish_1          | created by github.com/gophish/gophish/mailer.(*MailWorker).Start
gophish_1          |    /go/src/github.com/gophish/gophish/mailer/mailer.go:84 +0x85

I'll try to investigate further and come back here if I find something useful. Did you solve your problem @alecisec ?

[toniopelo]

My best recommendation at this point would be to see if the problem occurs with a traditional MySQL instance. If it does, then we can get more information about the campaign your launching to continue troubleshooting.

@jordan-wright It happens even on a traditional MySQL instance (using mysql latest docker hub image)

[toniopelo]

Ok, I finally found the problem here.

Probelm

• When a campaign is supposed to be sent immediately, it goes through PostCampaign here
• The campaign model is then created and persisted in database along with rows in the mail_logs table that represent each mail that should be sent with the send_date.
• Then, because the campaign should start immediately, it goes through LaunchCampaign here where we directly fetch the rows from the mail_logs and send what should be sent instead of waiting the next time the worker will poll the database to send them (every minutes).
• The problem is here inside LaunchCampaign. Because we fetched the mail_logs and we are most probably in the same second than the time we inserted it, and because on mysql the send_date field is set to DATETIME which doesn't take milliseconds into account, we can end up with a send_date a few milliseconds later than current time because mysql will round the DATETIME up to the upper second if milliseconds are > 500ms. That will cause the mailEntries array to be empty and will then fail in the worker when trying to access mailEntries[0] and bring gophish down.

Soluce

Update mysql schema to accept milliseconds, you can use the following statement :

ALTER TABLE `gophish`.`mail_logs` MODIFY send_date DATETIME(3);

[glennzw]

Hi @toniopelo - this is some great detective work, very nicely done!

Would you be able to test this on a later version? The docker image is a little old.

Also, could you help me reproduce the steps to cause this crash

[glennzw]

I've updated the docker images so you can test on the latest

[toniopelo]

@glennzw Sorry for the delay! My investigations were made on the latest version at the time of writing. I did fix the Dockerfile for that and I was going to open a PR about it but you did it already so that's fine now :).