AWS Integration: Testing

[jesuslinares]

Testing of https://github.com/wazuh/wazuh/issues/1522. Branch: https://github.com/wazuh/wazuh/tree/dev-aws

50th week sprint: Test services:

• [x] Cloudtrail
• [x] VPC
• [x] GuardDuty
• [x] Custom: KMS
• [x] Inspector

Test behavior:

• [x] All the files all collected.
• [x] Bucket with more than 1000 no processed logs.

51th week sprint: Test services:

• [x] Config
• [x] Macie

Test behavior:

• [x] Upgrade from v3.7.1 to this version (using cloudtrail and custom).
• [x] Review Elasticsearch logs (no parse mapping error)

[druizz90]

I discovered problems with Config pulling because this service doesn't store their logs properly (#2110).

Macie hasn't been tested (we haven't got alerts from it).

The rest of the services work fine (all my tests were OK).

[druizz90]

There is a problem with --reparse parameter in AWS Config. I opened this issue: #2157.

[jesuslinares]

Mapping error in Elasticsearch with updatedAt field.

[2018-12-18T13:43:52,389][DEBUG][o.e.a.b.TransportShardBulkAction] [wazuh-alerts-3.x-2018.12.18][2] failed to execute bulk item (index) BulkShardRequest [[wazuh-alerts-3.x-2018.12.18][2]] containing [index {[wazuh-alerts-3.x-2018.12.18][wazuh][h3ahwmcB8Q4rRLigozXF], source[n/a, actual length: [5kb], max length: 2kb]}] org.elasticsearch.index.mapper.MapperParsingException: failed to parse [data.aws.updatedAt] at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]

Configuration:

  <wodle name="aws-s3">

    <disabled>no</disabled>
    <interval>1m</interval>
    <run_on_start>yes</run_on_start>
    <skip_on_error>no</skip_on_error>

    <service type="inspector">
      <only_logs_after>1970-JAN-01</only_logs_after>
    </service>

    <bucket type="custom">
      <name>ta-test</name>
    </bucket>

    <bucket type="guardduty">
      <name>wazuh-aws-wodle</name>
      <path>guardduty</path>
    </bucket>

    <bucket type="cloudtrail">
      <name>wazuh-cloudtrail</name>
    </bucket>

    <bucket type="config">
      <name>wazuh-aws-wodle</name>
      <path>config</path>
      <only_logs_after>1970-JAN-01</only_logs_after>
    </bucket>

    <bucket type="vpcflow">
      <name>wazuh-aws-wodle</name>
      <path>vpc</path>
    </bucket>

    <bucket type="custom">
      <name>wazuh-aws-wodle</name>
      <path>kms_compress_encrypted</path>
      <only_logs_after>1970-JAN-01</only_logs_after>
    </bucket>

    <bucket type="custom">
      <name>wazuh-aws-wodle</name>
      <path>macie</path>
    </bucket>