Some tests of the email_import module

[adulau]

I did some test with email_import (based on the latest version of the VM with misp-modules).

Install related issues (just to keep track of those)

• The modules was not loaded because it was missing pyfaup (maybe the VM script is not including it, need to check if the bootstrap/install script includes it). After manual install, the email_import was loaded (if you have a recent version of PyMISP as the recent tools are required).

Module potential improvement

I did a test with a simple EML export from Google mail.

• Some interesting headers are not included in the object and especially the IP part which is the most interesting part for correlation. In the object we have two fields received-header-ip and received-header-hostnamewhich can be added from parsed headers.

• Another interesting field is the return-path.

Object template improvement

• Delivered-To sounds interesting but it's not in the object template. I'll add it.

Sample EML

Delivered-To: adulau@XXXXX.BE
Received: by 2002:a4a:ab46:0:0:0:0:0 with SMTP id j6csp189557oon;
        Tue, 3 Dec 2019 23:45:01 -0800 (PST)
X-Google-Smtp-Source: APXvYqwdwOEvLlrQH7qZ60zTZCMk/DbQJd+Xov21wgPxjOyV3U/M8hHhH1nJygxMc/1r4h3rTLY2
X-Received: by 2002:a17:906:4bd1:: with SMTP id x17mr1613264ejv.181.1575445501318;
        Tue, 03 Dec 2019 23:45:01 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1575445501; cv=none;
        d=google.com; s=arc-20160816;
        b=athg26ZWgKifzZGUdeSofH383Vu2PEwDyk6xkf3UM3QojvZTI8bA6yZm/ceaHblRa+
         4cTvybQniYLrJ1lAJIIbMCszXqj4AO5cVpjaQHvxwPOXL3foDQbJmG3QjQQT1MfPzo13
         ly/qoLoSLmY1PeXSFBgQTLKEgm8DF/p5An4aaR4Gb3LKIQvorVvpdhHHJDQwg2cj5kyd
         FD9FlB4nFJmOkuprwQRP9PGZ+r16qBdlLQplQQUdKRNUnOx3A/FJZS/AKihUJm2Yy9mg
         /YMPv3EgKapq412Qgart6jcT+U/JTcUZGdh/iCDObuMhqqwCtyZ/YEL9yMILhLlSjbJf
         ru/w==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=reply-to:date:from:to:subject:content-description
         :content-transfer-encoding:mime-version:message-id;
        bh=DwSEPwphL15yxv8vZkrLBa2HPqLxr5CpUE4AOHg/NtQ=;
        b=s3gOVbKXseQ2P9RjhiWCXZRMQ3dv7fhj7YZq9Eq9qULqFsca0h2K5WfIeJ6NwxBhE1
         o64ataRM+zUO3DVccJ+N/UUJ4o20NHJrdR5R/8o+dvaASl4RLP38QkazUu8UvJq3+8yQ
         gX5BdzljOIFvhzJs4Vlyd3+SKKlyCUGI0DLzoysjV49pwncsrHrnIvtvFPxGOSMRBAvA
         ws3zWuv1XNnX2OPQxLPGu/eGZtA95Tfa5mrvreRPYZV81ml5vuD6GmVRLkOZ46lT/c3E
         MAcc9GGofBU7X+S/3XJjbtvUUpM0KgPQF+PrUmsWBnEYBJY8pfA8P4Mv+xAzpwv4EkuY
         4TAQ==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=softfail (google.com: domain of transitioning lambiedavid01@gmail.com does not designate 2a02:21d0::68:69:25 as permitted sender) smtp.mailfrom=lambiedavid01@gmail.com;
       dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com
Return-Path: <lambiedavid01@gmail.com>
Received: from kb.quuxlabs.com (kb.quuxlabs.com. [2a02:21d0::68:69:25])
        by mx.google.com with ESMTPS id 7si4102380ejy.230.2019.12.03.23.45.01
        for <adulau@XXXX.be>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Tue, 03 Dec 2019 23:45:01 -0800 (PST)
Received-SPF: softfail (google.com: domain of transitioning lambiedavid01@gmail.com does not designate 2a02:21d0::68:69:25 as permitted sender) client-ip=2a02:21d0::68:69:25;
Authentication-Results: mx.google.com;
       spf=softfail (google.com: domain of transitioning lambiedavid01@gmail.com does not designate 2a02:21d0::68:69:25 as permitted sender) smtp.mailfrom=lambiedavid01@gmail.com;
       dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com
Message-ID: <5de763fd.1c69fb81.280e9.cbdeSMTPIN_ADDED_MISSING@mx.google.com>
Received: from smtp3.delta.net.id (smtp3.delta.net.id [116.50.25.222])
        by kb.quuxlabs.com (Postfix) with ESMTP id 78B132A2E56
        for <adulau@XXXX.be>; Wed,  4 Dec 2019 08:41:46 +0100 (CET)
Received: from [185.234.216.125] (unknown [185.234.216.125])
        by smtp3.delta.net.id (Postfix) with ESMTP id 0D477671E91
        for <adulau@XXXX.be>; Mon,  2 Dec 2019 08:32:10 +0700 (WIB)
Content-Type: text/plain; charset="iso-8859-1"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Description: Mail message body
Subject: Kindly Consider and Revert...
To: adulau@XXXX.be
From: "Mr. David Lambert" <lambiedavid01@gmail.com>
Date: Sun, 01 Dec 2019 17:58:48 -0800
Reply-To: lambiedavid01@gmail.com

[Rafiot]

pyfaup isn't installable as a pypi package so it is a little bit tricky to install (see https://github.com/stricaud/faup/issues/66)

[pettai]

pyfaup is on pypi now, time to add it to the REQ file