[ECS] Upgrade modules to 1.8

[andrewstucki]

This is to track changes needed to upgrade modules to ECS 1.8:

• [x] After 1.8 is released, update ecs dependency to 1.8

Carry-over from 1.7 upgrade:

• [ ] network.direction Filebeat cisco umbrella (need CIDR matching processors/painless support in elasticsearch) - can leverage https://github.com/elastic/elasticsearch/pull/66644
• [ ] network.direction in Filebeat rsa2elk modules - https://github.com/elastic/beats/issues/23114

Add os.type field:

• [x] add_host_metadata processor (@adriansr) https://github.com/elastic/beats/pull/23513
• [x] other places that fill in os information (@adriansr) https://github.com/elastic/beats/pull/23513

New event.category value registry:

• [x] Winlogbeat (@marc-gr) https://github.com/elastic/beats/pull/23563

New event.category value session:

• [x] Winlogbeat (@marc-gr) https://github.com/elastic/beats/pull/23563
• [x] Auditbeat (@adriansr)
  • [x] beats https://github.com/elastic/beats/pull/23594
  • [ ] integrations (auditd ingest pipeline)
• [x] Filebeat auditd fileset (@adriansr)
  • [x] beats https://github.com/elastic/beats/pull/23723
  • [x] integrations https://github.com/elastic/integrations/pull/715

Multiple users in an event elastic/ecs#914:

• [x] Auditbeat (sudo, iam events, AUDIT_CH_ID events, file ownership syscalls, set/getid syscalls, all auid values) (@adriansr) #23594
• [x] Journalbeat (maybe?, sudo & iam events) (@marc-gr) https://github.com/elastic/beats/pull/23737
• [x] Packetbeat (@marc-gr) https://github.com/elastic/beats/pull/23783
• [x] Winlogbeat (Run As, iam events) (@marc-gr)
  • [x] beats https://github.com/elastic/beats/pull/23563
  • [x] integrations https://github.com/elastic/integrations/pull/685
• [x] Filebeat auditd (same as auditbeat) (@adriansr)
  • [x] beats https://github.com/elastic/beats/pull/23723
  • [x] integrations https://github.com/elastic/integrations/pull/715
• [x] Filebeat rsa2elk modules (@adriansr) (no changes)
• [x] Filebeat checkpoint firewall (@marc-gr) (no changes)
• [x] Filebeat cisco asa (@marc-gr)
  • [x] beats #23819
  • [x] integrations https://github.com/elastic/integrations/pull/686
• [x] Filebeat cef (@marc-gr)
  • [x] beats https://github.com/elastic/beats/pull/23832
  • [x] integrations https://github.com/elastic/integrations/pull/687
• [x] Filebeat cisco ftd (@marc-gr)
  • [x] beats #23819
  • [x] integrations https://github.com/elastic/integrations/pull/686
• [x] Filebeat cisco umbrella (@marc-gr)
  • [x] beats #23819 ~- [ ] integrations~
• [x] Filebeat crowdstrike falcon (@marc-gr)
  • [x] beats #23875
  • [x] integrations https://github.com/elastic/integrations/pull/688
• [x] Filebeat fortinet firewall (@marc-gr)
  • [x] beats #23902
  • [x] integrations https://github.com/elastic/integrations/pull/689
• [x] Filebeat googlecloud audit (iam events) (@adriansr) (no changes, discuss)
  • [x] beats
  • [x] integrations
• [x] Filebeat microsoft (@adriansr)
  • [x] beats #23897
  • [x] integrations https://github.com/elastic/integrations/pull/468
• [x] Filebeat elasticsearch/audit (maybe?) (@marc-gr)
  • [x] beats #24000 ~- [ ] integrations~
• [x] Filebeat Gsuite/Workspace (@marc-gr)
  • [x] beats #23709
  • [x] integrations https://github.com/elastic/integrations/pull/690
• [x] Filebeat o365 (Actors, iam events) (@adriansr)
  • [x] beats #23896
  • [x] integrations https://github.com/elastic/integrations/pull/716
• [x] Filebeat zoom (@adriansr)
  • [x] beats #23904
  • [x] integrations https://github.com/elastic/integrations/pull/719
• [x] Filebeat okta (maybe? actors, iam events, targets) (@marc-gr)
  • [x] beats #23929
  • [x] integrations https://github.com/elastic/integrations/pull/691
• [x] Filebeat aws cloudtrail (assumed role, iam events) (@adriansr)
  • [x] beats #23911
  • [x] integrations https://github.com/elastic/integrations/pull/721
• [x] Filebeat aws s3access (assumed role) (@adriansr)
  • [x] beats #23920
  • [x] integrations https://github.com/elastic/integrations/pull/721
• [x] Filebeat azure (@adriansr) (not a lot of test data, discuss)
  • [x] beats #23927
  • [x] integrations https://github.com/elastic/integrations/pull/722
• [x] Filebeat juniper/srx (@marc-gr)
  • [x] beats #23936
  • [x] integrations https://github.com/elastic/integrations/pull/692
• [x] Filebeat panw (@marc-gr)
  • [x] beats #23931
  • [x] integrations https://github.com/elastic/integrations/pull/694
• [x] Filebeat sophos/xg (@adriansr)
  • [x] beats #23967
  • [x] integrations https://github.com/elastic/integrations/pull/479
• [x] Filebeat system/auth (sudo) (@marc-gr)
  • [x] beats #23961
  • [x] integrations https://github.com/elastic/integrations/pull/695
• [x] Filebeat mysql/mysqlenterprise (@adriansr)
  • [x] beats https://github.com/elastic/beats/pull/23978 ~- [ ] integrations~
• [x] Filebeat zeek (@marc-gr)
  • [x] beats #23847
  • [x] integrations https://github.com/elastic/integrations/pull/696
• [x] Review other modules to make sure none missing

---

• [x] Make all Beats and modules report ECS 1.8.0 https://github.com/elastic/beats/pull/23992

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[navilg]

Hi,

Below are Python2 related vulnerabilities in filebeat v7.10.1. I suppose that can not be fixed unless we completely move from Python2 to Python3. Are we expecting these to be fixed in 7.11 or 7.12.0

Thanks

CVE-2014-4650 CVE-2016-5636 CVE-2017-1000158 CVE-2019-9636 CVE-2019-9948 CVE-2013-1753 CVE-2014-1912 CVE-2015-5652 CVE-2017-17522 CVE-2018-1060 CVE-2018-1061 CVE-2018-14647 CVE-2019-13404 CVE-2019-16056 CVE-2019-20907 CVE-2019-5010 CVE-2019-9674

[andrewstucki]

@navilg thanks for your interest in beats. A couple of things.

1. AFAIK we migrated all of our tooling to Python 3 some months back, just prior to Python 2 being EOL, so I don't believe this is a problem.
2. This issue is unrelated to to Python and has to do with schema changes from Elastic Common Schema being incorporated into the project.
3. If, on the off chance you wish to disclose any security issues, please follow the guide at https://www.elastic.co/community/security and email security@elastic.co for responsible disclosure

Thank you!