chore: remove assertions related to output files

[mdelapenya]

This PR is still a PoC, please do not merge until further clarification

What is this PR doing?

It removes all assertions related to the output.json file for any Beat

Why is it important?

We detected an error in the APM Server pipeline, which is triggering this beats-tester project for the APM Server, in which the output.json file was never there.

After a lot of debugging, and thanks to @axw we knew about this:

APM Server no longer writes out an event on startup. For it to write events to the file, we'll either need to have the Ansible stuff send an APM Agent event payload (e.g. just using curl), or to configure the server's self-instrumentation and send a request to the server (again, using 8200).

I've verified locally that none of the Beats are generating that output.json file, so not sure if this is a breaking change that broke this test suite.

Nevertheless, this PR is a PoC, so please do not merge until further clarification

[elasticmachine]

:broken_heart: Build Failed

• pipeline
• Commit: 425456481a7a433eb0bb0dc81484308cac7963ed

[mdelapenya]

The pipeline fails when checking the next assertion: Beat logs:

[2021-12-02T20:02:36.001Z] TASK [test-beat : Save log to host] ********************************************
[2021-12-02T20:02:36.001Z] fatal: [tester-debian9-64]: FAILED! => {"changed": false, "msg": "file not found: /var/log/packetbeat/packetbeat.json"}
[2021-12-02T20:02:36.001Z] fatal: [tester-debian8-64]: FAILED! => {"changed": false, "msg": "file not found: /var/log/packetbeat/packetbeat.json"}
[2021-12-02T20:02:36.001Z] fatal: [tester-debian10-64]: FAILED! => {"changed": false, "msg": "file not found: /var/log/packetbeat/packetbeat.json"}
[2021-12-02T20:02:36.001Z] fatal: [tester-ubuntu1804-64]: FAILED! => {"changed": false, "msg": "file not found: /var/log/packetbeat/packetbeat.json"}
[2021-12-02T20:02:36.001Z] fatal: [tester-ubuntu2004-64]: FAILED! => {"changed": false, "msg": "file not found: /var/log/packetbeat/packetbeat.json"}

BUT it's important to note that apm-server passes!! so it does generate logs.

-rw------- 1 apm-server apm-server 1050 Dec  3 07:04 apm-server.json
-rw------- 1 apm-server apm-server 1050 Dec  3 07:04 apm-server.json.1
-rw------- 1 apm-server apm-server 1050 Dec  3 07:04 apm-server.json.2
-rw------- 1 apm-server apm-server 1050 Dec  3 07:04 apm-server.json.3
-rw-r--r-- 1 root       root       1702 Dec  3 07:04 metrics.json

The next Beat to be executed is packetbeat, and this one is failing the assertions for the logs. I double checked in my local vagrant instance that the files under /var/log/packetbeat are:

-rw------- 1 root root 7.6K Dec  2 20:20 packetbeat.json-20211202.ndjson
-rw------- 1 root root 1.5K Dec  2 20:20 output.json-20211202.ndjson

Which gives me a hint: is the beat receiving the configuration to set the output/log file correctly?

[mdelapenya]

This is the export of the config for packetbeat:

packetbeat export config -v
http:
  host: localhost
logging:
  files:
    name: packetbeat.json
  json: true
  level: info
  to_files: true
output:
  file:
    enabled: true
    filename: output.json
    path: /var/log/packetbeat
packetbeat:
  interfaces:
    device: any
  protocols:
  - ports:
    - 80
    type: http
path:
  config: /etc/packetbeat
  data: /var/lib/packetbeat
  home: /usr/share/packetbeat
  logs: /var/log/packetbeat

but this is the list of files in the path.logs dir:

root@jessie:/var/log/packetbeat# ls -l /var/log/packetbeat
total 28
-rw------- 1 root root  1496 Dec  2 20:30 output.json-20211202.ndjson
-rw------- 1 root root   892 Dec  2 20:33 packetbeat.json-20211202-1.ndjson
-rw------- 1 root root   565 Dec  2 20:34 packetbeat.json-20211202-2.ndjson
-rw------- 1 root root 15082 Dec  2 20:34 packetbeat.json-20211202.ndjson

@jsoriano do you see anything wrong with the configuration, which seems not be honouring file formats?

[jsoriano]

@mdelapenya there are some breaking changes in the file output in 8.0, files are now always suffixed with the timestamp (see https://github.com/elastic/beats/pull/28927).

e2e-testing autodiscover scenarios were also affected, we addressed it with https://github.com/elastic/e2e-testing/pull/1829, I guess that something similar will be needed here.

@kvch can give more details if needed.