Add s3 functionality in Bytes

Changes

Made it so that if the user adds S3_BUCKET_PREFIX or S3_BUCKET to their environment variables when building OpenKAT, that OpenKAT will use S3-buckets for saving the data from boefje's raw files instead of saving them on the local disk.

Issue link

Closes #3191

Demo

Example files made inside s3

Example of one of the files

QA notes

To test this feature you will need access to an existing s3 server. And inside of you .env file you will need to provide the following variables which are required for boto3:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_ENDPOINT_URL

Outside of these you also have to specify either of the following variables to let OpenKAT know you want to use s3 for the storage of bytes:

S3_BUCKET_PREFIX
S3_BUCKET

Make sure that these are compliant with the bucket naming rules of AWS.

These are the variables I used for testing:

S3_BUCKET_PREFIX=souf-prefix
S3_BUCKET=souf-bucket

# Required env variables for `boto3`
AWS_ACCESS_KEY_ID=GKbf4e89d08d6c5250e7f1de60
AWS_SECRET_ACCESS_KEY=406fef311a69f02fa1c3b58d73a6299651c7890dc0ef5448104f1762e29887b7
AWS_ENDPOINT_URL=http://garage:3900

I made use of the s3 service garage How to setup garage How to work with garage in a container

I built a container for garage with the following docker compose file on the same host machine as bytes

services:
  garage:
    image: dxflrs/garage:v1.0.0
    restart: unless-stopped
    ports:
      - "127.0.0.1:3900:3900"
    volumes:
      - /etc/garage.toml:/etc/garage.toml
      - /var/lib/garage/meta:/var/lib/garage/meta
      - /var/lib/garage/data:/var/lib/garage/data
    networks:
      - network1

networks:
  network1:
    name: nl-kat-coordination_default
    external: true

Code Checklist

[x] All the commits in this PR are properly PGP-signed and verified.
[x] This PR only contains functionality relevant to the issue.
[ ] I have written unit tests for the changes or fixes I made.
[ ] I have checked the documentation and made changes where necessary.
[ ] I have performed a self-review of my code and refactored it to the best of my abilities.

[ ] Tickets have been created for newly discovered issues.
[ ] For any non-trivial functionality, I have added integration and/or end-to-end tests.
[x] I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
[x] I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

[ ] The code does not violate Model-View-Template and our other architectural principles.
[ ] The code prioritizes readability over performance where appropriate.
[ ] The code does not bypass authentication or security mechanisms.
[ ] The code does not introduce any dependency on a library that has not been properly vetted.
[ ] The code contains docstrings, comments, and documentation where needed.

Checklist for QA:

[ ] I have checked out this branch, and successfully ran a fresh make reset.
[ ] I confirmed that there are no unintended functional regressions in this branch:
- [ ] I have managed to pass the onboarding flow
- [ ] Objects and Findings are created properly
- [ ] Tasks are created and completed properly
[ ] I confirmed that the PR's advertised feature or hotfix works as intended.
[ ] I checked the logs for errors and/or warnings and made issues where necessary

What works:

bullet point + screenshot (if useful) per tested functionality

What doesn't work:

bullet point + screenshot (if useful) per tested functionality

Bug or feature?:

bullet point + screenshot (if useful) if it is unclear whether something is a bug or an intended feature.

Checklist for QA:

[x] I have checked out this branch, and successfully ran a fresh make reset.
[x] I confirmed that there are no unintended functional regressions in this branch:
- [x] I have managed to pass the onboarding flow
- [x] Objects and Findings are created properly
- [x] Tasks are created and completed properly
[x] I confirmed that the PR's advertised feature or hotfix works as intended.
[x] I checked the logs for errors and/or warnings and made issues where necessary

What works:

We did a live demo of the functionality to check if it works. The code stores data to a local (Garage) S3 bucket and we are also able to retrieve the data stored in bytes. It doesn't seem to break anything on main.

What doesn't work:

n/a

Bug or feature?:

Requires a small change on the logging format to prevent INFO messages from flooding the Bytes logs. This is picked up in this PR.

We identified a few improvements and things that could probably be solved nicer. There are picked up in different tickets and will be created by @Souf149. These include documentation and whether or not both the PREXIX and Bucket env vars are both mandatory (by aws) or whether these can be also used separately.

minvws / nl-kat-coordination