Provide s3 storage as optional storage system instead of file storage

[de-jcup]

Currenlty we provide only file system based upload directories. There shall be an option to store data instead to an s3 bucket (e.g. source uploads).

This must be configurable, default shall be filesystem as before.

The easiest way would be to provide S3 storage system as pseudo file system so no changes would be necessary. Other way would be to provide a native approach and support S3 directly by SecHub.

This issue will gather information and has as result either implementation in SecHub or just provide a documentation how to setup S3 storage as a "pseudo filesystem" as a replacement for NFS.

[de-jcup]

Potential java library to use for native s3 object storage:

AWS

https://www.baeldung.com/aws-s3-java

Minio

https://github.com/minio/minio-java https://docs.min.io/docs/minio-client-complete-guide

Could also use minio inside a test container:

https://stackoverflow.com/questions/55402610/configuring-minio-server-for-use-with-testcontainers

[de-jcup]

S3 as shared storage inside kubernetes:

https://icicimov.github.io/blog/virtualization/Kubernetes-shared-storage-with-S3-backend/ uses https://github.com/s3fs-fuse/s3fs-fuse

alpine linux has s3fs-fuse inside official packages (but testing only) https://pkgs.alpinelinux.org/packages?name=s3fs*&branch=edge&repo=testing

Testing package dependencies for apk can be added as described at https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management

[de-jcup]

S3 PVC

https://medium.com/asl19-developers/create-readwritemany-persistentvolumeclaims-on-your-kubernetes-cluster-3a8db51f98e3

[de-jcup]

Goofys

Maybe a good alternative to s3s-fuse, written in Go. see https://github.com/kahing/goofys

not available in alpine linux packages ,but ...

https://github.com/gliderlabs/docker-alpine/issues/268 shows usage in alpine docker image. (but there is a git checkout on docker image build time - so not production ready. But we could build go client in a dedicated version/commitId and use this inside docker creation)

[de-jcup]

Architecture Decision

Should we provide S3 native or by "pseudo filesystem" ?

Result

A native approach is better than providing s3 by a "pseudo filesystem".

Reasons

• S3 is mature, standarized technology which makes sense to support inside SecHub
• We can integrate storage tests inside our integration test suite (e.g. by using https://github.com/adobe/S3Mock)
• Errors can be handled inside SecHub server code
• More information available on problems, so resilience should be easier to implement
• Scalable out of the box (in Kubernetes every POD instance has got its own connection to S3)
• Less complexity in deployment operations
• Not another (external) component that can crash and block SecHub working (we have already to handle potential problems with security product installations...)

If somebody wants to use the "pseudo filesystem" this will still work.

[de-jcup]

One S3 storage test did fail because of a minio problem already issued at https://github.com/minio/minio-java/issues/710 . This seems to be same, but the issue is closed.

There is also the possibliity to use amazon s3 sdk client instead of minio. We will check what memory footprint will be in sechub server bootJar when switching to aws s3 variant and if test do no longer fail with amzon s3 sdk client.