search

gchq / stroom

Stroom is a highly scalable data storage, processing and analysis platform.
https://gchq.github.io/stroom-docs/
Apache License 2.0
435 stars 53 forks source link

Shared Storage #2110

Open stroomdev66 opened 3 years ago

stroomdev66 commented 3 years ago

We need some better answers to shared storage other than just NFS especially when some customers only have access to local disk or SAN. Possible solutions might include the use of a shared file system that requires no software change such as ZFS, GlusterFs etc however other options might need software support, e.g. Ceph, S3, HDFS. There needs to be a discussion to decide what the requirement is and if any software changes are required.

at055612 commented 1 year ago

For S3 integration the following may be useful

An OutputStream for buffered writing to S3 with multi-part support https://gist.github.com/jcputney/b5daeb86a1c0696859da2a0c3b466327

Running a local S3 compatible server for dev https://dev.to/arifszn/minio-mock-s3-in-local-development-4ke6 https://min.io/download#/kubernetes