Open NISH1001 opened 2 years ago
cc: @ub0005
Re:
The minio version RELEASE.2022-05-26T05-48-41Z
is the latest minio release that works with our codebase. After this, it seems minio changed the way files are stored/listed in the bucket
We can get the binaries from the minio archives at: https://dl.minio.io/server/minio/release/
The following binaries should work. I can confirm that minio release > may 26, 2022 doesn't work.
Put the binaries into system path (preferred: ~/bin/
).
cc: @ub0005
@NISH1001 could you share why we are using minio again? couldn;t you use moto/python mocks?
@xhagrg Using minio we can emulate s3 bucket like behaviour -- creates s3 buckets locally -- physically for which the original transfer was intended. So, it's used in a way we can emulate the s3-s3 transfers for GBs of dummy files. Thus, it helps us to emulate the actual transfer realistically. For instance, initially while testing different tools, we had one test s3 bucket at ESA side and one at the NASA side; we used those tools for those bucket. However, for benchmarking purpose, if we assume the network state is almost the same, the benchmarking could be done locally for N different tools. Also, using minio we could ideally create 2 dummy s3 buckets at two different machines (with different network criteria) and use those for benchmarks too)
@ub0005 the nifi and mft links in the readme are broken. (specifically in SETUP.md, I segregated this from README.md).
@ub0005 the nifi and mft links in the readme are broken. (specifically in SETUP.md, I segregated this from README.md).
Fixed it @NISH1001