TODOs for Open Sourcing

NASA-IMPACT / data-transfer-evaluation

Evaluating two way data transfer of NASA and ESA

Apache License 2.0

4 stars 3 forks source link

TODOs for Open Sourcing #23

Open NISH1001 opened 2 years ago

NISH1001 commented 2 years ago

[x] Better code documentation (in a way we can generate HTML docs through sphinx)
[x] CONTRIBUTING.md and CODE_OF_CONDUCT.md
[ ] architecture diagram for the eval framework
[x] More load testing
[x] Check for any modules with restrictive licenses.
[x] More installation/setup guide
[ ] Review/Feedback loop through IMPACT's dev team (cc: @xhagrg @code-geek )

NISH1001 commented 2 years ago

cc: @ub0005

NISH1001 commented 2 years ago

Re: The minio version RELEASE.2022-05-26T05-48-41Z is the latest minio release that works with our codebase. After this, it seems minio changed the way files are stored/listed in the bucket

We can get the binaries from the minio archives at: https://dl.minio.io/server/minio/release/

The following binaries should work. I can confirm that minio release > may 26, 2022 doesn't work.

Put the binaries into system path (preferred: ~/bin/).

cc: @ub0005

xhagrg commented 2 years ago

@NISH1001 could you share why we are using minio again? couldn;t you use moto/python mocks?

NISH1001 commented 2 years ago

@xhagrg Using minio we can emulate s3 bucket like behaviour -- creates s3 buckets locally -- physically for which the original transfer was intended. So, it's used in a way we can emulate the s3-s3 transfers for GBs of dummy files. Thus, it helps us to emulate the actual transfer realistically. For instance, initially while testing different tools, we had one test s3 bucket at ESA side and one at the NASA side; we used those tools for those bucket. However, for benchmarking purpose, if we assume the network state is almost the same, the benchmarking could be done locally for N different tools. Also, using minio we could ideally create 2 dummy s3 buckets at two different machines (with different network criteria) and use those for benchmarks too)

NISH1001 commented 2 years ago

@ub0005 the nifi and mft links in the readme are broken. (specifically in SETUP.md, I segregated this from README.md).

ub0005 commented 2 years ago

@ub0005 the nifi and mft links in the readme are broken. (specifically in SETUP.md, I segregated this from README.md).

Fixed it @NISH1001