[COST-4215] send OCP payload to minio

[maskarb]

Summary

Adds 2 new CLI args for OCP report generation:

• --minio-upload: the url to minio
  • S3_ACCESS_KEY, S3_SECRET_KEY, S3_BUCKET_NAME env variables must be set. These are the credentials to access minio and the bucket in minio where the payload will be sent.
• --payload-name: a static name for the payload uploaded to minio. if not provided, the payload name will start with the assembly-id of the payload manifest.
  • payload-name is only used in tandem with minio-upload. If provided without minio-upload, the CLI will exit with an error
  • the payloads generated are monthly. all payload names will contain the month of the payload. the payload name is in the form {payload-name}.{YYYY_MM}.tar.gz.

Testing:

with --payload-name

1. using payload name:

   $ S3_ACCESS_KEY="kokuminioaccess" S3_SECRET_KEY="kokuminiosecret" S3_BUCKET_NAME="ocp-ingress" nise -lll report ocp -s 2023-10-01 --ocp-cluster-id test-001 --minio-upload http://localhost:9000 --daily-reports --payload-name my-payload-name

2. look in minio and see these payloads:

   my-payload-name.2023_10.tar.gz
   my-payload-name.2023_11.tar.gz

without --payload-name

1. cmd:

   $ S3_ACCESS_KEY="kokuminioaccess" S3_SECRET_KEY="kokuminiosecret" S3_BUCKET_NAME="ocp-ingress" nise -lll report ocp -s 2023-10-01 --ocp-cluster-id test-001 --minio-upload http://localhost:9000 --daily-reports

2. without the payload-name, the name uses the random manifest-id:

   8fb1c12b76ff4eafbc7c81cf90775c32.2023_10.tar.gz
   939f025fc5064809a646bde4102e86ee.2023_11.tar.gz

some error pathways:

1. 
   $ S3_ACCESS_KEY="kokuminioaccess" S3_SECRET_KEY="kokuminiosecret" nise -lll report ocp -s 2023-10-01 --ocp-cluster-id test-001 --minio-upload http://localhost:9000 --daily-reports

nise: error: --minio-upload http://localhost:9000 was supplied as an argument but the environment must have S3_ACCESS_KEY and S3_SECRET_KEY and S3_BUCKET_NAME defined when attempting an upload to Minio (or S3).

2.

$ S3_ACCESS_KEY="kokuminioaccess" S3_SECRET_KEY="kokuminiosecret" S3_BUCKET_NAME="ocp-ingress" nise -lll report ocp -s 2023-10-01 --ocp-cluster-id test-001 --daily-reports --payload-name my-payload-name

nise: error: --payload-name is only used with --minio-upload

[codecov[bot]]

Codecov Report

Merging #466 (ae0a9fa) into main (33e2786) will increase coverage by 0.0%. The diff coverage is 100.0%.

@@          Coverage Diff          @@
##            main    #466   +/-   ##
=====================================
  Coverage   93.6%   93.6%           
=====================================
  Files         60      60           
  Lines       4587    4603   +16     
  Branches     657     660    +3     
=====================================
+ Hits        4292    4307   +15     
  Misses       155     155           
- Partials     140     141    +1     

[djnakabaale]

followed testing instructions and looks good to me:

```log 2023-11-01 17:24:29,785 [INFO] Creating reports... 2023-11-01 17:24:29,872 [INFO] Generating data for ocp_pod_usage for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:34,641 [INFO] Generating data for ocp_storage_usage for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:34,675 [INFO] Generating data for ocp_node_label for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:34,793 [INFO] Generating data for ocp_namespace_label for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:34,794 [INFO] Writing to October-2023-test-001-ocp_pod_usage.csv 2023-11-01 17:24:36,465 [INFO] Writing to October-2023-test-001-ocp_storage_usage.csv 2023-11-01 17:24:36,466 [INFO] Writing to October-2023-test-001-ocp_node_label.csv 2023-11-01 17:24:36,499 [INFO] Writing to October-2023-test-001-ocp_namespace_label.csv 2023-11-01 17:24:37,783 [INFO] File uploaded successfully. 2023-11-01 17:24:37,783 [INFO] File key: 5fce732f35d84cb78d22de02da01a0c2.2023_10.tar.gz 2023-11-01 17:24:37,786 [INFO] Cleaning up local directory 2023-11-01 17:24:37,912 [INFO] Generating data for ocp_pod_usage for {'name': 'November', 'start': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 17, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:38,001 [INFO] Generating data for ocp_storage_usage for {'name': 'November', 'start': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 17, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:38,002 [INFO] Generating data for ocp_node_label for {'name': 'November', 'start': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 17, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:38,004 [INFO] Generating data for ocp_namespace_label for {'name': 'November', 'start': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 17, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:24:38,004 [INFO] Writing to November-2023-test-001-ocp_pod_usage.csv 2023-11-01 17:24:38,036 [INFO] Writing to November-2023-test-001-ocp_storage_usage.csv 2023-11-01 17:24:38,036 [INFO] Writing to November-2023-test-001-ocp_node_label.csv 2023-11-01 17:24:38,037 [INFO] Writing to November-2023-test-001-ocp_namespace_label.csv 2023-11-01 17:24:38,090 [INFO] File uploaded successfully. 2023-11-01 17:24:38,090 [INFO] File key: 35798f39ea9b4a008a999845e58b9950.2023_11.tar.gz 2023-11-01 17:24:38,091 [INFO] Cleaning up local directory ```

One other error to watch out for is: The specified bucket does not exist

```log 2023-11-01 17:30:57,768 [INFO] Creating reports... 2023-11-01 17:30:57,830 [INFO] Generating data for ocp_pod_usage for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:30:59,545 [INFO] Generating data for ocp_storage_usage for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:30:59,554 [INFO] Generating data for ocp_node_label for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:30:59,597 [INFO] Generating data for ocp_namespace_label for {'name': 'October', 'start': datetime.datetime(2023, 10, 1, 0, 0, tzinfo=datetime.timezone.utc), 'end': datetime.datetime(2023, 11, 1, 0, 0, tzinfo=datetime.timezone.utc)} 2023-11-01 17:30:59,597 [INFO] Writing to October-2023-test-001-ocp_pod_usage.csv 2023-11-01 17:31:00,220 [INFO] Writing to October-2023-test-001-ocp_storage_usage.csv 2023-11-01 17:31:00,221 [INFO] Writing to October-2023-test-001-ocp_node_label.csv 2023-11-01 17:31:00,232 [INFO] Writing to October-2023-test-001-ocp_namespace_label.csv 2023-11-01 17:31:00,680 [ERROR] 404 File upload failed. 2023-11-01 17:31:00,680 [INFO] NoSuchBucketThe specified bucket does not exist7d24fbe6fa124166b995890f5c9b9ec8.2023_10.tar.gztest-bucket/test-bucket/7d24fbe6fa124166b995890f5c9b9ec8.2023_10.tar.gz1793900AD69AAACBdd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 2023-11-01 17:31:00,682 [INFO] Cleaning up local directory ```