Split large reports into multiple files

[adberglund]

User Story

As a user I want my OpenShift reports to upload successfully so that I can view data in cost management.

Impacts

Korekuta Operator ?

Assumptions

• The Platform Ingress service has a limit of 100MB per file upload
• We should split files so that they do not grow larger than this
• This actually needs to be done at the tar level - might not take care of the header part

Acceptance Criteria

• [ ] OpenShift reports are automatically split into multiple files to meet the Ingress size limit
• [ ] As these are CSVs being generated we want to make sure the header exists in each file
• [ ] The tar.gz file size needs to be below 100MB

[nbon12]

Sprint planning meeting notes:

Andrew: "This impacts korekuta, potentially the operator,The platform ingress service is configurable, with local testing the default is 10MB but in deployed world, there's 100MB limit size. So we just need to split our reports into smaller files so we don't hit that limit. Maybe there's an Ansible command for that."

Brett: "There's standard unix command line tools for splitting the files, it's a matter of calling the right low level commands."

Andrew: "Since they're CSVs you want to make sure the header gets in all of them."

Luke: "When they upload they're put into a TAR, so the TAR needs to be less than 100MB not the individual CSVs."

Brett: TAR has builtin flags to automatically do the size splitting.

Chris: that might not take care of the header part. One could specify a certain number of rows that's of a size smaller than 100MB and split the files on that many rows and do individual TARs for each.

Andrew: "The next issue in line is doing this [splitting] in Nise."

[nbon12]

Sprint Planning Conversation: Brett: "Merged and needs review"