search

project-koku / masu

This is a READ ONLY repo. See https://github.com/project-koku/koku for current masu implementation
GNU Affero General Public License v3.0
5 stars 6 forks source link

Create shared volume in template to be utilized for processing downloaded reports #63

Closed chambridge closed 6 years ago

chambridge commented 6 years ago

Update the Openshift template to create a shared volume to be utilized within Pod between workers for processing downloaded reports.

The download and processing tasks from https://github.com/project-koku/masu/issues/61 could occur on different workers on different systems within Openshift. In order to allow processing of download files a shared volume is needed.

Associated with: https://github.com/project-koku/masu/issues/6

adberglund commented 6 years ago

A PersistentVolumeClaim was added to the masu-template.yaml file. This provides persistent, shared storage for the masu app. If we spin up multiple celery workers pods, each should have access to the same mounted volume -- currently at /var/tmp/masu/

My manual testing for this involved

1) I spun up masu, the postgres database, and rabbitmq in OpenShift. 2) I added a customer to the database from koku, with AWS credentials pointing to our AWS account and test S3 bucket cost-usage-bucket. 3) I ran the masu test download api endpoint using http://masu-myproject.127.0.0.1.nip.io/api/v1/download/ which triggered celery tasks to download report files from the S3 bucket. 4) I checked /var/tmp/masu in the celery worker container to make sure that the files existed at that location. 5) I redeployed the masu celery worker, which removed the first pod and container for the worker and created a new one. 6) I checked /var/tmp/masu in the new celery worker container and verified that the report files were still there, on a fresh pod/container.

LaVLaS commented 6 years ago

VERIFIED. A redeployed worker mounted the persistent volume and was able to read the existing reports that were processed earlier