Large Data Volume Generation for Performance Testing

Red-HAP commented 4 years ago

User Story

As a developer, I want to generate a large volume of reporting data so that I can test the performance of queries and database objects under simulated production data volume.

Impacts

API

Assumptions

Nise can be utilized to generate this data
The use of nise can be scripted in order to generate a large volume of data (> 1000000 rows for some large volume tables similar to production)
This files to generate this new data can be purged

Acceptance Criteria

[ ] Large volume of reporting data can be generated (>=1000000 rows in high volume tables somilar to production)
[ ] Generation of this data is scripted
[ ] Any temporary input files can be removed

nbon12 commented 4 years ago

Sprint Planning Meeting Minutes: Sergio: "Do we want to test also the uploader? I heard they reached the limit of whatever the Insights Client can upload. Perhaps part of this we can test that so that we can split" Andrew: "That's a separate issue. That's almost on the Korekuta/New Operator side of things. Do we know what there file limit is? Ingress." Chris: "We do. I think it's like 500 Mb but I could be wrong." HAP: "This is crucial to doing performance testing locally."

lcouzens commented 4 years ago

So this doesn't work for me. It's totally possible I did something wrong, see my error log below. That said this script generates data for local usage only right? I assume I can just run nise --ocp --ocp-cluster-name hccm --static-report-file ../test-iqe/my_test.yml then upload the data to CI/QA for example?

[lcouzens@localhost koku]$ make large-ocp-provider-testing nise_config_dir=~/Workspace/test-iqe/my_test.yml
make create-large-ocp-provider-testing-files nise_config_dir="/home/lcouzens/Workspace/test-iqe/my_test.yml"
make[1]: Entering directory '/home/lcouzens/Workspace/koku'
make purge-large-testing-ocp-files
make[2]: Entering directory '/home/lcouzens/Workspace/koku'
rm -rf /home/lcouzens/Workspace/koku/testing/pvc_dir/insights_local/large_ocp_1
make[2]: Leaving directory '/home/lcouzens/Workspace/koku'
make[1]: Leaving directory '/home/lcouzens/Workspace/koku'
make import-large-ocp-provider-testing-costmodel
make[1]: Entering directory '/home/lcouzens/Workspace/koku'
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
curl: (3) URL using bad/illegal format or missing URL
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
curl --header 'Content-Type: application/json' \
     --request POST \
     --data '{"name": "Cost Management OpenShift Cost Model", "description": "A cost model of on-premises OpenShift clusters.", "source_type": "OCP", "provider_uuids": , "rates": [{"metric": {"name": "cpu_core_usage_per_hour"}, "tiered_rates": [{"unit": "USD", "value": 0.007, "usage_start": null, "usage_end": null}]}, {"metric": {"name": "node_cost_per_month"}, "tiered_rates": [{"unit": "USD", "value": 0.2, "usage_start": null, "usage_end": null}]}, {"metric": {"name": "cpu_core_request_per_hour"}, "tiered_rates": [{"unit": "USD", "value": 0.2, "usage_start": null, "usage_end": null}]}, {"metric": {"name": "memory_gb_usage_per_hour"}, "tiered_rates": [{"unit": "USD", "value": 0.009, "usage_start": null, "usage_end": null}]}, {"metric": {"name": "memory_gb_request_per_hour"}, "tiered_rates": [{"unit": "USD", "value": 0.05, "usage_start": null, "usage_end": null}]}, {"metric": {"name": "storage_gb_usage_per_month"}, "tiered_rates": [{"unit": "USD", "value": 0.01, "usage_start": null, "usage_end": null}]}, {"metric": {"name": "storage_gb_request_per_month"}, "tiered_rates": [{"unit": "USD", "value": 0.01, "usage_start": null, "usage_end": null}]}]}' \
     http://:/api/cost-management/v1/costmodels/
curl: (3) URL using bad/illegal format or missing URL
make[1]: *** [Makefile:567: import-large-ocp-provider-testing-costmodel] Error 3
make[1]: Leaving directory '/home/lcouzens/Workspace/koku'
make: *** [Makefile:582: large-ocp-provider-testing] Error 2

lcouzens commented 4 years ago

So I tried just running nise --ocp --ocp-cluster-name hccm --static-report-file ../test-iqe/my_test.yml It took a long while and the process just gets killed :D

Red-HAP commented 4 years ago

Yeah, you can give nise a config that will eat all resources and get the process killed. If you tune everything in the generator config down real low (use 1 or 2 for each integer option) it should generate a config that nise can process without getting killed.

I found all of this out the hard way.

On Wed, Feb 12, 2020 at 3:44 AM lcouzens notifications@github.com wrote:

So I tried just running nise --ocp --ocp-cluster-name hccm --static-report-file ../test-iqe/my_test.yml It took a long while and the process just gets killed :D

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/project-koku/nise/issues/187?email_source=notifications&email_token=AOHDVJHA7AELJR3PYQDDDQ3RCOZHRA5CNFSM4KKMLQJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELP5PVI#issuecomment-585095125, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOHDVJAAXDHWSU2PR6L5Y73RCOZHRANCNFSM4KKMLQJQ .

lcouzens commented 4 years ago

Ok good to know. I will play around with it and see what I can come up with while perf testing. Thanks for doing this!

project-koku / nise