Scale OSIO analytics platform

krishnapaparaju commented 6 years ago

Need to scale OSIO analytics to process 100 stack analysis requests (SLA: 2 mins) , 500 components analysis (SLA: 1 sec) requests concurrently.

This work is being tracked here and all the code/configuration changes that would be made to achieve the above goal around scalability.

https://docs.google.com/a/redhat.com/spreadsheets/d/13CRoYJPxGrDkhgnTKeyjP80SVdY0Sdfg6bbGlIT6azY/

cc @msrb @samuzzal-choudhury @msrb @GeorgeActon @sivaavkd

Sprint 144

Scale the platform to support 1000 concurrent users:

[x] Identify scalability requirements to support 500 concurrent requests in approximately a minute.
[x] Document the test results Result Spreadsheet link
[ ] Promote changes to OSIO

Stretch:

[ ] Work with SD team to get necessary resources
[ ] Identify scalability requirements to support 1000 concurrent requests in approximately a minute.

Sprint 143 User Story

[x] https://github.com/openshiftio/openshift.io/issues/1763

Sprint 142 Checklist

[x] Figure out Performance Bottlenecks for stack analyses #1647

Checkist:

[x] Identify bottlenecks in the stack analysis #1242
[x] Identify bottlenecks in the component analysis #1243
[x] Component analysis is slow on the production server, ... #1191
[x] Fix bottlenecks in stack analysis #1527, #1763
[x] Perform and publish load test results for stack analysis #1467

Related issues:

[x] Create performance tests for common queries to the graph database: #1528

fridex commented 6 years ago

500 components analysis (SLA: 1 sec) requests concurrently.

How is this defined? 1 sec for one component analysis?

krishnapaparaju commented 6 years ago

@fridex Yes, OSIO user triggering component analysis would expect the response in 1 second and response for stack analysis in not more than 2 mins.

fridex commented 6 years ago

@fridex Yes, OSIO user triggering component analysis would expect the response in 1 second and response for stack analysis in not more than 2 mins.

Is it response from API server in both cases?

krishnapaparaju commented 6 years ago

Yes

fridex commented 6 years ago

Yes

OK, thanks for clarification.

krishnapaparaju commented 6 years ago

@samuzzal-choudhury @sivaavkd @GeorgeActon scaling the platform for significantly higher number of users is on the critical path for opening up the platform for IDE users.

msrb commented 6 years ago

Component analyses are OK-ish (not flawless though): https://github.com/openshiftio/openshift.io/issues/1243#issuecomment-342399807

There will be some work around stack analyses: https://github.com/openshiftio/openshift.io/issues/1242#issuecomment-342418630

openshiftio / openshift.io