search

cloudfoundry / cli

The official command line client for Cloud Foundry
https://docs.cloudfoundry.org/cf-cli
Apache License 2.0
1.75k stars 928 forks source link

`cf security-groups` is extremely slow on large systems #2117

Open bkrannich opened 5 years ago

bkrannich commented 5 years ago

Issue

Running a cf security-groups on a large Cloud Foundry system is extremely slow. For 13.500 security groups, the runtime is > 3 hours. I would hope that this can be way quicker given that cloud controller already has a security_group table containing the entries in question.

Context

see above

Steps to Reproduce

Run cf security-groups in a large CF system.

Expected result

Runtime in the minutes range at best.

Current result

Runtime > 3 hours.

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/165179334

The labels on this github issue will be updated when the story is started.

gberche-orange commented 5 years ago

@bkrannich I also reported this same problem in the related V3 API proposal for future V3 performance enhancement. Might be a good place to look at currently proposed V3 endpoint and provide feedback

weymanf commented 3 years ago

It appears that CAPI returns rather quickly when calling the cf security-groups. The real slowdown is when cf cli acts as an admin user. It would use the default per-page result which is 50, and the cf security-groups would try to walk through 201 pages to find 10000 security-groups.

@bkrannich Can you create this issue in https://github.com/cloudfoundry/cli?

Gerg commented 3 years ago

I'll transfer this issue to the CLI repo.

cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/176041626

The labels on this github issue will be updated when the story is started.

weymanf commented 3 years ago

I have tested running queries in our mysql database in bosh-lite. It returns in seconds with all 13000 fake records.

I have also tested cf curl /v3/security_groups but it looks like we limit the max page results.

I've ran cf security-groups and it's definitely slow ~45 seconds. It is slow at 10000+ security groups because we use the default per page and walk through 201 pages with 50 results each.

Looks like we can't safely bump up the walk limit per page number because users can configure the max limit https://github.com/cloudfoundry/capi-release/blob/6387524550d336afd40de43152a7b1abbfc25921/jobs/cloud_controller_ng/spec#L898

github-actions[bot] commented 3 weeks ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed.