search

adevinta / zoe

The Kafka CLI for humans
https://adevinta.github.io/zoe
MIT License
286 stars 21 forks source link

Schema list retreive info about each schema before list them #28

Closed Hubbitus closed 3 years ago

Hubbitus commented 3 years ago

On a relatively big cluster with thousands of schemes it practically never returns:

$ zoe -vv schemas list
2021-03-22 21:53:21 DEBUG zoe: trying to fetch config url for env 'common' with : EnvVarsConfigUrlProvider
2021-03-22 21:53:21 DEBUG zoe: trying to fetch config url for env 'common' with : LocalConfigDirUrlProvider
2021-03-22 21:53:21 DEBUG zoe: trying to fetch config url for env 'default' with : EnvVarsConfigUrlProvider
2021-03-22 21:53:21 DEBUG zoe: trying to fetch config url for env 'default' with : LocalConfigDirUrlProvider
2021-03-22 21:53:21 INFO zoe: loading config from url : file:/home/pasha/.zoe/config/default.yml
2021-03-22 21:53:22 INFO zoe: requesting schemas...
2021-03-22 21:53:22 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects
2021-03-22 21:53:24 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/promo.dbo.Employee/versions
2021-03-22 21:53:24 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/promo.dbo.Employee/versions/latest
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/datahub.epm_crm.qa.v1.epam_accountnamehistoryBase/versions
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/datahub.epm_crm.qa.v1.epam_accountnamehistoryBase/versions/latest
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-ad-gateway-user-group-sandbox-1-value/versions
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-ad-gateway-user-group-sandbox-1-value/versions/latest
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-sap.test_t1-key/versions
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-sap.test_t1-key/versions/latest
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-crm.qa.v1.dbo.BusinessUnitBase/versions
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-crm.qa.v1.dbo.BusinessUnitBase/versions/latest
2021-03-22 21:53:25 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/test1_requests_concur.data-value/versions
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/test1_requests_concur.data-value/versions/latest
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/hmmCGqRJOSMhRHrMEitOzpsXFtkqJHGQ/versions
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/hmmCGqRJOSMhRHrMEitOzpsXFtkqJHGQ/versions/latest
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/AmRmNewTopic2-key/versions
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/AmRmNewTopic2-key/versions/latest
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/datahub.wpm.na_delivery_structure_online/versions
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/datahub.wpm.na_delivery_structure_online/versions/latest
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/datahub.phoenix.employee/versions
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/datahub.phoenix.employee/versions/latest
2021-03-22 21:53:26 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-e3s.epam-comz-event-entity_qa-value/versions
2021-03-22 21:53:27 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-e3s.epam-comz-event-entity_qa-value/versions/latest
2021-03-22 21:53:27 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-hrms.hrms.external_user/versions
2021-03-22 21:53:27 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/epm-hrms.hrms.external_user/versions/latest
2021-03-22 21:53:27 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/rRmRMMZkcJAakoxHEQyPeOjhnoIRYhBULRvstH/versions
2021-03-22 21:53:27 DEBUG RestService: Sending GET with input null to http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects/rRmRMMZkcJAakoxHEQyPeOjhnoIRYhBULRvstH/versions/latest
$ http http://schema-registry-sbox.epm-eco.projects.epam.com:8081/subjects | jq length
20360

By log it took 0.1 seconds per scheme, so in total it should run more than 30 minutes!!

It will be nice to see at least --filter option by regexp and possibility just return list line-by-liine. Without gather full info about the scheme and buffer it into JSON archive (then tools like grep may be used there)

Hubbitus commented 3 years ago

Real time:

real 5163.34
user 36.80
sys 8.02
wlezzar commented 3 years ago

Makes perfect sense @Hubbitus :+1:

I will make a PR to fix this.

wlezzar commented 3 years ago

@Hubbitus I opened a PR to fix this problem. Do not hesitate to give a feedback if you see something missing or to be improved ;)