search

Haufe-Lexware / wicked.haufe.io

An API Management system based on Mashape Kong
http://wicked.haufe.io
Other
121 stars 37 forks source link

Portal loosing consumers #287

Open maksimlikharev opened 3 years ago

maksimlikharev commented 3 years ago

when # of consumers exceeds 1000, it seems that Wicked cannot detect outliers that would be retrieved with consecutive pagination call, and assumes that those consumers are new and will try to create them. This will result in DB primary key constraint violation on kong, resulting in failure of kong-adapter.

all sync ops, especially for potentially large # of entities, like consumers, should implement proper pagination, Kong does not allow > 1000 rows to be retrieved with a single call.

shreyas-segu commented 3 years ago

The same is also applicable to routes, plugins and apis. It is highly likely that the plugins and routes can also cause the same issue as number of plugins grow related to the number of subscriptions / plans

DonMartin76 commented 3 years ago

Yup, this has already been fixed in the kong-1xx branch, if you check the code there. All object retrievals from Kong are done via the paging mechanisms there. Would it be possible for you to verify the functionality there?

DonMartin76 commented 3 years ago

See here: https://github.com/Haufe-Lexware/wicked.haufe.io/blob/kong_1xx/src/kong-adapter/src/kong/utils.ts#L304

FWIW, it was not documented in previous versions that getting more than 1000 consumers/APIs/services/... would not work. As of Kong 1.0+ the API would complain with a size parameter larger than 1000.

maksimlikharev commented 3 years ago

we are not at 1x yet in production, and yes, > 1000 does not work. we can probably move 1x into our fork at least for the consumers