Open tanganellilore opened 4 months ago
Thanks for opening an issue @tanganellilore. The limit applied to search responses is intentional - such large datasets don't scale well for a system with an embedded database and embedded search engine. Without that in place, it's a recipe for OOM, which can cause the application to fail unexpectedly and result in database/index corruption and/or data loss.
What is your use case for queries that have such large result sets?
We have the same problem. The repository contains many artifacts and a search is needed for them all. Users have the right to decide how to limit the output. Previously, the index.max_result_window parameter in the elasticsearch configuration file worked. And it was a revelation to us that it was broken. It’s unclear why I hardcoded the parameter directly in the code. Set the parameter at the configuration file level so that it can be changed.
Thanks for opening an issue @tanganellilore. The limit applied to search responses is intentional - such large datasets don't scale well for a system with an embedded database and embedded search engine. Without that in place, it's a recipe for OOM, which can cause the application to fail unexpectedly and result in database/index corruption and/or data loss.
What is your use case for queries that have such large result sets?
Hi @nblair , I notice only now the answer, sorry for delay. I need simply export all "metadata" of all assets in all repos and subfolder, like checksum, last download etc.... and save in a external DB, to track changes and delation for internal process, (without usage of audit webhook).
In my case i have a big repo with a lot of subfolder, all of them with like 30/100 assets. So this repo is bigger than 10k elements. Via api, we can't get simply a list of subpath in the repo, to iterate over it (to reduce number of assets), so this is why i receive this error on call.
I understood that this limit is to avoid OOM, but with api we can't do it.
For my use case i used a groovy script that can be called and return this type of object per repository, but i notice also this that we have warning.
Hi team,
i notice a Bug on search query, probably is connected to this issue and this old not migrated issue https://issues.sonatype.org/browse/NEXUS-16917.
I notice that if I perform a search query on a large repository, with more than 10000 elements but with pagination, i recevie a error from api like this:
Log above is an example, but is very long and repated.
Some suggestion to solve it?
Thanks
P.S. repo have multiple folder with a lot of docker images, so i need to extract all of them.