search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] Salt API 3004.2 returning empty response when calling GET minions endpoint /minions/<mid> #64596

Open djmmatracki opened 1 year ago

djmmatracki commented 1 year ago

Description After some amount of calls to the SALT API checking if a minion exists I get empty responses from minions that do exist. When I call the SALT api I get empty response (16 bytes) and status code 200, but when I call salt 'minion1' test.ping from the command line, I get the proper response True. The amount of calls is 60 in parallel every 5 seconds.

[27/Jun/2023:03:54:45] "GET /minions/minion1 HTTP/1.1" 200 16 "" "Go-http-client/1.1"

Setup

Custom options that might be relevant:

worker_threads: 12 sock_pool_size: 15

System setup:

Steps to Reproduce the behavior We are calling the salt api multiple times in parallel. 60 calls in parallel every 5 seconds.

Expected behavior If the salt-api is overloaded why is it not returning 429 or 500 status code, it's returning 200 instead.

welcome[bot] commented 1 year ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

s0undt3ch commented 1 year ago

@djmmatracki 3004.x is out of support.

Can you verify that the same issue exists in 3006.1?

djmmatracki commented 1 year ago

@s0undt3ch Yes, we upgraded the version to 3006.1 and we observe the same behaviour.

s0undt3ch commented 1 year ago

So, the master might be getting overloaded with that many requests. If this happens, it might have a hard time keeping up with the minion returns and assume that the minion did not return, hence the 200 response.

The minion not returning in the expected timeout is not an error in salt.

This is just a wild guess of what could be happening.

How many minions are managed by the master where salt-api is running?

djmmatracki commented 1 year ago

We have an infrastructure with syndics and targeted minions is < 200. We are making tones of requests about job status to the salt-api. The problem is that instead of receiving 4xx or 5xx status codes we are getting a valid empty response with status code 200.

djmmatracki commented 1 year ago

We were also assuming that it is because of the load on the salt-master, but the fact that we receive the 200 status code is preventing us from handling the error on the api client side.