search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
14.12k stars 3.44k forks source link

AWX deleting old hosts from DB throws 502 #10080

Open endriu0 opened 3 years ago

endriu0 commented 3 years ago
ISSUE TYPE
SUMMARY
ENVIRONMENT
STEPS TO REPRODUCE
  1. Use AWS ASG and scale up / scale down
  2. Use AWX with ec2_inventory pulling directly from AWS.
  3. Inventory update set to delete and then update
  4. Something breaks with the inventory which causes it to have issues deleting hosts.
  5. On manual deletions not able to delete few hosts - getting 502
EXPECTED RESULTS

AWX to delete old hosts / pull new ones correctly.

ACTUAL RESULTS

Inventory update hangs / has to be either killed manually or timeout.

Manual removal of specific hosts causes 502.

ADDITIONAL INFORMATION

AWX web container logs : 

2021/05/03 12:02:12 [error] 25#0: *8648 upstream prematurely closed connection while reading response header from upstream, client: {service_endpoint_ip} , server: _, request: "DELETE /api/v2/hosts/3068/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:8050", host: "awx....", referrer: "https://awx..../api/v2/hosts/3068/"

{service_endpoint_ip} - - [03/May/2021:12:02:12 +0000] "DELETE /api/v2/hosts/3068/ HTTP/1.1" 502 568 "https://awx..../api/v2/hosts/3068/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.116 Safari/537.36" "{client_ip}"

DAMN ! worker 1 (pid: 1172) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 1176)
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x2098540 pid: 1176 (default app)

Vivaldi network console log :

DevTools failed to load SourceMap: Could not load content for https://awx..../static/rest_framework/css/bootstrap.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE
jquery-3.5.1.min.js:2 DELETE https://awx...../api/v2/hosts/3068/ 502
send @ jquery-3.5.1.min.js:2
ajax @ jquery-3.5.1.min.js:2
doAjaxSubmit @ ajax-form.js:75
dispatch @ jquery-3.5.1.min.js:2
v.handle @ jquery-3.5.1.min.js:2

Vivaldi also seems to be pointing to static/rest_framework/js/jquery-3.5.1.min.js as source of error and specifically this line throwing 502:

r.send(i.hasContent && i.data || null)

Also to add - this all started with the issue #9555. I was able to delete old hosts under AWX 17.1.0 but it took a bigger part of a day / under 19.1.0 this seems to just throw 502 all the time now on the broken hosts.

endriu0 commented 3 years ago

Another small update. The only workaround which worked was deleting the whole inventory completely and adding it anew. There was no way to delete the reminder of the hosts otherwise.

gen2fish commented 3 years ago

I'm hitting this issue as well on 19.4 and 19.5

crzysan commented 1 year ago

Hi,

Me and my team am getting this issue on awx version: 22.1.0

When we try to delete a project we get the following error: DELETE api/v2/projects/484/ 502

We tried to delete it on the UI and also tried to delete it using the api.

Is there a workaround for this issue? Can someone help?

Kind regards

Chris