Callback endpoints always give a 500 server error

rbobrowicz commented 6 years ago

ISSUE TYPE

Bug Report

COMPONENT NAME

API

SUMMARY

Job callbacks result in 500 error, no jobs are triggered. Exception in logs is Fields set(['launch_type']) are not allowed as overrides.

ENVIRONMENT

AWX version: 1.0.2.358
AWX install method: docker on linux
Ansible version: 2.4.2.0

STEPS TO REPRODUCE

Create a new job template
Check 'Allow Provisioning Callbacks'
Add Host Config Key
POST to job callback URI with Host Config Key

EXPECTED RESULTS

Job should be ran

ACTUAL RESULTS

500 Server Error {"detail": "A server error has occurred."}

ADDITIONAL INFORMATION

2018-01-12 20:08:02,444 ERROR    django.request Internal Server Error: /api/v2/job_templates/9/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)
  File "/usr/lib/python2.7/site-packages/awx/main/models/jobs.py", line 309, in create_job
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.
172.20.5.57 - - [12/Jan/2018:20:08:02 +0000] "POST /api/v2/job_templates/9/callback/ HTTP/1.1" 500 41 "-" "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393.1944" "-"
[pid: 30|app: 0|req: 140/965] 172.20.5.57 () {36 vars in 602 bytes} [Fri Jan 12 20:08:02 2018] POST /api/v2/job_templates/9/callback/ => generated 41 bytes in 61 msecs (HTTP/1.1 500) 5 headers in 177 bytes (1 switches on core 0)

secretbatcave commented 6 years ago

I'm also bumping into this as well. I'm using the playbook geerlingguy.awx to got through the versions to see if I can get to one that works. So far I have tried the devel branch(ccb40c8) and tags 1.02, 1.0.1 and 1.0.0 (although that required hacking the install scripts. )

ryanpetrello commented 6 years ago

This should be fixed in devel via https://github.com/ansible/awx/pull/998

secretbatcave commented 6 years ago

This appears to be still happening for me: I've spun up a fresh instance running from devel (fc5c5400cd1f91891ac2a125d24b005abf23e465) and I'm still seeing what appears to be the same error:

2018-01-18 15:43:50,451 ERROR    django.request Internal Server Error: /api/v2/job_templates/8/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3288, in post
    inventory_update = inventory_source.create_inventory_update(launch_type='callback')
  File "/usr/lib/python2.7/site-packages/awx/main/models/inventory.py", line 1443, in create_inventory_update
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/inventory.py", line 1456, in create_unified_job
    return super(InventorySource, self).create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.

STEPS TO REPRODUCE

Create new awx instance with ansible playbook from devel
Configure so template can run
Check 'Allow Provisioning Callbacks'
Add Host Config Key
POST to job callback URI with Host Config Key

ryanpetrello commented 6 years ago

@secretbatcave yep, looks like a really similar issue (your traceback is slightly different) - I'm looking into it.

ryanpetrello commented 6 years ago

@secretbatcave mind giving it another shot now (devel)?

secretbatcave commented 6 years ago

Hmm, Looks like its still erroring out

using devel(073feb74cb824029a1fe0327c13204f2afebbaf1) for one callback request I get the following two exceptions:

2018-01-18 20:48:08,602 ERROR    django.request Internal Server Error: /api/v2/job_templates/8/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)
  File "/usr/lib/python2.7/site-packages/awx/main/models/jobs.py", line 309, in create_job
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.
2018-01-18 20:48:08,602 ERROR    django.request Internal Server Error: /api/v2/job_templates/8/callback/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3197, in dispatch
    return super(JobTemplateCallback, self).dispatch(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/generics.py", line 253, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 489, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 449, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/rest_framework/views.py", line 486, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)
  File "/usr/lib/python2.7/site-packages/awx/main/models/jobs.py", line 309, in create_job
    return self.create_unified_job(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/main/models/unified_jobs.py", line 358, in create_unified_job
    raise Exception('Fields {} are not allowed as overrides.'.format(unallowed_fields))
Exception: Fields set(['launch_type']) are not allowed as overrides.
[pid: 27|app: 0|req: 173/334] 34.243.229.143 () {36 vars in 526 bytes} [Thu Jan 18 20:48:08 2018] POST /api/v2/job_templates/8/callback/ => generated 41 bytes in 61 msecs (HTTP/1.1 500) 5 headers in 177 bytes (1 switches on core 0)
34.243.229.143 - - [18/Jan/2018:20:48:08 +0000] "POST /api/v2/job_templates/8/callback/ HTTP/1.1" 500 41 "-" "curl/7.47.0" "-"

ryanpetrello commented 6 years ago

@secretbatcave are you certain you have the latest code from devel? The line number references in your traceback don't match what I see in HEAD of devel:

  File "/usr/lib/python2.7/site-packages/awx/api/views.py", line 3327, in post
    job = job_template.create_job(**kv)

https://github.com/ansible/awx/blob/43f19cc94b9e6e90111ff153422da2b2a104f379/awx/api/views.py#L3327

ryanpetrello commented 6 years ago

It looks to me like you're running an older version of this code from a few commits back:

https://github.com/ansible/awx/blob/1e8c89f5367c5c4060170a07164600b498fa4159/awx/api/views.py#L3327

secretbatcave commented 6 years ago

Ok, Let me redeploy.

secretbatcave commented 6 years ago

Aha! you are correct, it is the old commit. However I am confused as I'm installing via the built in ansible playbook, using geerlingguy.awx role (it checks out devel, and is on a fresh instance each time). I'm going off line, but I'll have a look tomorrow and try and have a wild stab as to where the problem might be

secretbatcave commented 6 years ago

I have run from devel, and I can confirm that callbacks work, which is excellent.

Thanks for the fix!

dedestell commented 6 years ago

Please someone help me I have Nextpost script 4, and I can not put instagram accounts semre of the error asks to come back later. In Logs error you are saying this but I do not know what it means .............. POST / accounts / new? callback = jQuery31107295573735186285_1528434746838 HTTP / 1.1 "500 5 ............. can anyone help me, I use VPS Centos 7 with Nginx

ansible / awx