Inconsistent content-type received in the response headers for request headers (Accept: */* or No Accept headers) in cornice+pyramid webserver running on python 3.8 alpine linux version

[ookla-RITESHVARPE]

• I have a cornice-based pyramid webserver running in Python 3.8(Alpine Linux) and a client calling one of its APIs using python requests without any request headers.
• Cornice+Pyramid webserver has msgpack(custom renderer added by me) and simplejson(default renderer of cornice) as renderers.
• Currently, I am getting inconsistent content-type(sometimes 'application/x-msgpack' and sometimes 'application/json') in the response headers.
• As per cornice documentation, the default renderer is simplejson thus I am expecting it should always convert the response object into json using json renderer factory of pyramid/cornice and send the response to the client with content-type as 'application/json'.
• Earlier, I used to have the same webserver running on python 2.7 version, where it was always sending response in json format with consistent content-type as 'application/json'.
• When I am passing request headers as {'Accept': 'application/json'} or {'Accept': 'application/x-msgpack'} then I am getting response as expected with the response been rendered with the appropriate renderer.
• I want to know the exact reason, why the default renderer being considered by the webserver is changing between msgpack and simplejson.
• Below are the files/configuration been used:

A. requirements.txt

simplejson==3.10.0
pyramid==1.5.7
chardet==3.0.4
colander==1.0b1
cornice==1.0
msgpack==1.0.4
lz4==4.0.2
lz4tools==1.3.1.2
PyYAML==3.11
SQLAlchemy==1.4.0
psycopg2-binary==2.9.5
enum34==1.1.6
python-dateutil==2.2
pytz==2014.10
requests==2.7.0
waitress==0.8.9
Paste==3.5.2
jsonschema==2.3.0

B. webserver/init.py

# -*- coding: utf-8 -*-

"""Main entry point
"""

import os

from pyramid.config import Configurator
from pyramid.events import NewResponse
from sdk.pyramid.viewutils import MsgpackRendererFactory, notfound, lz4_compress_response

class MsgpackRendererFactory(object):

    def __init__(self, info):
        pass

    def __call__(self, value, system):
        request = system.get('request')
        if request is not None:
            response = request.response
            response.content_type = 'application/x-msgpack'
        return msgpack.dumps(value)

def main(global_config, **settings):
    route_prefix = settings.get('reports.route_prefix')
    config = Configurator(settings=settings, route_prefix=route_prefix)

    config.add_settings(handle_exceptions=False)
    config.add_notfound_view(notfound, append_slash=True)

    config.add_renderer(name='msgpack', factory=MsgpackRendererFactory)

    config.include('cornice')

    return config.make_wsgi_app()

C. webserver/views.py

@resource(path='path_to_the_api')
class ResourceView():

    @view(accept='application/x-msgpack', renderer='msgpack')
    @view(accept='application/json', renderer='simplejson')
    def get(self):
            # Data fetched from DB stored in 'resource' variable
        resource = .......... # Calling DB to get data
            return {'resources': resource}

D. Client code

url = 'https://X.X.X.X/path_to_webserver_api'
response = requests.get(url)

body = response.json() # works only if the content-type received is 'application/json' else code                 
                       # breaks at this point 

• I tried upgrading the packages mentioned in the requirements.txt but was not able to resolve the issue I am facing.
• Also, debugged the package code locally but was not able to find the root cause.
• Early reply to the issue will be appreciable.

[mmerickel]

Our current response here, which I understand won't be satisfying to you, is "if a client didn't specify a clear accept header, then the client doesn't care what they get back" so anything is fine. We've explored some options in the past to specify a server-side preference, but nothing like that has shipped at this time.

If I really wanted control over this I might suggest defining your own renderer that can make a more intelligent decision here.

from pyramid.config import Configurator
from pyramid.renderers import get_renderer
from pyramid.view import view_config
from waitress import serve

def fancyapi_renderer_factory(info):
    json_renderer = get_renderer('json', package=info.package, registry=info.registry)
    string_renderer = get_renderer('string', package=info.package, registry=info.registry)
    def fancyapi_renderer(value, system):
        request = system['request']
        accept = 'application/json'
        if request.accept:
            offers = request.accept.acceptable_offers(
                ['application/json', 'text/plain'],
            )
            if offers:
                accept = offers[0][0]

        if accept == 'application/json':
            renderer = json_renderer
        else:
            renderer = string_renderer
        return renderer(value, system)
    return fancyapi_renderer

@view_config(renderer='fancyapi')
def cool_view(request):
    return {'success': True}

if __name__ == '__main__':
    with Configurator() as config:
        config.add_renderer('fancyapi', fancyapi_renderer_factory)
        config.scan(__name__)
        app = config.make_wsgi_app()
    serve(app, listen='localhost:8080')

If you have a reason you can't use the same view for both, then you'd have to make a view that can handle request.accept logic inside of it.

[mmerickel]

You know, it was bugging me, I swear I worked on this feature a few years ago and didn't find it.

Anyway I found it - we introduced config.add_accept_view_order which solves this problem.

from pyramid.config import Configurator
from pyramid.view import view_config
from waitress import serve

@view_config(accept='text/plain', renderer='string')
def string_view(request):
    return 'string'

@view_config(accept='application/json', renderer='json')
def json_view(request):
    return {'type': 'json'}

if __name__ == '__main__':
    with Configurator() as config:
        config.add_accept_view_order(
            'text/plain', weighs_less_than='application/json')
        config.scan(__name__)
        app = config.make_wsgi_app()
    serve(app, listen='localhost:8080')

[mmerickel]

https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/viewconfig.html#accept-content-negotiation

[RiteshVarpe]

Thanks @mmerickel for your informative reply. But I wonder, for python2.7, the server was responding with the application/json consistently even with the multiple renderers configured. I want to know the exact reason, why python 2.7 shows consistent behavior while python3.8 shows inconsistent behavior. Please find below requirements.txt files for both python versions.

For python 2.7

simplejson==3.10.0
pyramid==1.5.2
chardet==3.0.4
colander==1.0
cornice==1.0
msgpack-python==0.4.6
lz4==0.7.0
lz4tools==1.3.1.2
PyYAML==3.11
SQLAlchemy==0.9.9
psycopg2==2.5.4
enum34==1.0.4
python-dateutil==2.2
pytz==2014.10
requests==2.2.1
waitress==0.8.9
Paste==1.7.5.1
jsonschema==2.3.0

For python 3.8

simplejson==3.10.0
pyramid==1.5.7
chardet==3.0.4
colander==1.0b1
cornice==1.0
msgpack==1.0.4
lz4==4.0.2
lz4tools==1.3.1.2
PyYAML==3.11
SQLAlchemy==1.4.0
psycopg2-binary==2.9.5
enum34==1.1.6
python-dateutil==2.2
pytz==2014.10
requests==2.7.0
waitress==0.8.9
Paste==3.5.2
jsonschema==2.3.0

Also, I have raised the same issue in cornice repo(https://github.com/Cornices/cornice/issues/578). Please have a look at that too. It will be great, if I will be able to recognize the exact root cause between those servers of different python versions. Thanks in advance!

[mmerickel]

Like I said the order has never been guaranteed in pyramid and a huge amount of stuff has changed since 2.7 to 3.8. Plus you are using an old unsupported version of Pyramid (and likely webob). Use the new features that can guarantee stability.

If you want to know why the older versions work you can try to ask the community instead of the issue tracker please. My intuition is that it has to do with PYTHONHASHSEED.