Same problem with NewRelic

[konradhalas]

Hi guys,

did you solve your issue with New Relic? I have the same problem and I can't find any solution. Your repository is the only result when I ask Google.

Best!

[adamchainz]

We have the same problem too, trying to upgrade to Python 3.5. It seems to be a problem inside wrapt as opposed to new relic itself. Still investigating...

[btoueg]

I've created this repo to demonstrate the issue to New Relic support team. My initial ticket was created on March 2nd, and here is the answer I got on May the 23rd:

I have been discussing this with our Python agent engineering team. They are still working on tracking down the full cause of the issue, but in the meantime, they would like to at least update you on their progress so far. Perhaps one of these alternate combinations could be used as a workaround while we continue our investigation. We have done testing with different combinations of Python/Django/WSGI servers to get an idea of the scope of this issue.

What fails

This is the combination that you have reported using and we have verified that it does indeed fail.

• Django 1.9.6, Python 3.5.1, uWSGI 2.0.13.1, newrelic 2.60.0.46

What works

These are combinations that we have found to work as expected.

• Alternate WSGI servers
  • Django 1.9.6, Python 3.5.1, newrelic 2.60.0.46, mod-wsgi 4.5.2
  • Django 1.9.6, Python 3.5.1, newrelic 2.60.0.46, gunicorn 19.6.0
• Previous Django version
  • Django 1.8.13, Python 3.5.1, newrelic 2.60.0.46, uWSGI 2.0.13.1
• Python 2.7
  • Django 1.9.6, Python 2.7.11, newrelic 2.60.0.46, mod-wsgi 4.5.2

Our Python agent engineers are still actively working on this, but wanted to make sure to give you a status update.

Thank you and we will be in touch once we know more. If you have any follow up questions, please do not hesitate to ask them.

Regards,

A. G. Technical Support Engineer New Relic

Come to New Relic's Community Forum - ask questions and participate! http://discuss.newrelic.com

We eventually switched to gunicorn because downgrading Python or Django was not an option for us.

For further info, here is the last answer I got on May 31st:

I just chatted with our engineers regarding the SystemError you're running into. We may have a potential workaround that will allow you to use the agent and not have to update to Gunicorn.

We suspect that the source of this problems lies in the interaction between uWSGI and a c-extension the Python agent uses. We've noticed that we no longer see the issue when installing the Python agent without c-extensions. The following command will allow you to install the Python agent without the c-extensions:

NEW_RELIC_EXTENSIONS=false pip install newrelic

Please keep in mind, if you choose to do this, you need to be aware of the following caveats:

• Although the Python agent will run, it will do so without some non-core features (i.e. The Capacity Analysis Report).
• Part of the Python agent is written in C. Some of the agent's code that is written in C is meant to optimize the agent. It's possible that installing the Python agent without the C extensions may lead to a performance impact.

If you decide to move forward with running the agent without c-extensions, we highly recommend that you do some performance testing to get a better idea about the performance impact.

I'll go ahead and close out this support ticket but feel free to reply to us if you have any questions.

Regards,

H. J. New Relic, Technical Support Engineer

Come to New Relic's Community Forum - ask questions and participate! http://discuss.newrelic.com

[adamchainz]

Thanks very much. They didn't clarify that it's wrapt that's problematic but looking at the code I suspect it is since _NRBoundFunctionWrapper inherits most of its code from wrapt.BoundFunctionWrapper. I'm going to look for any relevant uWSGI issues, but will probably work around with NEW_RELIC_EXTENSIONS too, plus I'll ping their support.

[adamchainz]

N.B. we did go with NEW_RELIC_EXTENSIONS=false and have successfully deployed python 3 thanks to it 🎉 . The performance impact doesn't seem to be that large.

[konradhalas]

Thanks guys for your reply, I also go with NEW_RELIC_EXTENSIONS=false and it works like a charm :)

[andrewgoetz]

Hello all,

I would like to follow up and let everyone know that our Python agent engineers have engaged with the uWSGI team and have received verification that this is an bug with uWSGI and not New Relic. However, the "fix" will be to ensure that the wsgi-env-behavior is set to holy as the default, because currently it is set to cheat, which is the cause of the issue.

Here is the GitHub Issue filed with uWSGI - https://github.com/unbit/uwsgi/issues/1408

Our official workaround recommendation, until uWSGI releases version 2.1 (which will set holy as the default behavior), will be to manually set the wsgi-env-behavior to holy. We have updated our documentation to include this option under our uWSGI Mandatory Options section. This means that installing the agent without the C extensions will not be needed.

Thank you all again for your reports and help with the investigation.

[adamchainz]

@andrewgoetz thanks for following up and posting here. In the future you could create a github repo for issue reporting on the python agents so you can message us developers where we are 😉

[giorgiosironi]

NEW_RELIC_EXTENSIONS=false pip install newrelic didn't work for me, what worked was:

NEW_RELIC_EXTENSIONS=false pip install --no-binary :all: newrelic==2.78.0.57

because otherwise a binary (a wheel?) is downloaded that contains the C extensions. Not sure yet it solves the problem, but it removed the extensions as now:

find venv/lib/python3.5/site-packages/newrelic-2.78.0.57 -name *.so

finds no result after installing on a new virtualenv.

[adamchainz]

@giorgiosironi read @andrewgoetz message there, there's an official fix/workaround now

[giorgiosironi]

I tried changing the uWSGI options to no avail before, so I'm documenting here the path to correctly avoid installing C extension for the poor souls which will find themselves in the same situation.

[giorgiosironi]

I confirm that avoiding the C extensions seems to solve the problem.

[a-feld]

@giorgiosironi I believe uWSGI v2.0.15 contains a fix which prevents the SystemError issue with wsgi-env-behavior set to cheat.

If you upgrade to v2.0.15 and set wsgi-env-behavior=cheat (or avoid setting wsgi-env-behavior) does that eliminate the SystemError?

[giorgiosironi]

As we were on 2.0.14, we set on disabling the C extensions during installation (NEW_RELIC_EXTENSIONS=false pip install --no-binary :all: newrelic==2.82.0.62) as the final solution. To try out new combinations of versions and configuration we would have to deploy it and wait for a few days for the problem to possibly show up, so it's difficult to do.