Internal Error in stats view

[tonioo]

From @fionnb on February 18, 2016 20:25

Hi, on a fresh 1.4.3 install I get errors in the stats module.

File "modoboa_stats/graphics.py", line 94, in export
    timestamp = int(row.find('t').text)

AttributeError: 'NoneType' object has no attribute 'text'

i went digging a bit and had it dump the command, which is:

rrdtool xport --start -1week --end 1455825274 DEF:size_recv=/mail/modoboa/stats/global.rrd:size_recv:AVERAGE CDEF:size_recvpm=size_recv,60,* XPORT:size_recvpm:"Empfangsgr\xc3\xb6sse" DEF:size_sent=/mail/modoboa/stats/global.rrd:size_sent:AVERAGE CDEF:size_sentpm=size_sent,60,* XPORT:size_sentpm:"Versendegr\xc3\xb6sse"

Apparently, when using this command, the result will not contain any 't' fields:

<data>
    <row><v>3.9836091071e+05</v><v>6.2045392857e+04</v></row>
  ...

... and the subsequent code fails. I wonder what goes wrong while constructing the command?

Can I do anything to make it work properly?

Using rrdtool 1.5.5 btw.

Copied from original issue: tonioo/modoboa#836

[tonioo]

Indeed, it looks pretty strange... Could you try with an older version of rrdtool (like 1.4.7) ?

[tonioo]

From @fionnb on February 21, 2016 16:30

confirmed: appears in 1.4.7 and is missing in 1.5.5.

[tonioo]

Looks like the output is a bit different with 1.5.5. Anyway, 1.5.5 should have a working JSON output (unlike 1.4.7), I'll make a change to use it if present.

[tonioo]

From @fionnb on February 22, 2016 9:31

I get it that 1.4.4 does not have the fix yet?

[tonioo]

You're right. The stats plugin has its own repository (https://github.com/modoboa/modoboa-stats) so this issue is misplaced.

[tonioo]

From @fionnb on February 22, 2016 10:27

I am sorry then. Should I open an issue there or just wait for an update to appear? Feel free to close this issue if inappropriate.

[tonioo]

Don't worry, I'm going to move this issue.

[awein]

The above output shows that the timestamps are missing. This is true for the JSON output as well.

[awein]

I had a look at the rrd codebase and recognized that it's required to use the --showtime flag to get timestamps in the output.

The fix is easy:

Change line 103 from

        cmd = "{} xport --start {} --end {} ".format(

to

        cmd = "{} xport --start {} --end {} --showtime ".format(

[tonioo]

@awein Great, thank you ! I guess this option has been added to rrdtool 1.5.x, right ?

[kryskool]

@tonioo on the changelog, http://oss.oetiker.ch/rrdtool/pub/CHANGES

RRDtool 1.6.0 - 2016-04-19
==========================
* re-introducted --showtime option on rrdxport

I don't see in the changelog, when this option have been removed

[fionnb]

Too bad ubuntu 16.04 still has 1.5.5 and that version does not have the showtime option :-(

[ricardokuki]

Helo Guys,

first...sorry my bad english : |

Im getting a message "internal error" when i try to see the stats into modoboa 1.5.3

I tried edit the file graphics.py removing the option "extra_args" and "{}" to try to solve the problem but I stiil getting the message and the graphics dont show up.

Im using Ubuntu 16.04 with rrdtool 1.5.5

As fionnb told, the rrdtool 1.5.5 doesnt have the showtime option

this is part of my current graphics.py: . . . """Export data to XML using rrdtool and convert it to JSON.""" result = [] cmdargs = [] for curve in self._curves: result += [{ "name": curve.legend, "color": curve.color, "data": [] }] cmdargs += curve.to_rrd_command_args(rrdfile) code = 0 for extra_args in [" --showtime", ""]: cmd = "{} xport --start {} --end {}{} ".format( self.rrdtool_binary, str(start), str(end), extra_args) cmd += " ".join(cmdargs) if isinstance(cmd, unicode): cmd = cmd.encode("utf-8") code, output = exec_cmd(cmd) if code: continue if code: return [] tree = etree.fromstring(output) for row in tree.xpath('/xport/data/row'): . . .

and I tried to solve using this way:

. . . """Export data to XML using rrdtool and convert it to JSON.""" result = [] cmdargs = [] for curve in self._curves: result += [{ "name": curve.legend, "color": curve.color, "data": [] }] cmdargs += curve.to_rrd_command_args(rrdfile) code = 0 for extra_args in [" --showtime", ""]: cmd = "{} xport --start {} --end {} ".format( self.rrdtool_binary, str(start), str(end)) cmd += " ".join(cmdargs) if isinstance(cmd, unicode): cmd = cmd.encode("utf-8") code, output = exec_cmd(cmd) if code: continue if code: return [] tree = etree.fromstring(output) for row in tree.xpath('/xport/data/row'): . . .

any idea to solve the "internal error" ?

Best Regards, Ricardo

[tonioo]

Could you please enable DEBUG and send a proper stack trace ? (use a debugger in your browser)

[ricardokuki]

Hello Tonioo,

Sorry... but How I get a "Stack Trace" ?

Im using chrome.. and it is showing me these errors/warnings:

warning: Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/. GET http://mail.skymsen.com.br/stats/graphs/?gset=mailtraffic&_=1466187900551 500 (Internal Server Error) XHR finished loading: GET "http://mail.skymsen.com.br/core/top_notifications/check/".

error: jquery.min.js:5 GET http://mail.skymsen.com.br/stats/graphs/?gset=mailtraffic&_=1466188139587 500 (Internal Server Error)send @ jquery.min.js:5b.extend.ajax @ jquery.min.js:5History.check @ history.js:272b.isFunction.i @ jquery.min.js:3 jquery.min.js:5 XHR finished loading: GET "http://mail.skymsen.com.br/stats/graphs/?gset=mailtraffic&_=1466188139587".send @ jquery.min.js:5b.extend.ajax @ jquery.min.js:5History.check @ history.js:272b.isFunction.i @ jquery.min.js:3

Thank you Ricardo

[tonioo]

Thanks for your feedback but it is not usefull. Could send the response body instead ?

[cekpek]

@tonioo i am not sure whether this is what you want. I am having the same problem with the stats as well

<!DOCTYPE html>

<html lang="">
  <head>
    <meta charset="utf-8" />
    <title></title>
    <link rel="icon" type="image/png" href="/sitestatic/css/logo-icon.png" />
    <link rel="stylesheet" type="text/css" href="/sitestatic/bootstrap/dist/css/bootstrap.min.css" />
    <script src="/sitestatic/bootstrap/dist/js/bootstrap.min.js" type="text/javascript"></script>
    <script src="/sitestatic/eonasdan-bootstrap-datetimepicker/build/js/bootstrap-datetimepicker.min.js" type="text/javascript"></script>

    <script type="text/javascript" src="/jsi18n/"></script>
    <style>
      body {
          padding-top: 60px;
     }
    </style>
  </head>

  <body>
    <div class="container-fluid">
      <div class="row-fluid">

<div class="alert alert-danger">
  <h4>Sorry</h4>
  <p>An internal error occured.</p>
</div>

      </div>
    </div>
  </body>
</html>

[tonioo]

You need to enable debug information. Just set DEBUG to True inside the settings.py file, you should then see the real issue.

[dhaupin]

Im getting this too on the "Statistics" pane in admin GUI. Ubuntu 16.04 with rrdtool Version: 1.5.5-4. Requested route gives a 500 error - /stats/graphs/?gset=mailtraffic&_=1474661219137 causing "internal error" message in "Statistics" pane.

FYI those trying to use debug, do an "updatedb && locate settings.py". Find the one in your modoboa instance (prob in /srv/modoboa/instance/). Edit it to use True for DEBUG. Restart uwsgi and nginx services. Open chrome console, hit refresh on "Statistics" and look for a server error. Click the link in the error and you should see a debug trace. Click "Switch to copy-and-paste view" on the trace part. Here is an output from mine:

AttributeError at /stats/graphs/

'NoneType' object has no attribute 'text' Request Method: GET Request URL: https://srv.example.com/stats/graphs/?gset=mailtraffic&_=1474661219137 Django Version: 1.9.9 Exception Type: AttributeError Exception Value:
'NoneType' object has no attribute 'text' Exception Location: /srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/graphics.py in export, line 117 Python Executable: /usr/bin/uwsgi-core Python Version: 2.7.12 Python Path:
['.', '', '/srv/modoboa/env/lib/python2.7', '/srv/modoboa/env/lib/python2.7/plat-x86_64-linux-gnu', '/srv/modoboa/env/lib/python2.7/lib-tk', '/srv/modoboa/env/lib/python2.7/lib-old', '/srv/modoboa/env/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/srv/modoboa/env/local/lib/python2.7/site-packages'] Server time: Fri, 23 Sep 2016 16:07:30 -0400

Environment

Request Method: GET Request URL: https://srv.example.com/stats/graphs/?gset=mailtraffic&_=1474661219137

Django Version: 1.9.9 Python Version: 2.7.12 Installed Applications: ('django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.sites', 'django.contrib.staticfiles', 'reversion', 'rest_framework.authtoken', 'rest_framework_swagger', 'modoboa', 'modoboa.core', 'modoboa.lib', 'modoboa.admin', 'modoboa.relaydomains', 'modoboa.limits', 'modoboa_amavis', 'modoboa_pdfcredentials', 'modoboa_postfix_autoreply', 'modoboa_sievefilters', 'modoboa_stats', 'modoboa_webmail') Installed Middleware: ('x_forwarded_for.middleware.XForwardedForMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'modoboa.lib.middleware.AjaxLoginRedirect', 'modoboa.lib.middleware.CommonExceptionCatcher', 'modoboa.lib.middleware.RequestCatcherMiddleware')

Traceback

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response

1. response = self.process_exception_by_middleware(e, request)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response

1. response = wrapped_callback(request, _callback_args, *_callback_kwargs)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/utils/decorators.py" in inner

1. return func(_args, *_kwargs)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/utils/decorators.py" in inner

1. return func(_args, *_kwargs)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view

1. return view_func(request, _args, *_kwargs)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view

1. return view_func(request, _args, *_kwargs)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/views.py" in graphs

1. tplvars["graphs"] = gsets[gset].export(tplvars["domain"], start, end)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/graphics.py" in export

1. "curves": graph.export(rrdfile, start, end)

File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/graphics.py" in export

1. timestamp = int(row.find('t').text)

Exception Type: AttributeError at /stats/graphs/ Exception Value: 'NoneType' object has no attribute 'text'

Request Information

GET Variable Value gset
u'mailtraffic' _ u'1474661219137' POST No POST data FILES No FILES data COOKIES Variable Value csrftoken 'xxxxxxxxxxxxxxxxxxxxxxxxxxx' sessionid 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' _ga
'xxxxxxxxxxxxxxxxxxxx' __cfduid
'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' META Variable Value wsgi.multiprocess True HTTP_COOKIE
'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' UWSGI_SCRIPT
'instance.wsgi:application' SCRIPT_NAME
u'' REQUEST_METHOD
'GET' PATH_INFO u'/stats/graphs/' SERVER_PROTOCOL
'HTTP/1.1' QUERYSTRING
'gset=mailtraffic&=1474661219137' CONTENT_LENGTH
'' HTTP_USER_AGENT
'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2869.0 Safari/537.36' HTTP_CONNECTION
'keep-alive' SERVER_NAME
'srv.example.com' REMOTE_ADDR
'123.123.123.123' wsgi.url_scheme
'https' SERVER_PORT
'443' uwsgi.node
'srv' DOCUMENT_ROOT '/srv/modoboa/instance' wsgi.input
<uwsgi._Input object at 0x7f21ce1d89a8> HTTP_DNT
'1' HTTP_HOST 'srv.example.com' HTTPS 'on' wsgi.multithread
False HTTP_UPGRADE_INSECURE_REQUESTS
'1' REQUESTURI
'/stats/graphs/?gset=mailtraffic&=1474661219137' HTTPACCEPT
'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/_;q=0.8' wsgi.version
(1, 0) wsgi.run_once False wsgi.errors
<open file 'wsgi_errors', mode 'w' at 0x7f21ce167930> REMOTE_PORT
'56135' HTTP_ACCEPT_LANGUAGE
'en-US,en;q=0.8' REQUEST_SCHEME
'https' UWSGI_APPID
'srv.example.com|' uwsgi.version '2.0.12-debian' CONTENT_TYPE
'' wsgi.file_wrapper '' CSRF_COOKIE
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' HTTP_ACCEPT_ENCODING
'gzip, deflate, sdch, br'

Django settings truncated since they are default besides debug and listener domain

[tonioo]

This issue has been definitely fixed by https://github.com/modoboa/modoboa-stats/pull/13. I'll release a new version soon.