Get results (API call)

[chompomonim]

Return systemised results for given test (mps and fractions with their meta data and answers/positions, list of questions with user's answers) for user who already answered all test questions. If not all test questions was answered, return alert about that with no results.

If /test/results URL was opened by loggedin user, we check if user already answered test. If yes, then generate his results for current test, if no, redirect to start test page.

[sirex]

@chompomonim @laureenas pradėjau daryti šitą užduotį, bet iškilo toks dalykas, kad temos pozicijose (ką esu padaręs anksčiau) nėra padarytas susiejimas su ParliamentMember ir Group objektais iš mps_v2 programėlės.

Iš ResultsView pavyzdinių duomenų matau, kad frakcijų ir seimo narių id yra reikalingi, kaip ir logotipai, kurie yra prie minėtų ParliamentMember ir Group.

Šiandien šito dalyko nespėsiu padaryti.

Planuoju tuo užsiimti ketvirtadienį.

[sirex]

Well, that took me a bit more time than I expected, but topic positions should be in place.

Each time new voting is added to the topic via admin, positions for all fractions and MPs are recalculated.

All tests pass, but did not tested this manually yet.

Also https://github.com/ManoSeimas/manoseimas.lt/commit/ab0494f6a1c098b54ecac3e1efeebb8f7932e288 breakes MPs profiles, because of changed positinos structure. I need to update that bit, but I will do this tomorrow and will do manual testing.

Another thing, that I need to check is term of service for MPs, fractions and positions. Currently this part is a mess I need bit more time to think, how to fix that.

[chompomonim]

At the moment it doesn't work well, because it /results gives not expected results.

We're expecting:

{
   user_answers: ...,
   fractions: [
       {
           id: 1,
           title: 'Foo fraction',
           short_title: 'BAR',
           logo: 'some/URL',
           members_amount: 9,
           answers: {1: 1, 2: 0.3, 3: 0.5, 4: -1}
       }
       ....
   ],
   mps: [
       {
           id: 1,
           name: 'Foo Bar',
           fraction: 'BAR',
           fraction_id: 1,
           logo: 'some/URL',
           answers: {1: 1, 2: 0.3, 3: 0.5, 4: -1}
       },
      ...
   ]
}

Now we're getting:

{
    user_answrs: {}, // This is ok
    fractions: [
        {
           id: 1,
           title: 'Foo fraction',
           short_title: 'BAR',
           logo: 'some/URL',
           members_amount: 9,
           // HERE IS PROBLEM!
           answers: {
              against: [
                   {
                       formatted: "Nepalaiko 57%",
                       klass:"against"
                       node_ref: Object
                       permalink: "/visagino-atomine-elektrine/"
                       position:"-1.1522"
                       title:"Visagino atominė elektrinė"
                   },
                   ....
              ],
              for: [...],
              neutral: [...]
        ...
    ]
}

In answers i'm expecting Object where key is topic_id and value is integer in range -1 ... 1. Value is already pre-calculated for fraction (using mps votings summing them and dividing by mps amount).

[sirex]

You get old results, these should be replaced when you save any topic in admin, after saving all positions of all MPs and fractions will be recalculated to {1: 1, 2: 0.3, 3: 0.5, 4: -1} form.

But recalculation is also does no work yet, because I found, that MPs scraper does not work properly with MPs pages from new term, it does not take term of office field. Current implementation, takes MPs only from current term of office and scraper does not extract that field, so nothing gets updated.

Now I'm fixing the scraper, will let you know, when I'm done.

[sirex]

@chompomonim fixed issue with term of service. Now to make it work, you need to run bin/scrapy crawl mps and resave at least one topic containing assigned voting from 2012-2016.

Also in order for positions to be updated, at least on topic should have at least one voting from 2012-2016 trem.

Currently only those MPs and fractions coming through topic votings from 2012-2016 term will be updated, others will have old positions.

I'm not sure whether restriction 2012-2016 term should be applied or not, but different terms have different MPs from different flactions and it means, that without restricting just to 2012-2016 we will get all MPs and fractions pulled in. As I understand, we only want to show MPs from 2012-2016 term?

[sirex]

@chompomonim could you check if topic positions work as expected? And if they work, please close the task.

From my side, it looks, that now everything in place and should work.

[chompomonim]

I checked, and locally it works. But... 1) Will votings from old cadention work? 2) Locally I'm able to add new voting, but on server it doesn't work, I'm getting: The server encountered an internal error or misconfiguration and was unable to complete your request.

@sirex Could you help me with staging configuration? I'd like to show for TILS what we have.

[sirex]

1) Will votings from old cadention work?

No.

2) Locally I'm able to add new voting, but on server it doesn't work, I'm getting: The server encountered an internal error or misconfiguration and was unable to complete your request.

There are many pages in lrs.lt with slightly different designs, and voting import does not support all of them. Which voting url did you tried to import?

Currently only 2012-2016 votings should be supported.

[sirex]

@sirex Could you help me with staging configuration? I'd like to show for TILS what we have.

What issues do you have?

[chompomonim]

I can't add new votes for Topics. And even don't know where are logs to see what's the problem, and what is error message.

[sirex]

It looks, that issue with voting import appears only in staging server.

You can reproduce it, when you do voting import:

http://www3.lrs.lt/pls/inter/w5_sale.bals?p_bals_id=-22311

from Django admin, you will get following error:

2016-07-11 16:24:17,838 INFO connectionpool Starting new HTTP connection (1): www3.lrs.lt
2016-07-11 16:24:17,843 DEBUG connectionpool "GET /pls/inter/w5_sale.bals?p_bals_id=-22395 HTTP/1.1" 403 None
2016-07-11 16:24:17,845 ERROR base Internal Server Error: /valdymas/scrapy/voting/add/
Traceback (most recent call last):
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/contrib/admin/options.py", line 616, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/decorators.py", line 110, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/contrib/admin/options.py", line 1516, in add_view
    return self.changeform_view(request, None, form_url, extra_context)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper
    return bound_func(*args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/decorators.py", line 110, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/decorators.py", line 30, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/contrib/admin/options.py", line 1461, in changeform_view
    new_object = self.save_form(request, form, change=not add)
  File "/opt/manoseimas/app/manoseimas/scrapy/admin.py", line 32, in save_form
    return services.crawl_voting(source)
  File "/opt/manoseimas/app/manoseimas/scrapy/services.py", line 32, in crawl_voting
    items = list(spider.parse_person_votes(response))
  File "/opt/manoseimas/app/manoseimas/scrapy/spiders/sittings.py", line 421, in parse_person_votes
    hxs = HtmlXPathSelector(response).select(xpath)[0]
IndexError: list index out of range

[sirex]

Ok, it looks, that iv-5.pov.lt got banned:

>>> import requests

>>> r = requests.get('http://www3.lrs.lt/pls/inter/w5_sale.bals?p_bals_id=-22311')

>>> r
<Response [403]>

>>> print(r.text)

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>403 Forbidden</TITLE>
</HEAD><BODY>
<H1>Forbidden</H1>
You don't have permission to access /pls/inter/w5_sale.bals
on this server.<P>
</BODY></HTML>

This URL works perfectly for me, but it gets 403 on server.

[chompomonim]

Wow, interesting. Will they ban manoseimas.lt when we release it???

I'll try to deploy to some of my VMs, and will see how it works.

[sirex]

I have been banned several times before. Not sure how they decide who should be banned, but some times it happens. But there was a case, when I was scraping 25 Gb of legal documents whole week, they didn't banned me.

If TILS would talk to the LRS kanceliarija, maybe they can add our servers to the white list to prevent such accidents.

By the way, if you will deploy on other servers, I suggest you to disable all scraping cron jobs and run just mps scraper, this is the only scraper required for test to work, all other scrapers are used for mps_v2.

[chompomonim]

Any ideas why I'm getting such error when install into new server?

TASK [migrate database] ********************************************************
fatal: [manoseimas.nous.lt]: FAILED! => {"changed": true, "cmd": ["bin/django", "migrate", "--noinput"], "delta": "0:00:00.227493", "end": "2016-07-12 12:17:59.198067", "failed": true, "rc": 1, "start": "2016-07-12 12:17:58.970574", "stderr": "Traceback (most recent call last):\n
  File \"bin/django\", line 103, in <module>\n
    sys.exit(djangorecipe.binscripts.manage('manoseimas.settings.production'))\n
  File \"/opt/manoseimas/app/eggs/djangorecipe-2.1.2-py2.7.egg/djangorecipe/binscripts.py\", line 9, in manage\n
    management.execute_from_command_line(sys.argv)\n
  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/core/management/__init__.py\", line 351, in execute_from_command_line\n
    utility.execute()\n
  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/core/management/__init__.py\", line 325, in execute\n
    django.setup()\n
  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/__init__.py\", line 15, in setup\n    from django.utils.log import configure_logging\n
  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/log.py\", line 16, in <module>\n
    from django.views.debug import ExceptionReporter, get_exception_reporter_filter\n
  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/views/debug.py\", line 9, in <module>\n
    from django.core.urlresolvers import Resolver404, resolve\n  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/core/urlresolvers.py\", line 17, in <module>\n
    from django.http import Http404\n  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/http/__init__.py\", line 2, in <module>\n
    from django.http.request import (HttpRequest, QueryDict,\n  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/http/request.py\", line 12, in <module>\n
    from django.core import signing\n  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/core/signing.py\", line 46, in <module>\n
    from django.utils.crypto import constant_time_compare, salted_hmac\n
  File \"/opt/manoseimas/app/eggs/Django-1.8.5-py2.7.egg/django/utils/crypto.py\", line 8, in <module>\n
    import hmac\n  File \"/usr/lib/python2.7/hmac.py\", line 8, in <module>\n
    from operator import _compare_digest as compare_digest\nImportError: cannot import name _compare_digest", "stdout": "", "stdout_lines": [], "warnings": []}

[sirex]

Never saw anything like this. What OS do you use?

[chompomonim]

Ot's Digital Ocean with Ubuntu 14.04

[chompomonim]

It looks like some file system limitation on Digital Ocean. I just tried on IV servers, and script finished works successfully, but something is still wrong. I'm getting Bad Request (400) on http://manoseimas.nous.lt and there is 0 information in logs :/ Any ideas how and where to check? How manually start?

[sirex]

Probably easiest way is to temporarily enable debug mode in /opt/manoseimas/app/manoseimas/settings/production.py. After changing that, restart Apache.

If you still get 400 page, then it should be an Apache configuration error.

[sirex]

Forgot to mention, to enable debug mode, add DEBUG = True in /opt/manoseimas/app/manoseimas/settings/production.py.

[chompomonim]

Woohoo, debug helped ;)