Interface with Parliament.ge contact form (>5h)

[ddohler]

We've gotten the go-ahead from the Parliament to do this. I assume it will be >5h, but hopefully we can get it finished by the public beta, if not, we may push out the launch date so that we have this functionality. We'll be in touch with the Parliament IT department in case there are any issues that come up; I assume it can be done with a simple POST. An example of the form on a live page: http://www.parliament.ge/index.php?option=com_k2&view=item&id=3:abulashvili-nugzari&Itemid=129&lang=ge

The way I think this should work is that a unique email address should be generated for each question so that replies will get routed to the proper answer.

[ddohler]

As an additional note, the Parliament told us that they are eventually planning to require users to provide a personal ID number in order to use the Parliament website form. I'm not very happy about that, but to the extent that we can be ready to react to that if/when it happens, this new functionality should be designed with that in mind.

[ddohler]

We talked to the Parliament today, and it seems that we can move ahead with doing it as I mentioned above, i.e. auto-generated email address (e.g. question12345678@shenmartav.ge ) that will then be POSTed to the Parliament's form with some boilerplate text wrapping the user text. We still need to talk to another agency about this, but I'm pretty confident that we'll get the go-ahead.

Since we are down to mostly content issues on the Public Beta milestone, do you think we could have this one sometime next week? I'm willing to push back the Public Beta by a few days if we can get this functionality for it.

[sebastiantransparency]

hm, not sure.

• figure out how to POST to their form
• figure out how to teach the mail server to send from these addresses and when receiving, feed it into shenmartav
• implement the handling of these mails within shenmartav

i guess this will be 20h. i think i can't do that with a full-time job already within a week. i'd rather be done with the remaining 2 issues involving <5h and we will add that feature during the public beta - or hold it back for another week.

[sebastiantransparency]

what about Issue #7, btw? is that redundant, then?

[ddohler]

Okay, we'll make a final decision on Tuesday whether to hold back public beta or move ahead with the current functionality.

As far as #7, that's not redundant to this, we would still want to send reminder emails to representatives if they haven't answered a question after a certain period of time.

[ddohler]

I just realized I'm not entirely sure what the current functionality is; in the absence of interoperability with the Parliament's form, what will happen when someone writes a question? Will an email be sent directly to the MP? If so, where do we configure these emails, I didn't see them in the admin section for Representatives.

[sebastiantransparency]

currently, the question is stored unpublished in the database and the site's managers will receive an email with a link to the question in the admin. speaking of which: there are no managers defined yet, apart from me. what email addresses shall i use for the managers?

since the process had not been defined before, currently the manager would then publish the question, contact the representative and paste the answer in the admin.

i don't know if we have email addresses of all representatives.

[ddohler]

Okay, that'll work for us, I'm moving this to the release milestone.

[ddohler]

I'm adding Top Priority to this; having this functionality will significantly decrease our workload since we will be handling questions submitted via SMS manually. We will talk to the Civil Service Bureau about the web form next week, but it would be worth it to do some in-depth investigation of the issue beforehand in case there are any issues I can raise with them at our meeting.

[sebastiantransparency]

alright, i will do this on the weekend. i don't want to start doing it in the morning and then having to leave it sitting around for a day for a couple of days.

[ddohler]

As a note, submitting the contact form on the Parliament web site currently returns 403: Forbidden. We'll ask them why that is, next week.

[sebastiantransparency]

ok, i will hold back then and work on the remaining smaller issues first.

[sebastiantransparency]

i wrote a little script to use the contact facilities at parliament.ge (see at the bottom), but apart from a status code 200, i get no response which lets me verify that the data was accepted. could somebody please ask @ parliament if they have a received a bunch of messages containing data like below? i looked at the source code of the javascript sending the form data (http://parliament.ge/plugins/content/dmaskinfopopup/popup.js:sendMailInfo) and it doesn't expect any error messages either.

import urllib
import urllib2

url = 'http://parliament.ge/index.php?option=com_dmaskinfopopup'
id = '3'
values = {
#    'DMAskInfo_sendername' + id : 'tester',
#    'DMAskInfo_senderemail' + id : 'foo@bar.com',
#    'DMAskInfo_article' + id: id,
#    'DMAskInfo_adminemail' + id : '-1',
#    'DMAskInfo_actcomponent' + id : 'com_k2',
#    'DMAskInfo_mailsubject' + id : 'question',
#    'DMAskInfo_mailmessage' + id : 'please ignore',
    'mail' : 'foo@bar.com',
    'name' : 'tester',
    'subj' : 'question',
    'mess' : 'please ignore',
    'arid': id,
    'adminemail' : '-1',
    'extrafield' : '',
    'extrafield2' : '',
    'extrafield3' : '',
    'component' : 'com_k2',
}

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
#import pdb; pdb.set_trace()
print response.read()

[ddohler]

We're scheduled to meet with Irakli on Monday, so we'll bring this up then. I've got your questions on my list of points to raise.

[ddohler]

Some updates here:

• We couldn't meet with Irakli because he has resigned to run for Parliament (whether it is temporary or not, we shall see). We met with his assistant instead, but she was quite knowledgeable and was able to answer our questions.
• Apparently the only way to tell whether a message was successfully sent is to ask the IT guys at Parliament, because it goes through their mail server. No error message is sent, except (perhaps) the HTTP response code. So we'll go back and check with Merab again about your test messages.
• The questions are emailed directly to MPs after the form is submitted.
• There is a Javascript popup that appears when the message is "successfully" sent, but I think it just relies on receiving a 200 from the server.
• There is some friction between the CSB and Parliament over the new website, so it may be tough to get straight answers because there is confusion regarding who is in charge of what.

Anyway, I would say that at this point we should simply assume that the script works and move forward with the rest of the system, treating 200 responses as success. We'll get in touch with Parliament to see if they can look through their logs to confirm that the script is working. @EkaR

[sebastiantransparency]

Lasha Tordia exists in our database, but is not on the list at http://www.parliament.ge/index.php?option=com_content&view=article&id=1682&Itemid=371&lang=en (dunno if that's important)

[sebastiantransparency]

done now. the response from parliament.ge, incl. HTTP status code, is stored in a text field parliament_response with each question. so editors can quickly see if something is going wrong there.

note that this happens synchronously while the user sends the ask form. so if parliament.ge is not responding in a timely manner, the user will have to wait. i was doing a bit of research and people advise against spawning another thread within django, but rather use message queue systems, like RabbitMQ or Celery. i didn't want to include that piece of infrastructure yet. but if the response time from parliament.ge becomes an issue, we should do it.

[ddohler]

Celery has really good Django hooks, so I would suggest that if it becomes necessary.

Just to clarify, is the question submitted to Parliament's form on submission by the end user, or on approval by the site managers? It should be the second one, which also means that response times are not such a huge issue -- it's not a big deal if the editors have to wait.

[sebastiantransparency]

oops, sorry it's on submission by the end user. i will fix that.

[sebastiantransparency]

btw, the sender email used has the form ask-@shenmartav.ge . please configure the mail server to handle these addresses.

[sebastiantransparency]

the parliament response field will contain text like this:

SENDER: ask-@shenmartav.ge RESPONSE CODE: xxx (usually 200) RESPONSE CONTENT: (empty at the moment)

[ddohler]

Done, I've set up a catch-all address. If two people ask a question to the same representative, will there be any way to distinguish those two questions via the email address? For example "ask-davit-bakradze-1@shenmartav.ge" and "ask-davit-bakradze-2@shenmartav.ge"?

[ddohler]

A bit more on this:

• We're getting Unicode errors, which I assume you saw. ASCII should be banned...
• Could you automatically fill the required model fields in both languages with whatever the user input is? When we try to publish questions, we get errors that the translations for several required fields are missing. It would be easiest for us if the user input would simply be copied into both fields, and we can then translate them later if we need to.
• We need to wrap the text submitted to the Parliament form in the following template (sorry for forgetting to tell you this earlier):

{% if MP.gender == female%}ქალ{% endif %}ბატონო {{ MP.first_name as in firstname lastname format (no -ი) }}, #There is NO space between ქალ and ბატონო

მოქალაქემ {{ user.name }} მოცემული შეკითხვა დაგისვათ ვებგვერდის – ShenMartav.ge საშუალებით: 

„{{ question.text }}“

გაცნობებთ, რომ შეკითხვა და თქვენი პასუხი გამოქვეყნდება ვებგვერდზე. ვებვერდზე ასევე გამოჩნდება თქვენთვის დასმული შეკითხვებისა და თქვენ მიერ გაცემული პასუხების თანაფარდობა. 

„შენ მართავ!“ ხორციელდება არასამთავრობო ორგანიზაციების: „მშვიდობის, დემოკრატიის და განვითარების კავკასიური ინსტიტუტი“ და „საერთაშორისო გამჭვირვალობა – საქართველოს“ მიერ. შენ მართავ! მიუკერძოებელი ვებგვერდია და არ არის დაკავშირებული ან შექმნილი რომელიმე პოლიტიკური პარტიის რეკლამირებისათვის. ```

[ddohler]

Oop, one more thing: could you change the format to ask+[slug] rather than ask-[slug]? GMail will ignore anything after the plus and deliver the message as normal, so that way I won't have to set up a catchall address, and it will be easy to filter those messages.

[sebastiantransparency]

i won't have enough time today to finish all these, what i have done (locally only so far):

• ask+-@shenmartav.ge
• fill fields for both languages (that's another issue re previously not having understood how to handle modeltranslation)

still needs doing:

• template
• unicode errors sigh

[sebastiantransparency]

btw, what do you want as subject? currently, the first 32 chars of the question are used.

[ddohler]

If we're submitting to the form, does there need to be a subject? I don't see a field for it on the website, but maybe there's a POST parameter for it. Anyway, it may be good to have in the future in case we ever need to send direct emails, so @EkaR can you write one in Georgian?

[EkaR]

"თქვენ დაგისვეს შეკითხვა ShenMartav.ge-ზე"

[sebastiantransparency]

thanks for the subject, it is indeed possible to supply one to the actual send URL; at least that's indicated by the AJAX call to their server - the data actually sent slightly differs from the form the user fills in.

alas, it seems they have closed it down / changed it, i'm getting a 403 now. maybe they have included a csrf token now. could you please ask?

btw, if a question is set to public and has an empty parliament response it will be resent on question save. so if you ever want to repeat sending, because the first questions were made when the facility was not in place or that you got an error like 403 previously, ensure it's public, clear the field parliament response and save the question.

[ddohler]

We suspect that there is a hidden length limit: we frequently get 403s when sending actual messages, but 200s when sending "test". Can you confirm?

[sebastiantransparency]

i am surprised you ever get a 200. i never get that when sending 'test'. i added more fields to parliament response: the length of the data that was sent to parliament.ge and the (url-encoded) data itself. so you can play with possible lengths a little bit to come up with a different template. when cutting the length of the field 'mess', i got a 200 up to 490 bytes. the length of the template provided above is 1576 bytes (on my disk), maybe we have to do without it?

[ddohler]

Sorry, should have clarified. We get 200s when sending only "test" -- without the template. With the template we've never received a 200. A 490-byte limit is laughably small; Georgian characters in UTF-8 are 3 bytes each, so the maximum message length is barely longer than a tweet if written in Unicode.

We'll talk to Parliament about this. For now, could you remove the template?

[sebastiantransparency]

the template has been disabled. i tried to send the two published questions, but both still yield a 403. :(

[sebastiantransparency]

i have just fixed another utf-8 issue when sending questions to parliament.ge, then set two questions (where the exception occurred in recent days) to public and saved. both sendings seemed got a 200 response. maybe it's working now on their end?

[ddohler]

Did you send them with the template enabled? If so, that's good news!

[sebastiantransparency]

no, without the template. but i will try that the next weekend.