Fail to parse agenda questions

comsaint commented 9 years ago

Multiple issues with agenda questions:

Questions of agendas on 2014.06.18 are successfully scraped and shows up in the parsed source documents, but some questions are missing in view (for English version only).
When there are more than one responder for a question, the current view shows only one of them.
There is at least one occasion that the content of a question is completely ripped off from the view, although this content exists in the parsed source document (for both languages), and the question entry for it also exists in view. See Q5 in agenda on 2014.10.15.
It may be a good idea to include a field responders in the model RawCouncilQuestion.

comsaint commented 9 years ago

A side-note: The RawCouncilQuestion instances saved in database comes from scraping webpages such as http://www.legco.gov.hk/yr13-14/english/counmtg/question/ques1314.htm, which contains a link to its respective agenda (and reply), while the issue mentioned above comes from parsing the agendas directly. It seems that these hyperlinks all contain an anchor for a question - maybe this is a good start to break the issue.

comsaint commented 9 years ago

A side-note: The RawCouncilQuestion instances saved in database comes from scraping webpages such as http://www.legco.gov.hk/yr13-14/english/counmtg/question/ques1314.htm, which contains a link to its respective agenda (and reply), while the issue mentioned above comes from parsing the agendas directly. It seems that these hyperlinks all contain an anchor for a question - maybe this is a good start to break the issue.

comsaint commented 9 years ago

There is a reply link alongside each question on e.g. Legco 13-14. It returns a well-structured HTML page that consists of both question and reply. Since we will need to scrape the replies eventually, we may consider moving the creation of question instances here, i.e. scrape both questions and replies from that same page. The drawbacks are that:

We have to parse agendas anyway (since it contains more than questions), it would mean duplicating work.
There may be inconsistency (due to parsing errors) between questions from agenda, the above link, and Hansard (in future).

Older questions (from year 2005-2006 back) do not have such a reply link. Need to parse the Hansard instead. However, since we need to parse the Hansard anyway, there is no extra work.

comsaint commented 9 years ago

Found a note in raw.models.parsed.QuestionManager.populate(), which shares my idea above.

comsaint / legco-watch

Fail to parse agenda questions #2