Open danielcaseyAus opened 2 months ago
@danielcaseyAus - just wanted to say thanks very much for flagging this. @lindsaykatz and I are looking around to work out what's going on and develop a fix.
I think I’ve identified what may be the issue with this dataset. I assume you are relying on the “context” variable to identify questions? Unfortunately This variable was not consistently used across time periods. It seems like prior to May 2011. The options within this variable have changed, and the "questions without notice" option only appeared from 2011. So, if you rely on that variable prior to that, it won't work.
many of the older CSV files don’t seem to correctly identify questions? Take 15 March 2000 as an example, I filtered the body to include the phrase “My question is to…”, and ask you can see, all of these come up with a zero in the question column?![eg1](https://github.com/lindsaykatz/hansard-proj/assets/94588569/2e18a30a-f6cd-413a-8ce1-026edcedcd4e)
At some later point (see 19 June 2018) it is addressed:![eg2](https://github.com/lindsaykatz/hansard-proj/assets/94588569/33016f4c-ca3f-4cad-ba41-8d6af95d98ba)