Exploration and Summary Statistics

[EliasHadjAmmar]

Will use this issue to explore the data, compute summary stats and make visualisations. Get a feeling for the data and how the HNE shows up in it. Open questions at this point:

• is the data build working as intended?
• how many observations am I going to have on each level, and is that a lot?
• what varies how much on what level?

[EliasHadjAmmar]

Fixed a bug that caused duplicate city-year observations in the build. I used this now-deleted Rmd file to diagnose the problem, in case it comes up again. Also added summary statistics for cities to Exploration.Rmd.

Left to do here:

• [x] sanity check the extinctions by comparing to territories.csv
• [x] explore treatments, feasible treatment windows and sub-experiments
• [ ] ask Prof. Cantoni for yearly construction data
• [ ] make some city-level plots

[EliasHadjAmmar]

Results:

• the extinction lists obtained from the ruler and territories data differ a lot, but there is also quite a bit of overlap. Consider using only extinctions as treatments which you can find in both. I did set up an infrastructure to combine both into a final, authoritative version. I may eventually write a function that uses regex to get exact treatment years from the comments in the territories-based list.
• I wrote a minimal version of the code that turns the final extinction list into sub-experiments to be stacked. It's basically an assignment of city_ids to experiment groups - a dummy variable for each sub-experiment, which takes values 1 (treat), 0 (control), and missing (not in it). I still need to find the best/most elegant way to output the list of sub-experiments alongside it. (could've joined but it would have been huge)
• At least as long as you place no restrictions on count_diffs in the inclusion period, most treatment windows seem feasible. The maximum number of sub-experiments is always limited to the 382 ruler-extinctions, and it gets smaller as you increase the window (or minimum number of treatments per experiment), but the number of experiments you get declines slowly.
• I think it would be good to actually set a small window (10 to max 20 years) but a high threshold of treatments per experiment, if the data ultimately affords it.
• I should maybe do some plots of the number and timing of sub-experiments and the number of cities involved in them.

[EliasHadjAmmar]

The code is at a resting point; I will stop working on it for now. I can do the remaining exploration when I actually get yearly construction data; then I can actually do some exemplary city plots too, rather than just count affected cities. I have seen enough to be convinced that it is feasible to get stacked sub-experiments; while I haven't checked it, I am confident there will still be enough sub-experiments once I add restrictions on nonzero count_diff in the inclusion period. That's where I should begin once I pick this up again.