Check details of the forecast assessment.

[andybega]

Notes to future self to document some details about the 2016-H1 forecast assessment.

Are the forecasts for 2016-H1, or actually 2016-H2?

In 6mo_run_escalation_OOS.R, the actual forecast model:

train_model <- randomForest(escalation_6mo_train_formula_civil_ns,
                            type = regression, importance = TRUE, 
                            na.action = na.omit, ntree = 100000, maxnodes = 5, 
                            sampsize = 100, 
                            replace = FALSE, do.trace = FALSE , 
                            data = escalation_6mo_train_frame_civil_ns, forest = TRUE   
)   

escalation_6mo_train_formula_civil_ns is a formula that is created in 6mo_define_models_OOS.R.

escalation_6mo_train_formula_civil_ns <- as.formula(paste("train_DV_civil_ns",paste(escalation_6mo, collapse=" + "), sep = "~", collapse=NULL))

train_DV_civil_ns refers to the DV, which is created in +master.R line 524:

train_DV_civil_ns <- train$incidence_civil_ns_plus1

For the predictions, back to 6mo_run_escalation_OOS.R:

prediction <- as.numeric(predict(train_model, newdata = data.matrix(escalation_6mo_test_frame_civil_ns), type="response"))    

This is also turned into the pred_escalation_6mo_inc_civil_ns data frame in that file. The escalation_6mo_test_frame_civil_ns that is used to create the predictions is again created in 6mo_define_models.R, using as DV the vector test_DV_civil_ns, which is created in +master.R line 525 from test$incidence_civil_ns_plus1.

All this is created from a test data frame that is setup in lines 513 on in +master.R. It includes 2016-H1:

> range(test$year)
[1] 2008 2016

So, if the target of the forecast model is a lead 1 version of the DV ("_plus1"), then feeding 2016-H1 data into the model generates predictions for 2016-H2, not H1.

In 6mo_make_confusion_matrix.do, the non-lagged "incidence_civil_ns" is used for scoring. So, I think these are 2016-H2 forecasts scored using 2016-H1 outcomes. But probably doesn't matter anyways since there are very few onsets, only ongoing conflicts.

What the ongoing / new conflicts in 2016-H1/H2?

Using the test data that is created at the bottom OOS section of +master.R.

foo <- test %>% 
  as_tibble() %>%
  select(year:incidence_civil_ns, incidence_civil_ns_plus1) %>%
  filter(year >= 2015)

Both DV vars have only 0 and NA values in this data, there are no 1 (onset) values.

There are 15 countries with ongoing conflicts:

> foo %>% filter(year==2016, is.na(incidence_civil_ns)) 

Afghanistan, Democratic Republic of Congo, Colombia, India, Iraq, Libya, Mali, Nigeria, Pakistan, Russia, Sudan, Somalia, Syria, Ukraine, Yemen

In the hand-coded alternative coding (6mo_make_confusion_matrix.do), the Civil War in Colombia is coded as having finished in 2015-H1, with no civil war for the 2016 forecast period. Turkey and Burundi however are coded as seeing new civil wars in 2016, i.e. new onsets.

The 2 false positives in the B&S confusion matrices are DRC and Libya.

So with the original coding, they hit 13 ongoing civil wars and miss 2 ongoing civil wars. With the alternative coding, they hit 12 ongoing, miss 2 ongoing, and hit the 2 new onsets.

Re-reading the text, it now sounds to me like they do kind of acknowledge that they want to take credit for predicting ongoing civil wars, e.g.:

Importantly, we make no assumptions about the persistence of civil war between the second half of 2015 and the first half of 2016, meaning that all countries—even those with ongoing conflicts as of December 31, 2015—are included in the 2016 test set. As a result, 14 of the 30 countries on the list (and nine of the top 10) are cases of ongoing conflict.

Context: in the data ongoing civil wars otherwise are NA's on the DV and thus drop out.

This is so weird.

1. At the time the forecasts were made, in early 2016-H1 (the pre-registration date is in March, I think), we would have known the civil war state for 2015-H2
2. We would not have known, yet, the civil war state for 2016-H1.
3. However, we are not scoring the forecasts until after 2016-H1 has concluded, when we can observe civil war state and determine whether a conflict was ongoing, terminated, or new civil war started.
4. Does it make sense to make forecasts for civil wars that are ongoing in 2015-H2, i.e. put those cases back into our data as being at risk for a new onset?
   1. We know the 2015-H2 civil war state, including presumably termination, so it would make sense to include cases that terminated and now are at risk of another, new civil war outbreak.
   2. If the civil war coding was conflict specific, i.e. a country could have multiple distinct conflicts at any given, time, then it would make sense to not drop ongoing cases. But the Sambanis coding is at the country-level I think, so it doesn't. If you have ongoing conflict, no new onset is possible. Maybe back to back termination-onsets are possible, but I don't think any of those cases of conflict above are that.
   3. Aside from (i) above, I don't see how it makes sense to include 2015-H2 ongoing conflicts back into the data for 2016-H1. You'd have to assume that all those conflicts end in 2015-H2 and the countries are at risk again, but we know they didn't!
5. Anyways, by the time we are scoring the 2016-H1 forecasts, we know what the state of the world is. Ongoing is ongoing, weird to take credit for that as if it is an onset.

[wardlab]

this goes in the paper somehow, but briefly

[andybega]

It’s in now. You have the ball.

On Sep 25, 2020, at 03:29, Michael Ward notifications@github.com wrote:

this goes in the paper somehow, but briefly

---

From: Andreas Beger notifications@github.com Sent: Thursday, September 24, 2020 5:05 PM To: andybega/Blair-Sambanis-replication Blair-Sambanis-replication@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [andybega/Blair-Sambanis-replication] Check details of the forecast assessment. (#11)

Notes to future self to document some details about the 2016-H1 forecast assessment.

Are the forecasts for 2016-H1, or actually 2016-H2?

In 6mo_run_escalation_OOS.R, the actual forecast model:

train_model <- randomForest(escalation_6mo_train_formula_civil_ns, type = regression, importance = TRUE, na.action = na.omit, ntree = 100000, maxnodes = 5, sampsize = 100, replace = FALSE, do.trace = FALSE , data = escalation_6mo_train_frame_civil_ns, forest = TRUE )

escalation_6mo_train_formula_civil_ns is a formula that is created in 6mo_define_models_OOS.R.

escalation_6mo_train_formula_civil_ns <- as.formula(paste("train_DV_civil_ns",paste(escalation_6mo, collapse=" + "), sep = "~", collapse=NULL))

train_DV_civil_ns refers to the DV, which is created in +master.R line 524:

train_DV_civil_ns <- train$incidence_civil_ns_plus1

For the predictions, back to 6mo_run_escalation_OOS.R:

prediction <- as.numeric(predict(train_model, newdata = data.matrix(escalation_6mo_test_frame_civil_ns), type="response"))

This is also turned into the pred_escalation_6mo_inc_civil_ns data frame in that file. The escalation_6mo_test_frame_civil_ns that is used to create the predictions is again created in 6mo_define_models.R, using as DV the vector test_DV_civil_ns, which is created in +master.R line 525 from test$incidence_civil_ns_plus1.

All this is created from a test data frame that is setup in lines 513 on in +master.R. It includes 2016-H1:

range(test$year) [1] 2008 2016

So, if the target of the forecast model is a lead 1 version of the DV ("_plus1"), then feeding 2016-H1 data into the model generates predictions for 2016-H2, not H1.

In 6mo_make_confusion_matrix.do, the non-lagged "incidence_civil_ns" is used for scoring. So, I think these are 2016-H2 forecasts scored using 2016-H1 outcomes. But probably doesn't matter anyways since there are very few onsets, only ongoing conflicts.

What the ongoing / new conflicts in 2016-H1/H2?

Using the test data that is created at the bottom OOS section of +master.R.

foo <- test %>% as_tibble() %>% select(year:incidence_civil_ns, incidence_civil_ns_plus1) %>% filter(year >= 2015)

Both DV vars have only 0 and NA values in this data, there are no 1 (onset) values.

There are 15 countries with ongoing conflicts:

foo %>% filter(year==2016, is.na(incidence_civil_ns))

Afghanistan, Democratic Republic of Congo, Colombia, India, Iraq, Libya, Mali, Nigeria, Pakistan, Russia, Sudan, Somalia, Syria, Ukraine, Yemen

In the hand-coded alternative coding (6mo_make_confusion_matrix.do), the Civil War in Colombia is coded as having finished in 2015-H1, with no civil war for the 2016 forecast period. Turkey and Burundi however are coded as seeing new civil wars in 2016, i.e. new onsets.

The 2 false positives in the B&S confusion matrices are DRC and Libya.

So with the original coding, they hit 13 ongoing civil wars and miss 2 ongoing civil wars. With the alternative coding, they hit 12 ongoing, miss 2 ongoing, and hit the 2 new onsets.

Re-reading the text, it now sounds to me like they do kind of acknowledge that they want to take credit for predicting ongoing civil wars, e.g.:

Importantly, we make no assumptions about the persistence of civil war between the second half of 2015 and the first half of 2016, meaning that all countries—even those with ongoing conflicts as of December 31, 2015—are included in the 2016 test set. As a result, 14 of the 30 countries on the list (and nine of the top 10) are cases of ongoing conflict.

Context: in the data ongoing civil wars otherwise are NA's on the DV and thus drop out.

This is so weird.

1. At the time the forecasts were made, in early 2016-H1 (the pre-registration date is in March, I think), we would have known the civil war state for 2015-H2
2. We would not have known, yet, the civil war state for 2016-H1.
3. However, we are not scoring the forecasts until after 2016-H1 has concluded, when we can observe civil war state and determine whether a conflict was ongoing, terminated, or new civil war started.
4. Does it make sense to make forecasts for civil wars that are ongoing in 2015-H1?
   • We know the 2015-H1 civil war state, including presumably termination, so it would make sense to include cases that terminated and now are at risk of another, new civil war outbreak.
   • If the civil war coding was conflict specific, i.e. a country could have multiple distinct conflicts at any given, time, then it would make sense to not drop ongoing cases. But the Sambanis coding is at the country-level I think, so it doesn't. If you have ongoing conflict, no new onset is possible.
   • Aside from (i) above, I don't see how it makes sense to include 2015-H2 ongoing conflicts back into the data for 2016-H1. You'd have to assume that all those conflicts end in 2015-H2 and the countries are at risk again, but we know they didn't!
5. Anyways, by the time we are scoring the 2016-H1 forecasts, we know what the state of the world is.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/andybega/Blair-Sambanis-replication/issues/11__;!!OToaGQ!_nnUxMhmf7aTAFGeGfBnTEPVUVcoS1gG_UL9kE1k6xed2BzkeQEywPR9SQWIRpei$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAJRFBBBKIEBMCZ7BVQQ7Q3SHOYA7ANCNFSM4RYYVUQA__;!!OToaGQ!_nnUxMhmf7aTAFGeGfBnTEPVUVcoS1gG_UL9kE1k6xed2BzkeQEywPR9SWo9yMNq$. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/andybega/Blair-Sambanis-replication/issues/11#issuecomment-698655413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKKQHFM5ZSBE47CEYACRY3SHPQAJANCNFSM4RYYVUQA.