search

worldbank / ietoolkit

Stata commands designed for Impact Evaluations in particular, but also data work in general
https://worldbank.github.io/ietoolkit/
MIT License
214 stars 74 forks source link

Is it intended to push users towards using mean imputation? #260

Open Fstips opened 2 years ago

Fstips commented 2 years ago

I am not sure it is advisable to push users towards the balmiss() option. I receive this message when running iebaltab with ftest option.

"F-test is possible but perhaps not advisable. Some observations have missing values in some of the balance variables and therfore dropped from the f-stat regression. This happened in the f-tests for the following group(s): [(1)-(2)]. Solve this by manually restricting the balance table using if or in, or disable the f-test, or by using option balmiss(). Suppress this error message by using option fmissok."

But then the helpfile entry for balmiss() suggests the user not to use this option:

"balmiss(reptype) [...] WARNING: while technically possible it is far from certain this option generates a valid result. There is no guarantee that a missing value can correctly be assumed to be zero without making the interpretation of the balance table invalid. Best practice is most certainly to manually replace or omit these values manually before running this command. This option is only included to enable quick balance tables on raw data before the data has been cleaned."

Seems contradictory advice to me? I would suggest removing the last part of the message or at least not displaying as a red error. As it is right now, it really pushes users to specify ballmiss()

kbjarkefur commented 2 years ago

Thank you for this feedback!

We are doing a re-write of the iebaltab as it was originally written to have as many options as was added over the years and the code becaome too difficult to maintain. balmiss() is one of those options.

It was never our intention to push a user to anything. Do you really think it is contradictory? The error message lists different ways it may be possible address how to deal with the fact that some observations used in the pair-wise tests when testing across all variables, and the warning in the help file helps the user know that it is not always correct to use balmiss() and it is their responsibility to decide if it can be used.

We are displaying this as a red error as it is rarely correct to to have different samples across a table. If the f-test claim to test across all balance variables, but different sets of observations are used when testing for each of them compared to all of them, then a reader of this table really have to scrutinize this table in order what the numbers means in relation to each other. But I am sure there are cases where this is acceptable and that is why we have the option fmissok.

If a user explores the balmiss() we included that warning to make sure that the user understand that we by no means make the claim that balmiss() is an option that works in all contexts. While the option is an excellent shorthand when it fits the context, it is always the user's responsibility to understand their data and their analysis well enough to know when an option like this can be used on their data.

We hoped that this option would be useful to some users, but maybe it is too confusing and sounds too much like we are advising a user use this option. If that is the case, maybe we should not include an option like balmiss() and let the user either manually address this issue the the f-test or suppress the error with fmissok.

Fstips commented 2 years ago

Okay, maybe pushing is too far-fetched. But when reading this message balmiss() is the easy fix to the error. Maybe users could be warned that this is not advised? Also, to me it would make more sense to mention as fix an option like fullsample() which aligns the sample used for column means with the full F-test sample?

kbjarkefur commented 2 years ago

I like the fullsample() suggestion. Much better to restrict to the intersection of all samples used across the table. That excludes a lot of information but that is still better than make up information which is what balmiss() does when used incorrectly.

I will keep this issue open as we first need to complete the re-write of this command so that it is easier to maintain and add new options like fullsample() to. Then we will consider this suggestion together with other suggestions made by users like you and members of our team.

Thanks for the feedback!