What does the below hyperparameters indicates?

[bhavikapanara]

Hi @tomhosking

can you please elaborate what does the below hyperparameter indicates?

tf.app.flags.DEFINE_integer("filter_window_size_before", 1, "Filter contexts down to the sentences around the answer. Set -1 to disable filtering")
tf.app.flags.DEFINE_integer("filter_window_size_after", 1, "Filter contexts down to the sentences around the answer. Set -1 to disable filtering")
tf.app.flags.DEFINE_integer("filter_max_tokens", 100, "Filter contexts down to at most this many tokens around the answer. Set -1 to disable filtering")

Thanks, Bhavika

[tomhosking]

These parameters control cropping of the context - for a combination of speed and model reasons, the system only uses the sentence containing the answer plus the sentences adjacent (so 3 sentences in total. This is controlled by the filter_window_size_before/after parameters. The system also only includes 100 tokens before and after the answer span to remove weirdly odd sentences (there are some outliers in squad that are 600+ tokens long). This number of tokens is controlled by filter_max_tokens.

Hope that helps - looking back I realise I gave these parameters really unhelpful names!

[bhavikapanara]

Thanks, @tomhosking