Invalid Questions

[zmtomorrow]

Hi, thanks for developing this benchmark.

I've noticed that some forecasting questions are not meaningful without the context of the current time.

For example:

• 'Is growth in J&J's adjusted EPS expected to accelerate in FY2023?'
• 'What production rate changes is Boeing forecasting for FY2023?'

A meaningful question should include the time when the query was raised, e.g.,

• 'Is growth in J&J's adjusted EPS expected to accelerate in FY2023 from 2022?'

However, these types of questions are less frequent in practical scenarios.

In practice, people always want to predict the future based on their current time. For example, once the 10-K for FY2023 is released, the question 'Is growth in J&J's adjusted EPS expected to accelerate in FY2023?' becomes meaningless and can decrease the accuracy measurement of the QA system.

I am looking forward to your thoughts on this potential issue.

Thanks Mingtian

[zmtomorrow]

I also guess this question is not meaningful without further context.

"Were there any board member nominees who had substantially more votes against joining than the other nominees?"

[zmtomorrow]

Similarly, this question and the provided answer are not valid without time condition provided in the question:

Question: Are Best Buy's gross margins historically consistent (not fluctuating more than roughly 2% each year)? If gross margins are not a relevant metric for a company like this, then please state that and explain why.

Provided answer: Yes, the margins have been consistent, there has been a minor decline of 1.1% in gross margins between FY2022 and FY2023.