search

quantified-uncertainty / metaforecast

Fetch forecasts from prediction markets/forecasting platforms to make them searchable. Integrate these forecasts into other services.
https://metaforecast.org/
MIT License
56 stars 5 forks source link

Smarkets: missing contracts #80

Open berekuk opened 2 years ago

berekuk commented 2 years ago

In the process of doing #79 I slightly changed the logic of building options and added an explicit check for whether there's a contract in https://api.smarkets.com/v3/markets/ID/contracts/ list matching an id from https://api.smarkets.com/v3/markets/ID/last_executed_prices/ list.

Turned out a missing contract situation can happen.

Example:

Two contracts, 32605732 and 32605733, are missing. And they have a non-zero price!

They are also missing from the web page: https://smarkets.com/event/34757345. Probabilities on it doesn't add up to 100%.

(there are ~30 other cases like this)

@NunoSempere, we should still normalize by total price, including missing contracts, even if it doesn't add up to 100%, to match the smarkets values, right? Also, do you by any chance know what's going on here?

NunoSempere commented 2 years ago

Not really sure, I'll give a heads up to the smarkets people.

NunoSempere commented 2 years ago

we should still normalize by total price, including missing contracts, even if it doesn't add up to 100%...

So this is a bit tricky; we could maybe also save the un-normalized values in the history as a matter of general policy? It might be useful for research later on.

MatthewShaddick commented 2 years ago

Hi - the two missing contracts you highlighted are both ones that have now lost (although only one of them had been classified as a loser; now corrected).

MatthewShaddick commented 2 years ago

A user had left an order up for one of them which could have generated a non-zero probability, depending on how you derive the percentages (last traded price or midpoint of buy and sell price currently available?)

berekuk commented 2 years ago

@MatthewShaddick, thanks! I was able to retrieve the missing contracts with include_hidden=true param.

berekuk commented 2 years ago

So this is a bit tricky; we could maybe also save the un-normalized values in the history as a matter of general policy?

As a matter of general policy, I hope that we'll eventually be able to save everything we fetch from platforms, i.e., raw html and json (forever or for a long time). That'll also allow us to separate fetching and processing steps, for quicker development and sometimes for fixing parser issues when those arise.

But that's a different task I had in mind for a while, I'll explain it in a separate issue some time later.

For now, I just dumped prices and contracts into extra (but this is sub-optimal for performance, considering that Prisma selects all fields from the database by default).

berekuk commented 2 years ago

On normalizing: I'm still unsure about the best approach, since I'm not sure I completely understand smarkets price-setting logic.

Contracts don't have to cover the entire set of possibilities, right? They are mutually exclusive (I think? or maybe not?), but it could be possible that not all possible outcomes are listed.

In this case, should we ever normalize? Why was the normalization code there in the first place? @NunoSempere

MatthewShaddick commented 2 years ago

No, for many markets we don't have every possible outcome (e.g. 2024 US Presidential election). Obviously there are many others where we do and a lot of binary yes/no markets.

berekuk commented 2 years ago

Ok, a few cases.

1) https://metaforecast.org/questions/smarkets-12496911, https://smarkets.com/event/42048965/politics/us/2022-house-and-senate-elections/2022-house-and-senate-control

Total on smarkets is above 100%; seems like a common situation since the market is not large enough for arbitrage, I assume, but it's not clear that normalized values are better than original ones, since the original ones are closer to the actual beliefs of betters.

2) https://metaforecast.org/questions/smarkets-15544753, https://smarkets.com/event/42282284/politics/europe/ireland/taoiseach-after-next-general-election

Tiny market with just £1 traded for a single option. Metaforecast is clearly wrong here, it normalizes 55.56% to 100%.

3) https://metaforecast.org/questions/smarkets-22993480, https://smarkets.com/event/42667456/politics/europe/2025/01/01/00-00/russia/2022/06/30/12-00/putin-and-zelenskyy-to-meet-before-july-1st

Binary question for which smarkets again has a slightly above 100% total. It's not a big deal in this case, since it's pretty close, but maybe it'd be better to just display two lines, "yes" and "no", like smarkets does. People are allowed to have inconsistent beliefs. But in this case it's not clear to me whether we should normalize or not.