Add new questions page to API

[NunoSempere]

Hey @berekuk, could you add a new questions page to the API? Not extremely urgent, but good to have in the next few days

[berekuk]

It's already available! All pages are implemented on top of GraphQL API.

Here's an example: https://metaforecast.org/api/graphql?query=fragment+Question+on+Question+%7B%0A++id%0A++url%0A++title%0A++description%0A++timestamp%0A++options+%7B%0A++++name%0A++++probability%0A++%7D%0A++platform+%7B%0A++++id%0A++++label%0A++%7D%0A++qualityIndicators+%7B%0A++++stars%0A++++numForecasts%0A++++numForecasters%0A++++volume%0A++++spread%0A++++sharesVolume%0A++++openInterest%0A++++liquidity%0A++++tradeVolume%0A++%7D%0A++visualization%0A%7D%0A%0Afragment+QuestionWithHistory+on+Question+%7B%0A++...Question%0A++history+%7B%0A++++timestamp%0A++++options+%7B%0A++++++name%0A++++++probability%0A++++%7D%0A++%7D%0A%7D%0A%0Aquery+QuestionPage%28%24id%3A+ID%21%29+%7B%0A++result%3A+question%28id%3A+%24id%29+%7B%0A++++...QuestionWithHistory%0A++%7D%0A%7D%0A

You'll have to enter variables in the bottom box, e.g. {"id": "rootclaim-did-pakistan-know-that-osama-bin-laden-was-hiding-in-abbottabad-18034"} (Yoga playground doesn't allow links with variables).

---

Here's how to trace the graphql call(s) for any metaforecast page:

• find a page component and its getServerSideProps; in this case, this one
• check the arguments for the client.query call; in this case, QuestionPageDocument
• QuestionPageDocument comes from queries.graphql.generated file, so you'll need a matching queries.graphql file
• queries.graphql file can include fragments; in this case, it includes QuestionWithHistory fragment, which is defined in src/web/fragments.graphql; you can find it with search, or, if you have your graphql plugin for vscode set up properly, you could just "go to definition"
• then you glue all these graphql bits together, pass variables, and it should give you the same data which is used on the webpage

[NunoSempere]

It's already available!

Are you sure? I specifically mean new questions added in the last 24h. And I'm not sure we have any pages which fetch that.

[berekuk]

Oh, I read this wrong! I thought you meant (new (question page)), the one with charts.

[berekuk]

Right now we store a timestamp field on Question and History (and on dashboards, but that's not relevant here).

This field usually means "last time we fetched/stored this question", except for guesstimate, for which we use the timestamp from the guesstimate API. But we can ignore the guesstimate case for this task, since it's used for search only, and this task is for querying the postgres database directly, which doesn't have guesstimate data yet.

Anyway.

I think we need more timestamp fields with different semantics.

1. fetched or lastFetched or maybe lastSeen (with semantics ~identical to current timestamp)
2. firstSeen (by metaforecast; this field is necessary for solving this issue)
3. created (on platform; this might be different from firstSeen since metaforecast can fetch the question with some delay)
4. updated (by platform; this one is less necessary and is too vague; while many platforms provide something like this, their semantics differ, and we could approximately restore this from our history data)
5. resolved (important for #10)

There are a lot of details to consider for each of these, but for now I'll just implement (1) and (2) and keep in mind (3) and (5) for the future database & API changes.

[berekuk]

Here: https://metaforecast.org/api/graphql?query=%7B%0A++questions%28first%3A+5%2C+orderBy%3A+FIRST_SEEN_DESC%29+%7B%0A++++edges+%7B%0A++++++cursor%0A++++++node+%7B%0A++++++++id%0A++++++++title%0A++++++++firstSeenStr%0A++++++%7D%0A++++%7D%0A++%7D%0A%7D

Notes:

• FIRST_SEEN_DESC is currently the only supported value for orderBy; if orderBy is not specified then ordering is arbitrary (I think it's ordered lexicographically by id, but I hope no one would rely on that because it can change in the future)
• new timestamp fields are fetched and firstSeen; I also added firstSeenStr and fetchedStr for convenience
  • unfortunately, I couldn't add field args to fetched, e.g. fetched(format: "iso"), since GraphQL requires any field to have a single type
  • but I intentionally named new fields nameStr instead of nameISO so that we could add other date string formats with args in the future
• fetched is present both in questions and in history, while firstSeen is present only in questions
• timestamp field is deprecated; I'm going to remove it from the database immediately and will remove it from the GraphQL schema soon
• I populated firstSeen fields with values based on history table

[NunoSempere]

Awesome!