Search may be broken?

[dpsanders]

Hi,

Thanks again for a great site!

As of today (1 Jan 2021), searching for "April 2020" on downforacross.com returns only 5 puzzles. While this may be correct, it also seems too low.

I get similar results with other older searches.

[stevenhao]

Yeah, I'm aware of this; it's a direct result of https://github.com/downforacross/downforacross.com/pull/135 which limits the puzzles searched to the 4000 most recently uploaded. Unfortunately that means that search results may be incomplete.

This is temporary as I migrate the puzzle repo off of the previous system, which was just storing the jsons directly in firebase. The new system will store puzzles in a postgres table, which will support more advanced and efficient filtering done via query / the backend.

[dpsanders]

OK great, thanks!

[ewmson]

@stevenhao what is the status on moving to postgres? If it just needs more work on db schema and copying the existing firebase puzzles, I can take a stab at it and ensure that we have efficient enough indexes to do the standard searches (to unblock this issue) as well as potentially other fields in the content json (like #123 asks for).

[stevenhao]

@ewmson thanks for the reminder on this!

Here's a progress update: As of #130 and the follow-up fixes (https://github.com/downforacross/downforacross.com/commit/2736cf3c6b8921e1d04af4231bca20ff7381efeb and https://github.com/downforacross/downforacross.com/commit/8c9b74f8dc6f9a49293f5dea5c137eb950d054f6), new uploaded puzzles are mirrored to postgres. Let me DM you the credentials for the staging user & db on postgres (note: i don't know best practices around sharing these creds -- anyone else who wants to work with the backend, feel free to DM me on discord/messenger and I'll gladly share)

The same PR also adds a new frontend component to act as a drop-in replacement for the PuzzleList component rendered on the homepage. That component is feature-flagged and can be turned on via localStorage:

localStorage.newPuzzleListEnabled = 1 // turn it on
delete localStorage.newPuzzleListEnabled // turn it off

[stevenhao]

I think the remaining work for migrating search over completely is roughly:

1) Backfill the existing puzzle list from firebase to postgres 2) Fully migrate to the new puzzle list in frontend 3) Set up backend search endpoint in the backend and the appropriate indices 4) Hit the new endpoint in the frontend, appropriately mapping filter & page state to query parameters

Feel free to get started on [3] and [4]. I'll handle [1] and [2] since the backfill is easiest to do using a firebase service account / writing directly to prod postgres.

[stevenhao]

And yeah, with regards to #123, feel free to go nuts with new filter options! Feel free to add new columns and such to the puzzles schema as needed.