search

UVicMartletplace / martletplace

Recreating FB Marketplace for Students - UVic SENG499 Capstone Project
1 stars 0 forks source link

Gavin/212 add pgsync to keep elasticsearch up to date with postgres #214

Closed GDeane closed 2 days ago

GDeane commented 3 days ago

Description

I want to use PGSync to keep elasticsearch in sync with Postgres (reading from postgres and writing to elasticsearch).

I was working in the martletplace repository on the search endpoint and I have realized a problem after chatting with Max.

It is very easy for the elasticsearch index which search depends upon to get out of sync with the Postgres database (the source of truth).

Originally, I had planned to forward Listings updates from the backend to the search endpoint using the /api/search/reindex/listing-created endpoint (and two others, which are present in the search endpoint)

Unfortunately, this has issues with staying in sync particularly when either service can potentially go down.

Closes #212

How to Test

  1. Run docker-compose up -- build

  2. In a browser, paste http://localhost/api/search?query=chocolate&minPrice=5&maxPrice=10&status=AVAILABLE&searchType=LISTINGS&latitude=48.4633&longitude=-123.3108&sort=RELEVANCE&page=1&limit=6 expected output: image

  3. In a browser, paste http://localhost/api/search?query=user&status=SOLD&searchType=USERS&latitude=48.4633&longitude=-123.3108&sort=RELEVANCE&page=1&limit=6 expected output: image

Checklist

GDeane commented 2 days ago

I'm running into the problem where elasticsearch requires location fields to be mapped as geo_point with the fields "lat" and "lon" in order to search by location (https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html).

previously, (when using reindexing API) location fields looked like this: "location": { "lat": 45.4215, "lon": -75.6972 },

Now, they look like this after using PGSync: "location": { "latitude": 40.7128, "longitude": -74.006 },

When I search by location, I currently get this error: image.

In short, if location can't be changed to "lat" and "lon" instead of "latitude" and "longitude" using the PGSync schema (or some other method), we won't be able to search within 5 km of or sort by location.

Meanwhile, PGSync has the issue where it does not recognize the LOCATION_TYPE defined in PGSync image

I have not been able to figure out how to resolve this.