search

long2ice / meilisync

Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch
https://github.com/long2ice/meilisync
Apache License 2.0
247 stars 37 forks source link

Postgres timestamp column causes sync to fail #28

Open brandantck opened 10 months ago

brandantck commented 10 months ago

Postgres: v15

config.yml

progress: 
  type: file

I am trying to sync a table from a Postgres database with the column type:timestamp, and its throwing this error. image

This is an example of a row in the database image

Seems like the error occurs when its trying to do a json.dumps into the progress.json file.

Edit: Tried with redis but still getting the same error, issue might not lie with writing to progress

sanders41 commented 10 months ago

datetime is not json serializable so that is most likely causing the issue. I'm not sure the full context here so this my not be possible but if you can convert your dates to an iso string before dumping it should fix the issue.

An example of how to convert:

condition_start_datetime = datetime.datetime(2010, 4, 1, 0, 0)
converted = condition_start_datetime.isoformat()
long2ice commented 10 months ago

Maybe you can use plugins

brandantck commented 10 months ago

When i run the meilisync refresh command, it doesnt seem to run the plugin pre_event or post_event functions

long2ice commented 10 months ago

Yes, that's a problem...

brandantck commented 10 months ago

Ah okay, so currently, the plugins are only loaded and used on the meilisync start command?

long2ice commented 9 months ago

You can try latest code now

sokhuong-uon commented 1 week ago

Hello the postgres timestamp with time zone got converted to unix format for initial data. but the format is corrected after I update the date in the database. So the timestamp is only work correctly for newly inserted/updated data.

I use long2ice/meilisync:dev docker image.

For initial data image

After update image Note: I only update the date of the second item.

Is there any new settings need to be set?