Serialization error when importing from MongoDB

[alallema]

Description

I tried to import documents with meilisync from the MongoDB free plan: https://cloud.mongodb.com/ and got a serialization error: TypeError: Object of type ObjectId is not JSON serializable

Data format:

I used the sample data proposed by MongoDB which looked like:

{
  "_id": { "$oid": "64c91330c2a74205feef694b" },
  "account_id": { "$numberInt": "278603" },
  "limit": { "$numberInt": "10000" },
  "products": ["Commodity", "InvestmentStock"]
}

Configuration file

debug: true
plugins:
  - meilisync.plugin.Plugin
progress:
  type: file
source:
  type: mongo
  host: mongodb+srv://cluster_IP.mongodb.net
  port: 27017
  username: userName
  password: passWord
  database: sample_analytics
meilisearch:
  api_url: http://localhost:7700/
  api_key: 'masterKey'
  insert_size: 100
  insert_interval: 10
sync:
  - table: accounts
    index: accounts
    full: true

Screenshots & Logs:

I put my logs in a file just in case: output_error.txt

Note:

I tried to remove full: true by:

    fields:
      account_id:
      limit:
      products:

But I received another error that I will describe in another issue.

[sanders41]

Hi @alallema :wave: . Did you prepare the data from Mongo to send or did Meilisync do it itself (sorry, I'm only familiar with meilisync at a very high level). The issue here is the ObjectId needs to be converted to a string before it can be serialized to JSON. I can make some suggestions how to do that once I know where the data came from.

[alallema]

Hi @sanders41, Thank you for replying 😊. No, I haven't prepared anything. I've used the samples proposed by MongoDb to populate my database, and it seems that all _id fields are created in this format by default on MongoDb Cloud. I've also created a collection using the movie dataset and I've got another error described here #17

[long2ice]

fixed