Meilisearch indexes are not creating when product created

[harshajayaweeraXHJ]

Meilisearch indexes are not creating when product is created.

Meilisearch indexes are not creating when product created. It was working fine before and few days ago I changed the meilisearch domain name to a new one (realized previous one was expired). New domain name is working and frontend connects with the meilisearch instance on AWS successfully when make a query.

But when product created from the admin side it doesn't create a new index. I monitored journalctl logs for meilisearch instance and no activity when creating a product. seems like it never hits the server. And no error on medusa backend as well

Medusa is hosted on Heroku

Also I checked the Nginx logs for meilisearch. No hits

[olivermrbl]

From what I can tell, you haven't changed the configuration of your Meilisearch search plugin in medusa-config.js. This would explain why your frontend is requesting the correct instance, but your backend is not.

[harshajayaweeraXHJ]

@olivermrbl First of all thanks for the prompt response. Actually I have configured everything in the backend config. This is my current configuration

{
    resolve: `medusa-plugin-meilisearch`,
    options: {
      // config object passed when creating an instance of the MeiliSearch client
      config: {
        host: process.env.MEILISEARCH_HOST,
        apiKey: process.env.MEILISEARCH_API_KEY,
      },
      settings: {
        // index name
        products: {
          // MeiliSearch's setting options to be set on a particular index
          searchableAttributes: ["title", "description", "variant_sku"],
          displayedAttributes: ["title", "description", "variant_sku", "thumbnail", "handle"],
        },
      },
    },
  },

Also I have set up the environment variables correctly. Weird thing is it used to work before without any problems. Then this url expiration thing happened and suddenly stopped working. Then after I mapped it to the new domain only frontend is working.

Also I did update the backend and frontend env's with new url

[olivermrbl]

Did you try to trigger the search indexing manually by inserting a SEARCH_INDEX_EVENT into the staged job table?

This will re-index all products and create indexes in your Meilisearch instance if they do not already exist.

You can do that like so:

insert into staged_job (id, event_name, data) values ('some_id', 'SEARCH_INDEX_EVENT', '{}')

[harshajayaweeraXHJ]

I tried above query on Heroku database. This is the result. posted screenshot so it's easy to read. I'll add the actual log content below as well

2023-05-01T16:14:00.887141+00:00 app[web.1]: {"level":"info","message":"Processing SEARCH_INDEX_EVENT which has 1 subscribers","timestamp":"2023-05-01 16:14:00"} 2023-05-01T16:14:00.951544+00:00 app[web.1]: {"level":"warn","message":"This is an empty method: addDocuments must be overridden by a child class","timestamp":"2023-05-01 16:14:00"} 2023-05-01T16:14:20.000000+00:00 app[postgres.2547061]: [PINK] [12-1] sql_error_code = 0A000 time_ms = "2023-05-01 16:14:20.601 UTC" pid="323936" proc_start_time="2023-05-01 16:14:20 UTC" session_id="644fe55c.4f160" vtid="" tid="0" log_line="2" database="[unknown]" connection_source="64.62.197.47(46417)" user="[unknown]" application_name="[unknown]" FATAL: unsupported frontend protocol 0.0: server supports 3.0 to 3.0 2023-05-01T16:14:20.000000+00:00 app[postgres.2547062]: [PINK] [12-1] sql_error_code = 0A000 time_ms = "2023-05-01 16:14:20.737 UTC" pid="323937" proc_start_time="2023-05-01 16:14:20 UTC" session_id="644fe55c.4f161" vtid="" tid="0" log_line="2" database="[unknown]" connection_source="64.62.197.60(54175)" user="[unknown]" application_name="[unknown]" FATAL: unsupported frontend protocol 255.255: server supports 3.0 to 3.0 2023-05-01T16:14:20.000000+00:00 app[postgres.2547063]: [PINK] [12-1] sql_error_code = 28000 time_ms = "2023-05-01 16:14:20.874 UTC" pid="323938" proc_start_time="2023-05-01 16:14:20 UTC" session_id="644fe55c.4f162" vtid="" tid="0" log_line="2" database="[unknown]" connection_source="64.62.197.57(60537)" user="[unknown]" application_name="[unknown]" FATAL: no PostgreSQL user name specified in startup packet 2023-05-01T16:13:50.000000+00:00 app[heroku-postgres]: source=HEROKU_POSTGRESQL_PINK addon=postgresql-regular-07016 sample#current_transaction=4950 sample#db_size=18195235bytes sample#tables=99 sample#active-connections=15 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99831 sample#table-cache-hit-rate=0.99988 sample#load-avg-1m=0 sample#load-avg-5m=0.015 sample#load-avg-15m=0 sample#read-iops=0 sample#write-iops=0.10976 sample#tmp-disk-used=543600640 sample#tmp-disk-available=72435191808 sample#memory-total=3956720kB sample#memory-free=193800kB sample#memory-cached=3210456kB sample#memory-postgres=30484kB sample#wal-percentage-used=0.06402315918968281

BTW if it helps my medusa and meilisearch versions as below "@medusajs/medusa": "^1.6.0" "medusa-plugin-meilisearch": "^1.0.4"

Also when I restart Heroku server I see following log. Even though I have already set up Meilisearch in backend configurations as I have shared in above message

2023-05-01T18:51:31.172636+00:00 app[web.1]: {"level":"warn","message":"No search engine provider was found: make sure to include a search plugin to enable searching","timestamp":"2023-05-01 18:51:31"}

[olivermrbl]

It looks like your Meilisearch plugin is not being installed. Will dive a bit into this one.

Please make sure to update the plugin configuration to fit the updated shape, as explained in the upgrade guide. Not entirely sure if this is the root cause, but it will eventually also lead to issues.

[harshajayaweeraXHJ]

@olivermrbl I updated plugin configurations to updated shape as you advised and updated medusa-plugin-meilisearch npm package as well. This is the updated configurations now

{
    resolve: `medusa-plugin-meilisearch`,
    options: {
      // config object passed when creating an instance of the MeiliSearch client
      config: {
        host: process.env.MEILISEARCH_HOST,
        apiKey: process.env.MEILISEARCH_API_KEY,
      },
      settings: {
        // index name
        products: {
          indexSettings: {
            // MeiliSearch's setting options to be set on a particular index
            searchableAttributes: ["title", "description", "variant_sku"],
            displayedAttributes: ["title", "description", "variant_sku", "thumbnail", "handle"],
          },
        },
      },
    },
  }

But now I'm getting following error. Seems like medusa doesn't recognize this new structure

class_1 [MeiliSearchApiError]: Json deserialize error: unknown field indexSettings, expected one of displayedAttributes, searchableAttributes, filterableAttributes, sortableAttributes, rankingRules, stopWords, synonyms, distinctAttribute, typoTolerance, faceting, pagination at line 1 column 16

[harshajayaweeraXHJ]

@olivermrbl Any idea on how to solve this please

[olivermrbl]

Json deserialize error: unknown field indexSetting

This tells me that the plugin has not been updated to the latest version, since indexSettings is being passed directly to the Meilisearch client instead of its content. Can I get you to ensure this is the case?

yarn add medusa-plugin-meilisearch@latest

[harshajayaweeraXHJ]

After above yarn command, error disappeared. but sill not updating indexes. I have created a new product from admin to check if it triggers the indexing. This is the result. still does not index

[olivermrbl]

Hmm, this is very strange. It looks like your subscribers are not getting registered.

I just did a fresh install and I see the following server log on creating a product:

info:    Processing product.created which has 1 subscribers

Are you running locally?

[LarryStanley]

@harshajayaweeraXHJ I have the same problem. After upgrading all relative package (@medusajs/types, @medusajs/utils). I have solve the problem.

[harshajayaweeraXHJ]

@LarryStanley Just tried updating related packages. Still doesn't work. Any guess what could be the issue. is any issue with event bus or something. does indexing trigger based on events

[harshajayaweeraXHJ]

Hmm, this is very strange. It looks like your subscribers are not getting registered.

I just did a fresh install and I see the following server log on creating a product:

info:    Processing product.created which has 1 subscribers

Are you running locally?

No this is in production. on Heroku

[harshajayaweeraXHJ]

@olivermrbl If plugin is installed, configured and versions are up to date what could be the reasons that it's not subscribing to the event

[harshajayaweeraXHJ]

BTW not sure if this helps but I have created a custom subscriber for invite.created, order.payment_captured, order.placed events. something like below

class InviteSubscriber {
    constructor({ eventBusService, orderService }) {
        this.eventBusService_ = eventBusService;
        this.orderService = orderService;
        this.eventBusService_.subscribe("invite.created", this.handleInvite);
        this.eventBusService_.subscribe("order.payment_captured", this.handleOrderPaymentCapture);
        this.eventBusService_.subscribe("order.placed", this.handleOrderPlaced);

        .......

Could that cause any issue. So the issue here now seems to be that Meilisearch plugin does not subscribe for product created and related events if I am not mistaken

[olivermrbl]

@harshajayaweeraXHJ – can I get you to paste the entire subscriber?

[harshajayaweeraXHJ]

@olivermrbl sure, here it is

import logger from "heroku-logger";
import SendGrid from "@sendgrid/mail"
import {INVOICE_TEMPLATE, PAYMENT_CAPTURED_STAMP} from "../templates/invoice-template";
import { degrees, PDFDocument, StandardFonts, rgb } from 'pdf-lib'
import axios from "axios"

class InviteSubscriber {
    constructor({ eventBusService, orderService }) {
        this.eventBusService_ = eventBusService;
        this.orderService = orderService;
        this.eventBusService_.subscribe("invite.created", this.handleInvite);
        this.eventBusService_.subscribe("order.payment_captured", this.handleOrderPaymentCapture);
        this.eventBusService_.subscribe("order.placed", this.handleOrderPlaced);

        SendGrid.setApiKey(process.env.SENDGRID_API_KEY)
    }

    sendInvoicePdf = async (invoice_data, data) => {
        // ... invoice sending logic here...
        const response = await SendGrid.send(sendOptions)
        logger.info("invoice sent ", response)
    }
    sendInvoicePdfCaptured = async (invoice_data, data) => {
        const pdf_base64 =  await this.generateInvoice(invoice_data, true);

        // pdf sending logic here...
        const response = await SendGrid.send(sendOptions)
        logger.info("payment captured_invoice emailed ", response)
    }

    generateInvoice = async (invoice_data, captured) => {
        const invoicePdf = await this.generateinvoicePDF(invoice_data, captured)
        return invoicePdf
    }

    generateinvoicePDF = async (data, captured) => {
        const base64 = INVOICE_TEMPLATE;
        // pdf generation logic here
        return fileContent

    }

    // event handlers

    handleOrderPaymentCapture = async (data) => {

        logger.info('Starting order.payment_captured', data)
        const sendData = await this.orderService.retrieve(data.id, {
            relations: [
                "customer",
                "billing_address",
                "shipping_address",
                "discounts",
                "discounts.rule",
                "shipping_methods",
                "shipping_methods.shipping_option",
                "payments",
                "fulfillments",
                "fulfillments.tracking_links",
                "returns",
                "gift_cards",
                "gift_card_transactions",
                "items",
            ],
        })

        // some logic here...    

        await this.sendInvoicePdfCaptured(invoice_data, { email: sendData?.customer?.email}, true)
    }

    handleOrderPlaced = async (data) => {

        const sendData = await this.orderService.retrieve(data.id, {
            relations: [
                "customer",
                "billing_address",
                "shipping_address",
                "discounts",
                "discounts.rule",
                "shipping_methods",
                "shipping_methods.shipping_option",
                "payments",
                "fulfillments",
                "fulfillments.tracking_links",
                "returns",
                "gift_cards",
                "gift_card_transactions",
                "items",
            ],
        })
        // some logic here

        await this.sendInvoicePdf(invoice_data, { email: sendData?.customer?.email}, false)
    }

    handleInvite = async (data) => {

        const sendOptions = {
            template_id: process.env.SENDGRID_ADMIN_INVITE_TEMPLATE,
            from: {
                email: process.env.SENDGRID_FROM,
                name: "name here",
            },
            to: data.user_email,
            dynamic_template_data: {
                "invitation_url": `https://somebaseurl.com/invite?token=${data.token}`
            }
        }
        const response = await SendGrid.send(sendOptions)
    };
}

export default InviteSubscriber;

[harshajayaweeraXHJ]

@olivermrbl Any thoughts?

[olivermrbl]

@harshajayaweeraXHJ – I believe I might have found the culprit in a tricky case with an incorrect peer dependency installing.

Can I get you to bump ALL Medusa packages to their latest version? That should resolve the issue.

[harshajayaweeraXHJ]

I did update the medusa packages to their latest versions. And also updated storefront to their latest version as well per this guild https://docs.medusajs.com/upgrade-guides/medusa-react/4-0-2

One thing I noticed is current latest version is 7.0.0 and this documentation is not up to date about it. Issue is when I updated the storefront and build it I keep getting Types of property .. are incompatible errors everywhere. How can I update storefront and admin apps accordingly without breaking existing apps. is there any migration guide or steps which correctly upgrade it

[harshajayaweeraXHJ]

@harshajayaweeraXHJ – I believe I might have found the culprit in a tricky case with an incorrect peer dependency installing.

Can I get you to bump ALL Medusa packages to their latest version? That should resolve the issue.

Updating dependencies fixed the issue. Thanks