feat: Search collator that works with new backend system

[Revyy]

The current collator is not possible(afaik) to use with the new Backend system.

Adding the collator in a module for the search plugin similarly like below gives errors relation questions does not exist when the collator performs its queries.

const searchModuleCustomExtensions = createBackendModule({
  pluginId: 'search', // name of the plugin that the module is targeting
  moduleId: 'customExtensions',
  register(env) {
    env.registerInit({
      deps: {
        //... and other dependencies as needed
        searchIndex: searchIndexRegistryExtensionPoint,
        loggerService: coreServices.loggerService,
        database: coreServices.database,
        // ... and other dependencies as needed
      },
      async init({
        // ... other dependencies as needed
        searchIndex,
        loggerService,
        database,
      }) {
        //... other setup code
        // collator gathers entities from questions.
        searchIndex.addCollator({
          schedule,
          factory: QetaCollatorFactory.fromConfig(config, {
            logger,
            database,
          }),
        });
      },
    });
  },
});

I think the issue is that when you extend the search plugin with a module to add a collator the collator get access to the database belonging to the search plugin, and therefore all queries will fail because the qeta tables do not exist in that database.

Instead, I think the collator needs to use an api provided by the backend to fetch questions.

I hacked together a solution just to verify that it kind of works. Something like the code below would be needed except the /questions endpoint does not play well with Service to Service authentication, so might need a new endpoint for this use case.

type QetaCollatorFactoryOptions = {
  logger: Logger;
  discovery: DiscoveryApi;
  tokenManager: TokenManager;
};

export class QetaCollatorFactory implements DocumentCollatorFactory {
  public readonly type: string = 'qeta';
  private readonly logger: Logger;
  private readonly discovery: DiscoveryApi;
  private readonly tokenManager: TokenManager;

  private constructor(_config: Config, options: QetaCollatorFactoryOptions) {
    this.logger = options.logger;
    this.discovery = options.discovery;
    this.tokenManager = options.tokenManager;
  }

  static fromConfig(config: Config, options: QetaCollatorFactoryOptions) {
    return new QetaCollatorFactory(config, options);
  }

  async getCollator() {
    return Readable.from(this.execute());
  }

  async *execute(): AsyncGenerator<QetaDocument> {
    this.logger.info('Executing QetaCollator');

    const baseUrl = await this.discovery.getBaseUrl('qeta');
    const { token } = await this.tokenManager.getToken();

    const response = await this.fetchApi.fetch(`${baseUrl}/questions`, {
      headers: {
        Authorization: `Bearer ${token}`,
      },
    });

    if (response.status !== 200) {
      this.logger.error(
        `Error while fetching questions: ${response.status} : ${response.statusText}`,
      );
      return;
    }

    const data = (await response.json()) as QuestionsResponseBody;

    if ('errors' in data) {
      this.logger.error(
        `Error while fetching questions from Qeta: ${JSON.stringify(
          data.errors,
        )}`,
      );
      return;
    }

    for (const question of data.questions) {
      yield {
        title: question.title,
        text: question.content,
        location: `/qeta/questions/${question.id}`,
        docType: 'qeta',
        author: question.author,
        score: question.score,
        answerCount: question.answersCount,
        views: question.views,
      };

      for (const answer of question.answers ?? []) {
        yield {
          title: `Answer for ${question.title}`,
          text: answer.content,
          location: `/qeta/questions/${question.id}`,
          docType: 'qeta',
          author: answer.author,
          score: answer.score,
        };
      }
    }
  }
}

interface CustomError {
  message: string;
}

interface ErrorResponse {
  errors: ErrorObject<string, any>[] | CustomError[] | null | undefined;
  type: 'query' | 'body';
}

interface QuestionsResponse {
  questions: Question[];
  total: number;
}

type QuestionsResponseBody = QuestionsResponse | ErrorResponse;

[Revyy]

Thoughts? I could give it a go if you like.

[drodil]

Hmm, it must be that the collator cannot access the qeta database as the database dependency in the createBackendModule is actually the search module database. So one should pass the PluginDatabaseManager that is also passed to the Q&A plugin.

Other option would be to use the Q&A API like you mentioned. Not sure how much extra delay that would add to the collator but maybe it's a good way to go forward. If you'd like to do a PR for this, it would be great. By the way, the types for the API probably should be moved from the frontend to the common plugin so that they could be utilized in this case as well.