Collator behaviour for missing spaces

Hi, I am setting up the plugin and I came across an issue during the execution of the scheduled task that queries the confluence spaces.

As per code, if a space is returning a non-ok response, the execution is terminated, and the relevant error is printed. I'd like to propose a change, where if a space returns a non-ok response, the execution continues, and the error generated from the bad space is logged.

My backstage instance produces this error:

2023-03-24T12:15:00.934Z search info exploring space DEVT type=plugin
2023-03-24T12:15:01.654Z search info exploring space MORP type=plugin
2023-03-24T12:15:02.080Z search warn non-ok response from confluence type=plugin
2023-03-24T12:15:02.081Z search error Collating documents for confluence failed: Error: Request failed with 404 Not Found type=plugin documentType=confluence
2023-03-24T12:15:02.081Z backstage error Request failed with 404 Not Found type=taskManager task=search_index_confluence stack=Error: Request failed with 404 Not Found
    at ConfluenceCollatorFactory.get (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:146:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ConfluenceCollatorFactory.getDocumentsFromSpace (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:93:20)
    at async ConfluenceCollatorFactory.getDocumentsFromSpaces (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:83:29)
    at async ConfluenceCollatorFactory.execute (/workspace/node_modules/@k-phoen/backstage-plugin-confluence-backend/dist/index.cjs.js:40:27)
    at async next (node:internal/streams/from:85:11)
(node:922) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)

I have 3 spaces configured:

spaces: ['DEVT', 'MORP', 'REL']

DEVT and REL are valid, MORP doesn't exist. As you can see from the logs, MORP triggers the error, and as the execution terminates REL is never explored, and ultimately pages from DEVT and REL never get indexed.

The second error, more problematic for us, appeared after a few failed executions of the scheduled task:

2023-03-24T12:19:48.887Z backstage error Request failed with status 500 There was a problem performing the search query: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? type=errorHandler stack=Error: There was a problem performing the search query: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at /workspace/node_modules/@backstage/plugin-search-backend/dist/index.cjs.js:258:15
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)

It appears that the broken collation leaves the index in a bad state, causing the search query to fail for all results types. and the search page shows infinite loading in the UI : loading

Looking at the code from ``,

private async getDocumentsFromSpace(space: string): Promise<string[]> {
        const documentsList = [];

        this.logger.info(`exploring space ${space}`);

        let next = true;
        let requestUrl = `${this.wikiUrl}/rest/api/content?limit=1000&status=current&spaceKey=${space}`;
        while (next) {
            const data = await this.get<ConfluenceDocumentList>(requestUrl);
            if (!data.results) {
                break;
            }

            documentsList.push(...data.results.map(result => result._links.self));

            if (data._links.next) {
                requestUrl = `${this.wikiUrl}${data._links.next}`;
            } else {
                next = false;
            }
        }

        return documentsList;
    }

it might be sufficient to change the logic to something like continue instead of break when a bad space is explored:

...
while (next) {
            const data = await this.get<ConfluenceDocumentList>(requestUrl);
            if (!data.results) {
                continue;
            }

            documentsList.push(...data.results.map(result => result._links.self));
...

K-Phoen / backstage-plugin-confluence

Collator behaviour for missing spaces #176