undefined in "searchResults" but present in "rawResults"

[florianmatz]

Hi Luca,

me again^^

I have the following, quite strange behaviour.

You can add entries to the list of elements, that can be searched with mini-search. While you add the new entry, the query stays active.

The result should be, if the new entry matches the query, it should be displayed.

So, if the input data changes I perform search via a useEffect hook and display the new data.

But what happens is that despite having a match in the raw results (correct field, correct match, everything ok), the entry in searchResult is undefined causing a crash.

   useEffect(() => {
        if (data && !isFirstRender.current) {
            removeAll();
            addAll(data);
        }

        isFirstRender.current = false;

    }, [data]);

    useEffect(() => {
        if (data) {
            search(filter.query, {
                filter: filterOptions.categoryField && filter.categories.length > 0 ? matchCategory : undefined,
            });
        }
    }, [data, filter, filterOptions, matchCategory]);

The order of execution is correct, double checked on that.

Result looks like:

rawResults

[
    {
        "id": "6a901953144c411580520ed07B4567",
        "terms": [
            "einholung"
        ],
        "score": 8.600445452294665,
        "match": {
            "einholung": [
                "custom.bezeichnung"
            ]
        },
        "custom.kategorien": [
            "Gefahr in Verzug",
            "Sicherheit",
            "Qualität"
        ]
    },
    {
        "id": "h1662s",
        "terms": [
            "einholungsbums"
        ],
        "score": 4.152082359120152,
        "match": {
            "einholungsbums": [
                "custom.bezeichnung"
            ]
        },
        "custom.kategorien": [
            "Gefahr in Verzug"
        ]
    }
]

[
    {
        "id": "6a901953144c411580520ed07B4567",
        "datum": "2020-09-17T20:34:56.170914Z",
        "custom": {
            "id": "6a901953144c411580520ed07a21ef39",
            "erstelldatum": "2020-09-17T20:34:56.170914Z",
            "baumassnahmenId": "45042621",
            "bezeichnung": "Einholung weiterer Informationen",
            "kategorien": [
                "Gefahr in Verzug",
                "Sicherheit",
                "Qualität"
            ],
    .....
    },
    undefined -> Where the second element should be.
]

Help would be highly appreciated. I'm kind of confused...

[lucaong]

Hi @florianmatz , I assume that the issue is with react-minisearch rather than with MiniSearch itself. That said, I will have a look into it.

One way this could happen, is if the raw result is not found in the internal mapping of ID to document. It might be a bug in react-minisearch, but can I see how you initialize it?

[florianmatz]

Hi Luca, I'll drop you the complete code - without Interfaces and render stuff

/ Configs & Defaults
// ----------------

let delayTimer;
const defaultFilter = { query: '', categories: [] };
const defaultPaginationSettings = { mode: 'by-pages', itemsPerPage: 10, showDividerAbovePagination: false };

const defaultFilterOptions: Omit<FilterOptions, 'fields'> = {
    extractField: (document: any, fieldName: string) =>
        fieldName.split('.').reduce((doc, key) => doc && doc[key], document),
    searchOptions: {
        prefix: (term: string) => term.length > 2,
        fuzzy: (term: string) => (term.length > 2 ? 0.3 : null),
    },
};

const defaultInputSettings = {
    placeholder: '',
    queryMinLength: 2,
    throttleTypingInterval: 300,
};

// The component itself
// ----------------

const NbFilterable: FC<NbFilterableProps> = ({
    CategorySwitch,
    data,
    headerStyles,
    initialFilter,
    inputSettings: inputSettingsProp,
    minHeight,
    mode,
    pagination,
    paginationSettings: paginationSettingsProp,
    filterOptions,
    title,
    ...props
}: NbFilterableProps): ReactElement => {
    const { tableSettings } = props as TableModeProps;
    const { listSettings } = props as VirtualizedListModeProps | StandardListModeProps;
    const { getScrollToItem } = props as VirtualizedListModeProps;
    const isFirstRender = useRef(true);

    // Using prop as initial state in full awareness
    const [filter, setFilter] = useState<Filter>({ ...defaultFilter, ...initialFilter });
    const [page, setPage] = useState(0);
    const [categoryLoading, setCategoryLoading] = useState<string>(null);
    const [scrolledToActive, setScrolledToActive] = useState<boolean>(false);
    const [isTyping, setIsTyping] = useState<boolean>(false);
    const listRef: RefObject<any> = useRef();
    const inputRef: RefObject<HTMLDivElement> = useRef();

    // Variables for the fuzzy search performed by MiniSearch
    // ----------------
    const { categoryField, sortResultsCompare, ...restFilterOptions } = filterOptions;

    const miniSearchOptions: MiniSearchOptions = {
        ...defaultFilterOptions,
        ...restFilterOptions,
        storeFields: categoryField ? [categoryField] : [],
        searchOptions: { ...defaultFilterOptions.searchOptions, ...(filterOptions.searchOptions || {}) },
    };

    const { search, searchResults, removeAll, addAll, rawResults } = useMiniSearch(data, miniSearchOptions);

    const paginationSettings = {
        ...defaultPaginationSettings,
        ...paginationSettingsProp,
    };

    const inputSettings = {
        ...defaultInputSettings,
        ...inputSettingsProp,
    };

    // Helper Methods
    // ----------------

    const memoizedSetCategoryLoading = useCallback(() => setCategoryLoading, [setCategoryLoading]);

    const resetInput = () => {
        const input: HTMLInputElement = inputRef.current.querySelector('.MuiInputBase-input');
        input.value = '';
    };

    const updateFilter = useCallback(
        (newFilter: IFilter) => {
            setFilter(prevState => ({
                ...prevState,
                ...newFilter,
            }));
            setScrolledToActive(false);
            if (pagination) {
                setPage(0);
            }
        },
        [pagination]
    );

    const matchCategory = useCallback(
        result => {
            const categories = get(result, filterOptions.categoryField);
            return categories?.filter(category => filter.categories.includes(category)).length > 0;
        },
        [filterOptions, filter.categories]
    );

    // Set some variables for easier mapping
    // ----------------

    const getCurrentData = () => {
        const hasSearchResults = Boolean(searchResults?.length);
        let renderData = [];
        if (hasSearchResults) {
            renderData = searchResults.filter(Boolean);
        } else if (!hasSearchResults && !filter.query.length) {
            renderData = filter.categories.length && filterOptions?.categoryField ? data.filter(matchCategory) : data;
        }

        if (sortResultsCompare) {
            renderData.sort((a, b) => sortResultsCompare(a, b, filter.categories));
        }

        return renderData;
    };

    const currentData = getCurrentData();
    const paginatedData = pagination ? chunkArray([...currentData], paginationSettings.itemsPerPage) : [];

    // Handle updates
    // ----------------

    useEffect(() => {
        if (data && !isFirstRender.current) {
            removeAll();
            addAll(data);
        }

        isFirstRender.current = false;

    }, [data]);

    useEffect(() => {
        if (data) {
            search(filter.query, {
                filter: filterOptions.categoryField && filter.categories.length > 0 ? matchCategory : undefined,
            });
        }
    }, [data, filter, filterOptions, matchCategory]);

   renderStuff....
 ...

Initialized in this case with:

  <NbFilterable
      .... some more props...
        filterOptions={{
              fields: [
                  'custom.bezeichnung',
                  'custom.beschreibung',
                  'custom.ersteller.vorname',
                  'custom.ersteller.nachname',
                  'baudokumentation.ersteller.nachname',
                  'baudokumentation.ersteller.nachname',
                  'baudokumentation.typ',
                  'baudokumentation.bezeichnung',
              ],
              categoryField: 'custom.kategorien',
       }}
    />

Where filterOptions basically transform to MiniSearch Options. Sorry for the extensive code, but I thought better more, than less^^.

And apologies for creating the issue here and not in react-minisearch!

[lucaong]

I did find a small bug with removeAll in react-minisearch, fixed in the latest release, but I think it's not what causes your issue. You can try out the latest version of react-minisearch though (3.0.2), and let me know.

[florianmatz]

Hi Luca. Just updated the package. Still the same results concerning the difference between searchResults and rawResults (having still one undefined in searchResults).

As a quickfix I just filter out the undefined item in the searchResults and reset the query and categories when data changes.

But still, kind of strange, isn't it? Obviously there is a match, but the item does not get mapped back to searchResults?

[lucaong]

Definitely strange. I am sure the problem lies with react-minisearch (not with MiniSearch itself), but I don’t know what causes it.

Did you notice anything specific about the documents that are missing from the results but present in the rawResults?

[florianmatz]

Hi Luca, sorry for the late response.

No, not at all. Tried it with different datasets, always the same behaviour. Do you need a Fiddle for further investigation?

[lucaong]

if possible, a Fiddle reproducing the issue would be awesome.

[florianmatz]

Hi Luca:

Here you go: https://codepen.io/florianmatz/pen/xxqEaGE?

Steps to reproduce:

• Codepen's console
• Enter "Zum Testen" into the search field ("Enter query")
• You'll get a "No results" warning
• Now switch the dataset; choose "Dataset 1" from select
• App crashes -> discrepancy between rawResults and searchResults (see console) as described in the issue

What's completely weird:

If you switch the datasets BEFORE you perform the reproduction steps, it works like a charm. So, what's important to reproduce: Only with data, that has not been indexed before... So make sure you've reloaded the codepen before testing it.

Here you find a short screen recording, if my description was too confusing... :) https://we.tl/t-3Xpiz6bMq6

It kind of seems, that the indexing is done, but the mapping of terms -> results is not ready yet... (if that makes any sense at all :P)

[lucaong]

thanks a lot, that's really useful! I should be able to dedicate some time to it soon, and hopefully get to the bottom of this

[florianmatz]

Cool! Looking forward to it. Hoping it's not just some kind of my stupidity that costs you time^^

Btw, the download links is only valid for 7 days.

[lucaong]

Great news @florianmatz , the problem was indeed in react-minisearch. It was using useState instead of useRef to store local values such as the map of documents by ID: the problem is that setting the state is asynchronous, while adding/removing things to the index is synchronous by default. Therefore, the effective order of update of the index and the map of documents by ID was swapped, and under certain conditions this would surface as the bug you found.

The new version 3.0.3 of react-minisearch fixes the problem, and seems to work well with your app: https://codepen.io/lucaong/pen/yLMVxxy

Thanks again for reporting this, and for the reproduction example on codepen! I am closing the issue, but feel free to comment on it if you find out that it is not completely solved.