Currently when we search on OpenAlex by title (enrich.py), we remove the following characters: """()[]{}'@#:;"%&`’,.?!/\^®"""
Some initial testing showed that removing the ' results in not finding a paper.
I assume OpenAlex made some improvements to the search and what it crashes on or not, so we can also update the list of special tokens to remove.
Currently when we search on OpenAlex by title (enrich.py), we remove the following characters: """()[]{}'@#:;"%&`’,.?!/\^®"""
Some initial testing showed that removing the ' results in not finding a paper. I assume OpenAlex made some improvements to the search and what it crashes on or not, so we can also update the list of special tokens to remove.