extraneous newlines and whitespace in output

[akhmerov]

Describe the bug

Various string parameters contain \n where they really shouldn't: title and abstract most prominently. In the title there's also extra whitespace.

To Reproduce

import arxiv
print("\n  " in arxiv.query(id_list=["2008.01734"])[0].title)
# >>> True

Expected behavior

No extraneous whitespace

Versions

• python version: 3.8.3
• arxiv.py version: 0.5.3
• feedparser version: 5.2.1 (the problem may well be due to feedparser).

[lukasschwab]

Thanks for the easy instructions for reproducing this!

The whitespace here is returned by the underlying arXiv API. It seems to be a ragged line wrap at 80 characters arXiv introduces themselves, but authors can validly include newlines of their own to break paragraphs (at least in abstracts):

So if you want a new paragraph or a table of contents, be sure to indent the lines after the carriage return.

I'd like to find an example paper with a multiparagraph abstract to confirm how this is represented by the API; "strip all newlines that aren't followed by whitespace" may be a valid rule here.

As far as I know, titles can never include user input newlines, so this might be a more straightforward place to strip them. I'm a little concerned about introducing a find-and-replace operation on every query result. I'll shoot an email to the arXiv API Google Group about this behavior.

In the mean time, a small function for stripping newlines that aren't followed by whitespace from strings:

import re

def rm_newlines(entry):
  rm_newlines = lambda s: re.sub(r'\n[^\s]', ' ', res)
  entry.title = rm_newlines(entry.title)
  entry.summary = rm_newlines(entry.summary)
  return entry

[lukasschwab]

Sounds like this was a bug introduced into the API itself (link to the Google Group message), but the arXiv API is likely frozen while they work on a new JSON API to replace the current Atom API.