lemon24
commented
3 years ago The logical pipeline of parsing a feed:
- detect encoding (from headers or stream)
- detect xml:base (from headers; needed for relative link resolution – #125)
- detect high-level format (XML, JSON)
- parse XML/JSON stream into intermediary generic Python data structure (ElementTree, JSON dict)
- can be secure or not (#212)
- can support malformed high-level markup (both feedparser and lxml can do this; broken XML does exist in the wild)
- store xml:base
- detect feed format (RSS, Atom, JSON Feed)
- convert generic Python data structure into feed data structure
- resolve relative links
- optional
- sanitize content (https://github.com/lemon24/reader/issues/125#issuecomment-522333200, #227)
- optional
- unify feed data structure (so it looks the same regardless of feed format)
Currently:
- feedparser goes directly from stream to feed data structure by using xml.sax.
- lxml has a way of converting an etree into sax events (https://github.com/lemon24/reader/issues/212#issuecomment-956418094).
- For JSON Feed, reader only relies on the inferred MIME type; feedparser does some sniffing to detect the feed format, and we rely on that (that is, reader has no logic to tell RSS apart from Atom etc.).
- Both relative link resolution (requires xml:base) and content sanitization can happen before or after storage; feedparser does them before storage, and I'm not sure if we can use it to do it after (the things it uses probably aren't stable, and they tie into the sax parsing logic).
In light of various issues feedparser has (see #265), I think it's wise we consider other feed parser implementations to use.
In this issue, we'll: