Open tboby opened 3 weeks ago
Is openschema intended to be a universal input? Otherwise, is this an downstream issue and should I be trying to see if Tandoor would accept recipes in JSON (avoiding the use of recipe-scrapers) instead of HTML?
Hi @tboby - thanks for the discussion.
You're probably aware of some/all of this, but as a recap/background:
We have three independent projects in this case, each with their own slightly different perspectives.
schema.org
standard (including the Recipe
type), as you refer to as openschema (I haven't seen that name myself, but maybe it is called that elsewhere) was put forward by a few companies (Google, Microsoft, Yahoo, Yandex) -- it's a simple format that websites can easily adopt and that makes it easier for those companies and others to retrieve metadata of various kinds. It allows data on the web to be more structured and accessible - generally good things.recipe-scrapers
that seems to have a large-ish known user base, so I try to keep them informed of updates like the v15 upgrade.recipe-scrapers
- this Python library; a way to retrieve recipe metadata from webpages -- initially HTML-only, but then much-expanded after adding schema.org
and other metadata formats -- and until v15, supportive of multi-request (request an HTML page, then from there request a JSON API) scraping.That's important background to answer your question:
Is openschema intended to be a universal input? Otherwise, is this an downstream issue and should I be trying to see if Tandoor would accept recipes in JSON (avoiding the use of recipe-scrapers) instead of HTML?
I can't really provide a useful answer about the ambitions of schema.org
-- but it certainly meets the requirements for recipe-scrapers
, and that simplifies our code a lot. In some ways it'd be nice if we also had parallel HTML retrieval for the recipe websites we support, because then we could verify that what is claimed in the schema.org
matches what appears to users -- but that's not something we support today.
I think it's certainly valid to ask Tandoor whether they would consider retrieving from schema.org
JSON directly. If I were them, then on the one hand I would be quite comfortable offloading the support/accuracy checks about recipes to a separate component (e.g. recipe-scrapers
) - but there is also certainly a performance and dependency management angle to consider too (fewer dependencies is good) - and, maybe at a lower priority, there could potentially be a vendor-lockin concern (schema.org
is good, but perhaps the ability to support sites that don't use it for whatever reason is also beneficial).
Gousto is in some ways an optimal source for recipes. For each recipe (https://www.gousto.co.uk/cookbook/vegetarian-recipes/3-cheese-veg-packed-pasta-bake) they have a public api which provides JSON data used to build the recipe page. This includes the useful grouping of tiny steps into stages.
In v5 the ability to make requests in a scraper was removed, and with it the Gousto scraper. As I couldn't see a way forward using a custom scraper in this repo, I decided to quickly put together a script to generate an open schema document that could be fed to recipe-scrapers instead.
This works fine, except for one major issue: the open schema scraper concatenates all instruction steps together with "\n" by default.
The downstream app I'm using (Tandoor) provides a handy "split instructions by newline" feature to help parse recipes with a single block of text; however due to the joining of instructions it can't tell the difference between newlines in the original steps (that I'd like to keep) and newlines inserted by the open schema scraper.
Is openschema intended to be a universal input? Otherwise, is this an downstream issue and should I be trying to see if Tandoor would accept recipes in JSON (avoiding the use of recipe-scrapers) instead of HTML?