Closed andonagio closed 2 years ago
I believe the issue is the presence of the srcset
attribute. This is a common problem when fetching images, try use the modify block to change that attribute to a null, this appears to work for myself.
{
"type": "xpath",
"xpath": "article",
"modify": [
{
"type": "replace",
"search": [
"srcset="
],
"replace": [
"null="
]
}
]
}
I'm able to get the article header image and the author bio image. However, any images embedded within the article are missing, and the above filter doesn't seem to help:
I think the issue is that there is no image data for FeedIron to modify. I can see the image tags in in my browser's inspection tool:
However, when I then look at the page's source, there are no picture
or img
tags to be found:
This is also what I see when I try to download the source via wget
or lynx
. I'm confused as to how the site is handling images and how I can get around this.
Is anybody able to successfully get embedded images in NYT articles? If so, how?
So I did a wget on the page you used as an example. It looks like there is a bunch of javascript they are using to lazy load the images. This used to be very common prior to the introduction of the loading="lazy"
attribute.
There isn't a great method to get the rendered webpage outside of something like FlareSolver or another headless browser setup. Currently there isn't anything built into feediron, although I have planned on introducing modular fetching like the filters for a while now.
Prerequisites
Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.
Expected Behavior
When getting the content of a New York Times article, the entire article should load, including any images.
Current Behavior
Everything but the images loads.
Steps to Reproduce
Please provide detailed steps for reproducing the issue.
Checking the XML content of the resulting RSS feed, the
picture
andimg
tags from the website are not present in the feed's contents. If Iwget
the article's URL, those tags are missing from these results as well. The issue seems to be that the New York Times website doesn't load the images immediately with the article. Perhaps it's doing some kind of lazy loading? I can't tell, but I'd like to know if there is a way around this problem.