Gatsby Remark Images ignores embedded pixel density of an image

ichik commented 4 years ago

Description

According to documentation the pixel density of a processed image should be retained, but it's not. Images in 144×144 resolution are displayed in twice the size they were intended to be displayed as a result.

Steps to reproduce

The issue is currently present on Gatsby's own demo site for Remark: https://using-remark.gatsbyjs.org/responsive-images-and-iframes/#what-about-retina-images

For reference, I found old snapshot from 2017 that renders the image in question properly: https://deploy-preview-1987--using-remark.netlify.com/responsive-images-and-iframes/#what-about-retina-images

Expected result

Retina images should be displayed in their intended size (scaled to resolution).

Actual result

Retina images are displayed in their natural pixel size.

Environment

  System:
    OS: macOS 10.15.2
    CPU: (8) x64 Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 12.13.1 - ~/.nvm/versions/node/v12.13.1/bin/node
    npm: 6.13.4 - ~/.nvm/versions/node/v12.13.1/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 79.0.3945.88
    Firefox: 71.0
    Safari: 13.0.4
  npmPackages:
    gatsby: ^2.18.16 => 2.18.16
    gatsby-image: ^2.2.37 => 2.2.37
    gatsby-plugin-manifest: ^2.2.34 => 2.2.34
    gatsby-plugin-mdx: ^1.0.64 => 1.0.64
    gatsby-plugin-offline: ^3.0.30 => 3.0.30
    gatsby-plugin-react-helmet: ^3.1.18 => 3.1.18
    gatsby-plugin-sharp: ^2.3.10 => 2.3.10
    gatsby-plugin-styled-components: ^3.1.16 => 3.1.16
    gatsby-plugin-typescript: ^2.1.23 => 2.1.23
    gatsby-remark-images: ^3.1.39 => 3.1.39
    gatsby-remark-unwrap-images: ^1.0.1 => 1.0.1
    gatsby-source-filesystem: ^2.1.43 => 2.1.43
    gatsby-transformer-sharp: ^2.3.9 => 2.3.9
  npmGlobalPackages:
    gatsby-cli: 2.8.21

github-actions[bot] commented 4 years ago

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open! As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

ichik commented 4 years ago

Well, would be nice to have any comment from the team.

sidharthachatterjee commented 4 years ago

@ichik Hmm, you're right. This looks like a regression.

I'll take a look at this and get this fixed.

bripkens commented 4 years ago

You can use the sizeByPixelDensity option of gatsby-remark-images to achieve this.

johnnydecimal commented 4 years ago

Bump! Just ran in to this one, any action? Those of us with modern Macs ... kinda large set there ... take screenshots all the time and they’re all at 144 dots.

While I’m here, can anyone tell me how to get round/remove the aggressive styling applied to these images? Lots of margin, for instance. I’m using Tailwind and if the image just came in a bare <div> I could at least style it down to size myself.

polarathene commented 4 years ago

Comparing the two links, the older netlify deploy displays the image at half the size, width is set to max-width: 400px and the newer one max-width: 740px. Both have the same srcSet and media condition sizes="(max-width: 740px) 100vw, 740px" with 800w as the largest provided srcSet and 740w being the next smallest.

This is viewed on Linux system without a HiDPI display. I have used Chrome dev tools to toggle the device mode for larger resolution or device like "iPad Pro". Viewed on android phone as well which would also be HiDPI. Note that the image to use by width descriptor w is the images actual pixels, while the sizes attribute provides media conditions querying viewport width in CSS pixels, those get normalized based on display DPI iirc to 96 CSS px == 1 physical inch.

Seems that you're referring to old netlify example as this 2017 PR, which was meant to fix the problem apparently?

This helped add some more clarity to what the original size is meant to be, 800px wide. Sharps defaults are to produce 25% 50% 100% 150% 200% sizes based on max-width/max-height or 800px wide by default.

I'm used to graphql queries for sharp to handle, but it seems for remark-images(which I'm not familiar with), it's been specified in gatsby-config.js at 740px max-width, like so:

https://github.com/gatsbyjs/gatsby/blob/de867d9858917bc2ab2a37a65656b9bdecc09a19/examples/using-remark/gatsby-config.js#L39

The image on Github is 800px, afaik DPI is only relevant to rendering out an image to a physical size, eg 1080p image is still 1920x1080 pixels in dimensions, all DPI does is affect a scale of how many pixels fit within an inch, on a 80" TV, you're going to see obvious pixels, on a smart watch it'd look crisp.

Some file formats may support metadata related to DPI, but generally that should have nothing to do with the actual pixels, in Photoshop your document may have a DPI value, that if you change it scales the pixels accordingly, and retain that within the PSD file so it's not lost. I don't see any metadata in the repo image, it's just 800px wide, only part that is 144 DPI is the filename and is presumably an assumption by whomever created the image? (correct me if I'm wrong about any of this)

Had an exchange with @jen729w who had a cropped macOS screenshot. They were confused why their image asset was being enlarged, I had the original file sent to me and inspected it, no DPI metadata, and only a mere 364x306 pixel dimensions.

So as mentioned earlier, what has happened is sharp with it's defaults, is unable to output the larger sizes? I haven't looked at code or done any additional tests, so I'm not sure if 100% refers to 364px or 800px here. Assuming the 100% scale is 364px, it doesn't output anything larger since there isn't anymore pixels to do so, which just like the gandalf 800px wide image with a max-width of 740px suffers from, this causes a problem. If a 1x DPI scale for a regular display is 364w, then a HiDPI display may ask for a 2x DPI scale (2x more pixels), but 364w is the largest available srcSet, while the media condition in sizes has (max-width 364px) 100vw, 364px on a 2x display, this would be equivalent to 728w.

~~To compensate, the browser takes the 364w image and upscales it to 728w(still 364px in CSS width) internally or something(I don't have a mac or HiDPI display to inspect myself.~~ I was provided a live netlify site where their srcSet only provided 300w and 364w.

Asking @jen729w to configure the max-width setting to 182 resolved their immediate problem, but as it's a global setting that does not help being set to half their input image, which is probably the solution that deprecated setting for sizing by pixel density was doing?

They had also provided a previous version of their site pre-gatsby where the image was sized how they expected it to be, I just looked at the CSS and they had explicitly set the width to 182px, which is what they were expecting apparently. So no upscaling was actually happening on the Mac, it was just using the size of the image available and meeting the media condition expectation?

If the problem is the same as what @jen729w had, and an explicit CSS width is desired with the image providing enough pixels to satisfy that at 200% that CSS size, then the plugin needs a way to do per-image settings like graphql supports.

I don't know how these resources are parsed in markdown to process by sharp, but if they're local URIs, you could use querystring params as a solution to this, which would keep the config per image local to where the URI is actually used(vs a separate file, or whatever other options are available... frontmatter?).

Responding to initial statement:

According to documentation the pixel density of a processed image should be retained, but it's not.

The 144 DPI image is apparently 800px wide, which as mentioned above, is just image pixels if there is no DPI metadata accessible, how can the image pixels be adjusted otherwise? 144 DPI display would show 800px at 25% the size that a regular 72DPI display does.

Although note that when we talk about DPI in this way, it's often not accurately depicting the displays actual DPI, but the systems DPI, eg IIRC Windows has been known to report 120 DPI, while Linux 72 DPI, and web 96 DPI. It was a major source of confusion for me when "printing" my webpages to PDF and some third-party code trying to handle DPI conversion.

Images in 144×144 resolution are displayed in twice the size they were intended to be displayed as a result.

If they're being displayed as 800px (CSS pixels), but you were expecting 400px, then chances are the problem is the sizes should be using 400px for the max-width and fallback values, macOS then would choose the 800w instead of 400w due to the browser knowing it has a higher device-pixel-ratio (DPI scale). It has been manually configured as shown above to 740px though.

ichik commented 4 years ago

My bad, I was so happy to find the example elsewhere in the wild, that I just assumed that metadata will be there. In any case, even when the image has DPI metadata it is ignored (using sizeByPixelDensity in plugin options seeming only produces warnings during build but doesn't change anything else).

polarathene commented 4 years ago

So, what should sharp do with that type of image? If the 2560x1600 pixels stored in the image is at 144 DPI, then is 72 DPI 1280x800 (drops 75% of pixels), or does it mean 144 DPI is 5120x3200, in which case the there's no added details from upscaling pixel data.

By default if DPI is ignored, then a max-width of 800px would then use that as 1x/100% and output 1200px and 1600px larger variants, which your HiDPI system should prefer. Is there an actual issue in this case? Or are you experiencing it when the image size is smaller?

I don't own a HiDPI desktop display to reproduce this issue unfortunately, I've always assumed it works similar to my Android phone which has 1920x1080 6-inch display, 367 DPI. For a 180px image (1x / 100% / 180w @ 96 DPI CSS pixels), it would choose up to ~688px (180px in CSS but 688w srcSet would be chosen based on 367/96 == ~3.82) roughly(fairly certain something like this but not an expert on device-pixel-ratio, I haven't confirmed the value), it's possible it might be rounded to 3x or 4x, either way it would select a larger srcSet than my desktop if a larger size is available to meet that higher DPI.

Based on the referenced before/after project, I'm guessing reducing to 72 DPI will drop 75% of the pixels, and be the expected size you want, just lacking the extra pixels at HiDPI displays? Then the problem is the max-width default being applied for sure, which'd explain why it's only really an issue for Remark images, you'd be able to reproduce by doing that DPI conversion as a pre-process step, which would be sized right, then upscale it (while keeping it 72 DPI in metadata) and reproducing the 144 DPI wrong size issue.

Suggested PR solutions would then be querystring params suggested earlier, or if DPI can be read from metadata, then a multiplier of the difference from 72 DPI could be used so that the different resized outputs can leverage the extra pixels appropriately. I don't think assuming 72 DPI as the base DPI is the right approach though, if the pixels are just pixels, the metadata doesn't matter so much, and the querystring approach works better at fixing the real issue.

ichik commented 4 years ago

Reducing it to 72 DPI isn't really the result I'd want. Typical scenario with 144 DPI images on web is setting them to 0,5 size in pixels which produces nice crisp looking images with all the data rendered in HiDPI. This is an unfortunate result of convention that browsers use: pixel values in CSS for HiDPI screen are scaled to match regular DPI.

polarathene commented 4 years ago

TL;DR: I think you have a misunderstanding of how DPI works here, along with how CSS units interact with displays of various pixel densities. Correct me if I'm mistaken.

Reducing it to 72 DPI isn't really the result I'd want.

I think you misunderstood. I said that doing so, then upscaling it back in pixel count should still cause the issue you're experiencing, which means your image "DPI" is just metadata and doesn't really contribute to the actual problem.

I later state that downscaling isn't a solution if it does yield a proper rendered visual size from the reduced pixel count, since you are removing 75% of the pixel information. The other user was explicitly setting half the pixel count as their display size, their issue was that the image was below 800px for the default sharp resize handling, so there was no additional 1.5x/2x images to output for HiDPI, the issue wasn't the lack of higher pixel count images for HiDPI displays, it was that this also meant the 100% or 1x scale size is what gets set as the CSS pixel sizes media condition as well as the max-width CSS or fixed width/height dimensions IIRC.

Typical scenario with 144 DPI images on web is setting them to 0,5 size in pixels which produces nice crisp looking images with all the data rendered in HiDPI.

Again, I do not change my CSS for images on my phone that are at 367 DPI, there is no need to do this when you use the viewport metadata tags to leverage device-pixel-ratio which will scale CSS units according to the display DPI detected. The crispness then comes from the extra pixels that can be retrieved via srcSet.

In both your case(presumably, or at least with the example you provided) and the one of the other user, there wasn't enough pixels in the input image to (by default) output larger images, as it's assumed the pixel content is just pixels, nothing to do with DPI (which it really shouldn't). I think there is a misunderstanding with how DPI works here, and either I failed to communicate that properly, or I've misunderstood technical details myself.

This is an unfortunate result of convention that browsers use: pixel values in CSS for HiDPI screen are scaled to match regular DPI.

Browsers are smart here. Regardless of whatever DPI your OS or display provides as information, CSS units are 96 DPI, if you use 96px for a square, you would get roughly 1 inch on any display output, I say roughly as some devices manipulate the value reported to the browser with device-pixel-ratio IIRC, but it's roughly matching to that not 96 physical pixels.

When it comes to image source selection, the <picture> element will use sizes attribute for fluid images to select a width descriptor unit in the srcSet list. Those values represent the images actual pixel width, note DPI has little to do here, internally the browser will convert the CSS units via the device-pixel-ratio it has to scale the amount of pixels to an appropriate image source to choose, eg 2x 300w would become 600w, if you have a 600w or higher image option, that would get used for your HiDPI display. CSS unit wise, physically it should output the same size, but pixel density wise, it will be higher and you should be able to note the added detail.

On my 367 DPI phone(1080x1920 6"), comparing to my 1920x1080 21.5" desktop monitor (physically 102 DPI, my OS apparently reports 89 DPI) it's roughly the same physical size. The phone is a bit smaller, I assume it's only scaling by 3x, not 3.82 that I would have expected math wise, chances are the disparity is due to browsers being provided inaccurate scale ratios. The comparison site I used is my CSS grid pugs example site. IIRC it's 360px wide(3x == 1080px on mobile), and the higher DPI image source will be used on the mobile device. I am not an expert about this part of the scaling, 367/102 == ~3.6 could be just as valid I guess.

Can you confirm that your image being used/tested on the webpage is less than 1600 pixels wide?

Have you only tested on a HiDPI display? Or have you also confirmed a disparity in layout proportions being significantly different on a non-HiDPI display?

ichik commented 4 years ago

there wasn't enough pixels in the input image

But there certainly was, it was just rendered in twice the size of what you'd expect.

Can you confirm that your image being used/tested on the webpage is less than 1600 pixels wide?

I think we have the misunderstanding here on that 2560 × 1600 line in metadata. This line in combination with double density means that the image should be 1280 x 800 in terms of setting its size in CSS / HTML to achieve that crispiness. It should be so both on regular DPI screens and HiDPI (with the regular ones just omitting pixels they can't render).

Can you confirm that your image being used/tested on the webpage is less than 1600 pixels wide?

Yes, I've tested it with various images, they are always double the size of what I want them to be.

Have you only tested on a HiDPI display? Or have you also confirmed a disparity in layout proportions being significantly different on a non-HiDPI display?

Proportions should be the same, and they're always off from those intended. The only difference here is that regular DPI screens have crisp images (since their DPI matches the internal ones).

Sorry, if I'm unclear here, but the short summary of what I would expect is:

If the image has internal DPI that is different from regular 72 scale it by factor of difference between regular and internal DPI, so it appears in intended physical size with additional HiDPI pixels. If relying on metadata is too flaky some other methods such as Apple's @2x suffixes or other explicit settings applied somewhere (in markdown, in GraphQL query maybe?) would also work.

polarathene commented 4 years ago

TL;DR: I'm pretty sure DPI isn't the real issue here. It's default settings and the image inputs you're providing, the remark images plugin needs to allow users to configure sharp transforms on a per-image basis like graphql with gatsby-image allows for.

If you feel that it really is a DPI issue and sharp is messing with the image in someway, please modify the HTML markup that the remark plugin generates, modify all image URIs to point to your original image with 144 DPI. The problem should still persist:

If it's a width/height issue, that's due to CSS values being selected and is probably working as intended, just needs user configuration per image, to confirm, you can change the plugins max width setting, just like the before/after project does, assign it a 1x width, that should match the CSS px size you'd expect it to be sized to.
If it's a density issue, where you want more pixels to be displayed, the srcSet must provide an image with sufficient pixels, and the sizes attribute should also specify adequate CSS media query to select that srcSet, that should also be possible to confirm expectations by setting the plugins default max width option.

I understand that what you'd like is for automatic handling based on DPI, if metadata provides that. However, I think that can result in unreliable/inconsistent situations that confuses users when some of their images lack the metadata (since images are DPI agnostic), plus to my knowledge there isn't really a reliable base DPI value anymore.

But there certainly was, it was just rendered in twice the size of what you'd expect.

Can you provide a sample image? The before/after project example only provides a "144" DPI 800px wide JPG, that is insufficient for HiDPI when sharp will just see 800px and use the mentioned 800px max-width default for 1x / 100% size.

This line in combination with double density means that the image should be 1280 x 800 in terms of setting its size in CSS / HTML to achieve that crispiness

Can you please back that statement up somewhere? Feel free to send the image to me, but I assume that even on my low DPI display I'd have 2560px wide and can inspect each pixel regardless of the lower DPI, which I've tried to explain to the best of my knowledge is only relevant to a renderer, the image data itself is DPI agnostic.

It should be so both on regular DPI screens and HiDPI (with the regular ones just omitting pixels they can't render).

If the image were 200px, and I go into photoshop and scale it to 100px, regardless of display it's only 100px of data visible. If you instead scale to physical sizes, then yes we're missing pixels on the lower density display.

With CSS units, we're getting similar, what's important here is the image selection so we're not wasting bandwidth on an image with twice the pixels for a low DPI display, while getting the denser image if the display can support that. Different images is fine.

Yes, I've tested it with various images, they are always double the size of what I want them to be.

But they're fine/correct size on low DPI displays?

Can you try a large image that's 4k or larger? Plenty available on sites like Unsplash, my CSS pugs project uses such images.

Proportions should be the same, and they're always off from those intended. The only difference here is that regular DPI screens have crisp images (since their DPI matches the internal ones).

So that's a no for testing on low DPI displays?

DPI matching internal ones? Weren't you just stating that your inputs are higher DPI images? If images were DPI limited, then my mobile phone should be having similar problems due to it's significantly higher DPI of 367? What DPI are we talking about here? You've mentioned 144 DPI on macOS screenshot, while regular DPI is what... 72? Yet my OS reports 89px and my physical display for a 21.5" monitor with 1080p resolution is 102 DPI.

I'd like to bring up the other users 364px 144 DPI image again. Their working to expectation example happened to explicitly set the width to 182px in CSS on the img element. Due to mentioned sharp behaviour, this was instead being output as 364px, configuring the remark plugin settings to use a default max-width of 182 instead corrected this issue and meet the users expectations, it just wasn't a suitable fix since it's a global setting instead of per image. In regular gatsby-image with graphql, you solve the issue exactly the same way but can do per-image, DPI is irrelevant, so long as there is enough pixels for higher resolutions and you specify what the base resolution should be, you're good. Hence suggestion for querystring params as a fix for remark images plugin.

If the image has internal DPI that is different from regular 72 scale it by factor of difference between regular and internal DPI, so it appears in intended physical size with additional HiDPI pixels.

Where is 72 DPI a regular/default DPI? In Windows it can be 120 DPI as an OS DPI, each physical display can have it's own DPI, as mentioned earlier Linux reported DPI can vary too. If you were going to normalize to a specific DPI it'd probably be 96 which CSS uses.

Again though, I'd like to stress that an image is DPI agnostic, it should only contain the pure pixel data. DPI only matters with how many logical pixels fit into a physical unit. You could strip that DPI metadata and you would get the same representation on any display, provided there is ample pixels to display it at 100% size, otherwise some scaling will happen, which is still not relevant to DPI of a display, 1080p image is the same details on a smart watch size 1080p display as it is a 80" TV.

If relying on metadata is too flaky some other methods such as Apple's @2x suffixes or other explicit settings applied somewhere (in markdown, in GraphQL query maybe?) would also work.

Fixed type images use scales such as 1x and 2x for image selection. For clarity, sharp is taking the image input you provided it, and if it's less than 800px, that full size is being treated as the 1x. With graphql, if you lowered the max-width that would then become the 1x size, 2x is then available if there is that many pixels available.

I have suggested what you brought up here as providing parameters via querystring, these would be appended to the end of your image URI. It's simpler to take the approach of specifying the CSS pixel units which map to the 1x size then worry about what DPI a source image is in and varying consistency in param inputs based on that, that becomes more obvious if you end up with mixed DPI image resources.

ichik commented 4 years ago

Yes, indeed, querystring is a good enough solution for my use case. Sorry for not catching up to that earlier and making you go through the lengthy explanation of how it works.

polarathene commented 4 years ago

Great we're on the same page now, stuff like DPI in situations like this can get confusing :)

Now the hard part, finding someone to volunteer time to commit the feature, or hoping it gets prioritized by the Gatsby team.

ichik commented 4 years ago

@polarathene can you help me a bit with figuring the proper syntax in markdown here. Suppose I have an image of 850 width which I want to show in this @2x mode (width 425 "pixel" width). I've tried so far ![Image](./image.jpg?maxWidth=425) and ![Image](./image.jpg?max-width=425) with no luck.

polarathene commented 4 years ago

@ichik sorry if it wasn't clear. I was suggesting that approach as a solution, it would still need to be implemented though.

I understand it's been a long 7 month wait so far, and getting someone other than the Gatsby team to work on it will likely mean the volunteer contributor needs the fix themselves or they've been given enough financial incentive, there's a chance someone chooses to contribute and not need it, but they'd need to discover the issue and read these comments at the end first. Getting Gatsby team to notice and potentially increase priority on it would probably need some active reaching out.

The only solution you have available right now with the plugin is the global max width plugin setting in gatsby-config.js, like so:

https://github.com/gatsbyjs/gatsby/blob/de867d9858917bc2ab2a37a65656b9bdecc09a19/examples/using-remark/gatsby-config.js#L39

That's global though....so only good for fixing one image size for an entire project.

ichik commented 4 years ago

Oh, right, sorry for being so clueless.

polarathene commented 4 years ago

No worries, I hope someone does find the time to contribute the feature in the near future! :)

artt commented 3 years ago

Any updates on this? :(

LekoArts commented 2 years ago

Hi!

I'm closing this as a stale issue as in the meantime Gatsby 4 and related packages were released. You can check our Framework Version Support Page to see which versions currently receive active support.

Please try the mentioned issue on the latest version (using the next tag) and if you still see this problem, open a new bug report. It must include a minimal reproduction.

Thanks!

gatsbyjs / gatsby