mozilla / page-metadata-service

DEPRECATED - A RESTful service that returns the metadata about a given URL.
Mozilla Public License 2.0
19 stars 8 forks source link

Return actual image dimensions #89

Closed nchapman closed 7 years ago

nchapman commented 7 years ago

I noticed that real image dimensions aren't being returned. Having the image dimensions is super helpful to make sure placeholders are the right size and to do custom layouts based on the size of the images.

Here's a cool lib that could make this easy/efficient: https://github.com/ShogunPanda/fastimage

pdehaan commented 7 years ago

Dupe of #38?

nchapman commented 7 years ago

Dupe of #38?

Yes except I only care about width and height and provided an efficient solution 😄

pdehaan commented 7 years ago

Probably blocked on #48; "Add Redis caching" That way we won't be sending network requests to 3rd party sites for images on each request, just to get dimensions.

pdehaan commented 7 years ago

Actually, that fastimage module is pretty awesome. There is one "gotcha" with it though, you need to set fastimage.threshold(-1); (ref https://github.com/ShogunPanda/fastimage/issues/2), otherwise it seems to be barfing on a bunch of images (or at least the sample set I used from the recommendations proxy endpoint).

const fs = require('fs');

const { getMetadata, getRecommendations } = require('proxy-services');
const fastimage = require('fastimage');

fastimage.threshold(-1);

getRecommendations()
  .then((urls) => getMetadata(urls))
  .then((metadata) => {
    fs.writeFileSync('metadata.json', JSON.stringify(metadata, null, 2));
    return metadata;
  })
  .then((metadata) => {
    return Object.keys(metadata).map((url) => {
      const result = metadata[url];
      if (result.images[0]) {
        return fastimage.info(result.images[0].url)
          .then(({width, height, url}) => {
            // Overwrite the bogus width/height with the returned values.
            result.images[0].width = width;
            result.images[0].height = height;
            return result;
          });
      }
      return result;
    });
  })
  .then((promises) => Promise.all(promises))
  .then((result) => console.log(JSON.stringify(result, null, 2)))
  .catch((err) => console.error(err));
**Output:** (click-o to expand-o) ``` json [ { "description": "If you’re anything like me, you’re incredibly tempted several times a day to spend money on something that you don’t really need.", "favicon_url": "https://i.kinja-img.com/gawker-media/image/upload/s--OqAhAoNZ--/c_fill,fl_progressive,g_center,h_200,q_80,w_200/u0939doeuioaqhspkjyc.png", "images": [ { "entropy": 1, "height": 450, "url": "https://i.kinja-img.com/gawker-media/image/upload/s--0zrMrngY--/c_fill,fl_progressive,g_center,h_450,q_80,w_800/ognrc5oe4tikygoi2fd0.png", "width": 800 } ], "original_url": "http://lifehacker.com/ten-useful-strategies-for-learning-financial-self-contr-1785904154", "title": "Ten Useful Strategies for Learning Financial Self-Control", "url": "http://lifehacker.com/ten-useful-strategies-for-learning-financial-self-contr-1785904154" }, { "description": "Susan* bought her 6-year-old son John an iPad when he was in first grade. “I thought, ‘Why not let him get a jump on things?’ ” she told me during a therapy session. John’s school had begun using t
", "favicon_url": "https://s0.wp.com/wp-content/themes/vip/nypost-2015/static/images/apple-icons/nypost/apple-icon.png", "images": [ { "entropy": 1, "height": 1333, "url": "https://i0.wp.com/thenypost.files.wordpress.com/2016/08/655554041.jpg?quality=90&strip=all&ssl=1", "width": 2000 } ], "original_url": "http://nypost.com/2016/08/27/its-digital-heroin-how-screens-turn-kids-into-psychotic-junkies/", "title": "It’s ‘digital heroin’: How screens turn kids into psychotic junkies", "url": "http://nypost.com/2016/08/27/its-digital-heroin-how-screens-turn-kids-into-psychotic-junkies/" }, { "description": "An in-depth profile of the man who went from Xanax zombie to heavy rotation at the White House.", "favicon_url": "http://www.gq.com/favicons/touch-icon-iphone.png", "images": [ { "entropy": 1, "height": 1125, "url": "http://media.gq.com/photos/57bb651ec4fb86bf3ac6f3c5/16:9/pass/chance-the-rapper-gq-0916-lede-3x2.jpg", "width": 2000 } ], "original_url": "http://www.gq.com/story/how-chance-the-rappers-life-became-perfect", "title": "How Chance the Rapper's Life Became Perfect", "url": "http://www.gq.com/story/how-chance-the-rappers-life-became-perfect" }, { "description": "Now she watched as her ruin seemed to unfold before her. It was enough to send her to jail, and more than enough to destroy her name.", "favicon_url": "http://www.latimes.com/img/favicon.ico", "images": [ { "entropy": 1, "height": 365, "url": "http://www.trbimg.com/img-57c66b83/turbine/la-framed-part-3-share-the-image/650", "width": 649 } ], "original_url": "http://www.latimes.com/projects/la-me-framed/", "title": "She was the PTA mom everyone knew. Who would want to harm her?", "url": "http://www.latimes.com/projects/la-me-framed/" }, { "description": "Will future historians remember the former prime minister for anything more than his great Brexit bungle?", "favicon_url": "http://www.newstatesman.com/sites/all/modules/touch_icons/apple-touch-icon.png", "images": [ { "entropy": 1, "height": 520, "url": "http://www.newstatesman.com/sites/default/files/styles/thumb_730/public/Longreads_2016/08/2016_34_cameron_opener.jpg?itok=HvslSky2", "width": 730 } ], "original_url": "http://www.newstatesman.com/politics/uk/2016/08/david-camerons-fatal-insouciance", "title": "David Cameron's fatal insouciance", "url": "http://www.newstatesman.com/politics/uk/2016/08/david-camerons-fatal-insouciance" }, { "favicon_url": "http://www.nytimes.com/favicon.ico", "images": [], "original_url": "http://www.nytimes.com/2016/08/29/opinion/why-did-we-stop-teaching-political-history.html", "title": "Log In - The New York Times", "url": "http://www.nytimes.com/2016/08/29/opinion/why-did-we-stop-teaching-political-history.html" }, { "favicon_url": "http://www.nytimes.com/favicon.ico", "images": [], "original_url": "http://www.nytimes.com/2016/09/04/magazine/edward-snowdens-long-strange-journey-to-hollywood.html", "title": "Log In - The New York Times", "url": "http://www.nytimes.com/2016/09/04/magazine/edward-snowdens-long-strange-journey-to-hollywood.html" }, { "description": "Do mission-driven organizations with tight budgets have any choice but to demand long, unpaid hours of their staffs?", "favicon_url": "https://cdn.theatlantic.com/assets/static/b/theatlantic/common/img/apple-touch-icon-iphone.png", "images": [ { "entropy": 1, "height": 500, "url": "https://cdn.theatlantic.com/assets/media/img/mt/2016/08/USPIRG_1/facebook.jpg?1471976894", "width": 960 } ], "original_url": "http://www.theatlantic.com/business/archive/2016/08/the-plight-of-the-overworked-nonprofit-employee/497081/", "title": "The Plight of the Overworked Nonprofit Employee", "url": "http://www.theatlantic.com/business/archive/2016/08/the-plight-of-the-overworked-nonprofit-employee/497081/" }, { "description": "The average age of a Fox News viewer is about 70. The average life expectancy of a white American male is about 80.", "favicon_url": "https://cdn.theatlantic.com/assets/static/b/theatlantic/common/img/apple-touch-icon-iphone.png", "images": [ { "entropy": 1, "height": 500, "url": "https://cdn.theatlantic.com/assets/media/img/mt/2016/08/RTR4BYJM/facebook.jpg?1472477144", "width": 960 } ], "original_url": "http://www.theatlantic.com/business/archive/2016/08/the-twilight-of-fox-news/497684/", "title": "Fox News’ Biggest Competitor: Mortality", "url": "http://www.theatlantic.com/business/archive/2016/08/the-twilight-of-fox-news/497684/" }, { "description": "No one will ever find a closer exoplanet—now the race is on to see if there is life on its surface.", "favicon_url": "https://cdn.theatlantic.com/assets/static/b/theatlantic/common/img/apple-touch-icon-iphone.png", "images": [ { "entropy": 1, "height": 500, "url": "https://cdn.theatlantic.com/assets/media/img/mt/2016/08/eso1629a_1-2/facebook.jpg?1471990330", "width": 960 } ], "original_url": "http://www.theatlantic.com/science/archive/2016/08/astronomers-have-found-a-habitable-planet-orbiting-our-suns-nearest-neighbor/497117/", "title": "An Epochal Discovery: A Habitable Planet Orbits Our Neighboring Star", "url": "http://www.theatlantic.com/science/archive/2016/08/astronomers-have-found-a-habitable-planet-orbiting-our-suns-nearest-neighbor/497117/" }, { "description": "Proxima Centauri, the closest star to our Solar System, has a planet that may have just the right features to support life on its surface. That makes this world the closest possible residence for...", "favicon_url": "https://cdn0.vox-cdn.com/images/verge/2.0/iphone-touch-icon.v3486ec7.png", "images": [ { "entropy": 1, "height": 900, "url": "https://cdn0.vox-cdn.com/thumbor/JdNrrPPLnkXTiWXgBNp0gw-jABI=/9x17:4000x2262/1600x900/cdn0.vox-cdn.com/uploads/chorus_image/image/50507795/eso1629a_20_1_.0.jpg", "width": 1600 } ], "original_url": "http://www.theverge.com/2016/8/24/12604828/proxima-centauri-exoplanet-possible-alien-life-starshot", "title": "A planet orbits around the closest star to our Solar System — and it may be habitable", "url": "http://www.theverge.com/2016/8/24/12604828/proxima-centauri-exoplanet-possible-alien-life-starshot" }, { "description": "The story is that there is no scandal.", "favicon_url": "https://cdn0.vox-cdn.com/uploads/hub/sbnu_logo_minimal/441/touch_icon_ipad_retina_1000_yellow.755.png", "images": [ { "entropy": 1, "height": 600, "url": "https://cdn0.vox-cdn.com/thumbor/WwSDeXFaxMZxQPEkac-nmGW_o20=/0x239:3000x1906/1080x600/cdn0.vox-cdn.com/uploads/chorus_image/image/50506645/594438236.0.jpg", "width": 1080 } ], "original_url": "http://www.vox.com/2016/8/24/12618446/ap-clinton-foundation-meeting", "title": "The AP’s big exposĂ© on Hillary meeting with Clinton Foundation donors is a mess", "url": "http://www.vox.com/2016/8/24/12618446/ap-clinton-foundation-meeting" }, { "description": "The story is still that there’s no story.", "favicon_url": "https://cdn0.vox-cdn.com/uploads/hub/sbnu_logo_minimal/441/touch_icon_ipad_retina_1000_yellow.755.png", "images": [ { "entropy": 1, "height": 600, "url": "https://cdn0.vox-cdn.com/thumbor/zQFkprJsfWRAjEf1vMaHzSBojV0=/0x372:3000x2039/1080x600/cdn0.vox-cdn.com/uploads/chorus_image/image/50513071/594429438.0.jpg", "width": 1080 } ], "original_url": "http://www.vox.com/2016/8/24/12630586/ap-response-clinton-foundation", "title": "The AP’s defense of its bad Clinton Foundation story is also bad", "url": "http://www.vox.com/2016/8/24/12630586/ap-response-clinton-foundation" }, { "description": "The state’s new climate law will mean so much more than a few solar panels and electric cars.", "favicon_url": "https://cdn0.vox-cdn.com/uploads/hub/sbnu_logo_minimal/441/touch_icon_ipad_retina_1000_yellow.755.png", "images": [ { "entropy": 1, "height": 600, "url": "https://cdn0.vox-cdn.com/thumbor/G_q4AQVXP02V9BqwhUtkF967RDc=/0x269:4892x2987/1080x600/cdn0.vox-cdn.com/uploads/chorus_image/image/50562673/shutterstock_114083809.0.jpg", "width": 1080 } ], "original_url": "http://www.vox.com/2016/8/29/12650488/california-climate-law-sb-32", "title": "California is about to find out what a truly radical climate policy looks like", "url": "http://www.vox.com/2016/8/29/12650488/california-climate-law-sb-32" }, { "favicon_url": "https://blog.codinghorror.com/assets/images/codinghorror-app-icon.png?v=bbef415673", "images": [], "original_url": "https://blog.codinghorror.com/can-software-make-you-less-racist/", "title": "Can Software Make You Less Racist?", "url": "https://blog.codinghorror.com/can-software-make-you-less-racist/" }, { "description": "We have two popular historians to blame for our profound misunderstanding of young people’s lifestyle choices.", "favicon_url": "https://newrepublic.com/assets/favicons/apple-touch-icon-57x57.png", "images": [ { "entropy": 1, "height": 577, "url": "https://images.newrepublic.com/90dfb62fb0e78675155bc1528ec41a1c91088227.jpeg?w=1109&h=577&crop=faces&fit=crop&fm=jpg", "width": 1109 } ], "original_url": "https://newrepublic.com/article/136415/myth-millennial-cultural-rebel", "title": "The Myth of the\nMillennial as Cultural Rebel", "url": "https://newrepublic.com/article/136415/myth-millennial-cultural-rebel" }, { "description": "Astronomers using ESO telescopes and other facilities have found clear evidence of a planet orbiting the closest star to Earth, Proxima Centauri. The long-sought world, designated Proxima b, orbits its cool red parent star every 11 days and has a temperature suitable for liquid water to exist on its surface. This rocky world is a little more massive than the Earth and is the closest exoplanet to us — and it may also be the closest possible abode for life outside the Solar System. A paper describing this milestone finding will be published in the journal Nature on 25 August 2016.", "favicon_url": "http://www.eso.org/public/archives/favicon.ico", "images": [ { "entropy": 1, "height": 831, "url": "https://cdn.eso.org/images/screen/eso1629a.jpg", "width": 1280 } ], "original_url": "https://www.eso.org/public/news/eso1629/", "title": "Planet Found in Habitable Zone Around Nearest Star - Pale Red Dot campaign reveals Earth-mass world in orbit around Proxima Centauri", "url": "https://www.eso.org/public/news/eso1629/" }, { "description": "It’s hypocritical of Mark Zuckerberg to sing the praises of the web’s founder when he’s trying to monopolise the internet", "favicon_url": "https://assets.guim.co.uk/images/favicons/451963ac2e23633472bf48e2856d3f04/152x152.png", "images": [ { "entropy": 1, "height": 630, "url": "https://i.guim.co.uk/img/media/9e5c737225b17447e400cee1c456ab0637ea6d98/0_82_4320_2593/4320.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxNi8wNS8yNS9vdmVybGF5LWxvZ28tMTIwMC05MF9vcHQucG5n&s=e70e5e72561a1385a7b7e6208abd9f01", "width": 1200 } ], "original_url": "https://www.theguardian.com/commentisfree/2016/aug/28/tim-berners-lee-open-web-mark-zuckerberg-facebook", "title": "Why Tim Berners-Lee is no friend of Facebook | John Naughton", "url": "https://www.theguardian.com/commentisfree/2016/aug/28/tim-berners-lee-open-web-mark-zuckerberg-facebook" }, { "description": "Experts say human impact on Earth so profound that Holocene must give way to epoch defined by nuclear tests, plastic pollution and domesticated chicken", "favicon_url": "https://assets.guim.co.uk/images/favicons/451963ac2e23633472bf48e2856d3f04/152x152.png", "images": [ { "entropy": 1, "height": 630, "url": "https://i.guim.co.uk/img/media/29ee7b95089d9bafb573576c2e786866eb47f321/0_350_5216_3131/5216.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxNi8wNS8yNS9vdmVybGF5LWxvZ28tMTIwMC05MF9vcHQucG5n&s=8838351eed6518e78c6fbfc6b810d9a0", "width": 1200 } ], "original_url": "https://www.theguardian.com/environment/2016/aug/29/declare-anthropocene-epoch-experts-urge-geological-congress-human-impact-earth", "title": "The Anthropocene epoch: scientists declare dawn of human-influenced age", "url": "https://www.theguardian.com/environment/2016/aug/29/declare-anthropocene-epoch-experts-urge-geological-congress-human-impact-earth" }, { "description": "There were 10,000 lookouts, scanning the wilderness for signs of smoke. Now just a few hundred remain, and they pass the time hiking, writing and knitting", "favicon_url": "https://assets.guim.co.uk/images/favicons/451963ac2e23633472bf48e2856d3f04/152x152.png", "images": [ { "entropy": 1, "height": 630, "url": "https://i.guim.co.uk/img/media/5305f5e2b08fe5c7ee142f507733fc7fd15f21f0/0_151_2016_1210/2016.jpg?w=1200&h=630&q=55&auto=format&usm=12&fit=crop&bm=normal&ba=bottom%2Cleft&blend64=aHR0cHM6Ly91cGxvYWRzLmd1aW0uY28udWsvMjAxNi8wNS8yNS9vdmVybGF5LWxvZ28tMTIwMC05MF9vcHQucG5n&s=4102db3ce2e8bb94abfe8571d0ad3993", "width": 1200 } ], "original_url": "https://www.theguardian.com/us-news/2016/aug/30/us-national-parks-fire-lookout-forest-wildfire", "title": "'Freaks on the peaks': the lonely lives of the last remaining forest fire lookouts", "url": "https://www.theguardian.com/us-news/2016/aug/30/us-national-parks-fire-lookout-forest-wildfire" } ] ```