Assess impact of IG API changes

danrademacher commented 6 years ago

https://techcrunch.com/2018/04/02/instagram-api-limit/ details major changes to the IG API that came unannounced earlier this week and have clobbered the ability of CaliParks to load most IG photos.

Let's post findings here about the new limitations and their implications for the site. This is not yet a "fix it" task since we're not sure it can be fixed.

gregallensworth commented 6 years ago

Item one: development of scrapers is no longer possible

https://www.instagram.com/developer/

The existing APIs had apparently been slated for deprecation in July 2018 and December 2018, so the Instagram features may have ceased working. Certainly the reinstatement of scrapers would have become impossible at that point.
At this point, there is no ability for a user to embed a "my instagram photos" feed onto external websites without using an access token. Come 2020, users would not even be able to access their own data and images by way of external apps and access tokens.

As such, further scraping of photos is no longer feasible, even if interest and funding existed.

danrademacher commented 6 years ago

A previous version of the site featured Flickr photos as well (before we assumed control of the code and data). I wonder if those old Flickr photos are still present in the database and might represent an alternative source of images. As far as I know, Flickr's API remains usable for things like harvesting, though the platform is much less widely used than IG, so hard to know how much new content one would get there.

Could you take a look in the dbase and see if there's a table of flickr images tied to CPAD ids?

gregallensworth commented 6 years ago

CDN URL changes

It would appear that Instagram has either restricted usage of their CDN and/or changed the URLs of prior photos as well, as all photos are now giving an "Access denied" message.

This does not use any API, merely requesting previously-public photos via their CDN URL.

Perusing some Instagram user pages today and using my debugger on the photos, it seems that the URL formats being used today are indeed quite different.

A Specific Test Case

One test case, a photo previously linked to Golden Gate Park (Superunit ID 10900)

Photo URL with frame: http://instagram.com/p/wZJPWIuC7g/

Old URLs, not working: http://scontent-a.cdninstagram.com/hphotos-xaf1/t51.2885-15/10848105_355658264614724_1128453539_a.jpg https://1gu2c4mrsgexdinbogq000000.igsonar.com/hphotos-ak-xaf1/10848105_355658264614724_1128453539_s.jpg http://scontent-a.cdninstagram.com/hphotos-xaf1/t51.2885-15/10848105_355658264614724_1128453539_n.jpg

New URL: https://instagram.fsnc1-1.fna.fbcdn.net/vp/112360b104fafe391d9e553c42b10952/5B747E7A/t51.2885-15/e15/10848105_355658264614724_1128453539_n.jpg

Looking Forward

In theory, one could go over all photos in the database and:

extract the URL for viewing them over at Instagram in the page
visit that URL
examine the page content (may come from a data feed, not be present in DOM HTML) to determine the new URL of the photo
Update a new URL field in the photo

Challenges may include:

IG's mechanism for obtaining the photo for insertion into the page, may not be simple HTML, but some JavaScript endpoint; such an endpoint may include countermeasures against such scraping
There are 5,268,786 photos, so this could take significant time

gregallensworth commented 6 years ago

Feasibility of reprogramming to show Flickr photos instead of Instagram

Summary

This table ultimately boils down to 4 specific SQL queries, all of which look as if they would transition over to flickr_photos with little trouble.
These photos are properly related to CPAD 2017a parks.
For the most part their URLs still appear to be valid, with failure rates about what one would expect 3+ years after photos are posted.
Photos are of different sizes than at Instagram, so there may be layout adjustments needed as well.
There are 1/20th as many Flickr photos are there are Instagram photos, so expect significantly fewer photos all around.

DB table: flickr_photos

264,344 photo-records (vs 5.2M for Instagram)
superunit_id field exists to connect to parks
- all SU-IDs correspond to a park in CPAD 2017a
metadata JSON field exists
- different structure than Instagram; adaptation required
- no links to photo's "framed" page, nor user account
- Photo URLs provided at 6 sizes
- Random sampling, about 30% of photos no longer available; error condition is that it does display a Photo Not Available image, and not a loading failure, so we wouldn't readily be able to detect these conditions as they are a photo
- https://farm8.staticflickr.com/7281/16620775488_67a9404ae5_b.jpg
- https://farm8.staticflickr.com/7387/13253318044_52451f07fb_b.jpg
- https://farm2.staticflickr.com/1498/26291663430_ce0e457fb9_b.jpg
- https://farm9.staticflickr.com/8808/28224938305_58ccc2156d_b.jpg
- https://farm6.staticflickr.com/5562/15164856821_9398171e60_b.jpg

Front Page Photos

Client-side sliderMostShared component is passed featuredParks array; each such item has standard_resolution for the image SRC URL and su_name as the ALT text
This comes from latestPhotoFromMostSharedPark query, see app/server/services/queries/parks.js lines 45-65
This looks readily feasible to adapt the query, to join against flickr_photos instead
- There being 1/20th as many photos, perhaps that date filter could be removed as well

Photos for a Specific Park

Client-side app/public/views/park.jsx partial, calls fetchSelectedParkPhotos()
This ultimately lands at a server API endpoint which calls the getSelectedParkPhotos query; see app/server/services/queries/parks.js lines 286-300
This looks readily feasible to adapt the query, to join against flickr_photos instead

Other Usages of instagram_photos TBD

parksNotIn query, app/server/services/queries/parks.js
- Altering this to join to flickr_photos looks to be easy
- Used by the /api/most_shared_parks.json URI, to filter mostSharedParks output against some specified park IDs
- It is not clear that this is used at all
randomPark query, app/server/services/queries/parks.js
- Altering this to join to flickr_photos looks to be easy
- API endpoint /api/wander is called by home page, to populate the WANDER link

gregallensworth commented 6 years ago

Front page: Most Shared Parks image slideshow banner
- reworked to use Flickr photos
- filters to photos sized 1024x768 so aspect ratio is known
- now uses pre-generated table for faster performance
Park details page:
- adjusted to load from Flickr
- slider and selected photo, layouts and CSS adjusted to fit varying sized photos

gregallensworth commented 6 years ago

Most Shared Parks (front page slideshow) re-collected, to avoid selecting Death Valley National Park -- a lot of photos have been deleted, so the front-page slideshow keeps getting a No Such Photo photo selected
Top Ten table re-collected based on Flickr data, so Explore now has updated counts -- e.g. Yosemite reports 14,580 photos instead of 156,660

danrademacher commented 6 years ago

Flickr photos looking good. Will open separate issue for FAQ update

GreenInfo-Network / caliparks.org

Assess impact of IG API changes #665