Cloud image analysis pipeline

[mbod]

From images pulled from a timeline (image stored locally? metadata in Mongo?) adapt instagram WWE image analysis code to submit to one or more cloud image coders and store result in Mongo

[mbod]

@atnjqt thinking through the steps here...

Steps:

1. On login pass token through to flask app for use in backend API. Store (in mongo? encrypted?)
2. Set off post collection process
   • get user posts (page through results)
   • check if post already processed previous (i.e. in mongo)
3. Process post
   • focusing on posts that contain images (need to see if there is overlap between user photos and posts with photos)
   • download image to processing hot folder
     • this folder will be watched and run the cloud image rec steps and interact with database
   • one image processed and logged in db filter on tagged features and rank photo as candidate for use

[mbod]

Were we thinking Azure for this?

[atnjqt]

Thanks for the note, some helpful resources for Facebook Graph API for Photos:

• specifically to get your photos, this is me/photos or {user-id}/photos, for more information see https://developers.facebook.com/docs/graph-api/reference/photo/

  • important that me/photos shows photos that my friends tagged me in... for your own photos, this is me/photos/uploaded, what we are currently implementing, see: https://github.com/atnjqt/compassionapp/blob/aac8f807cdfc10a74ee1da2af73afddd4b7a997e/frontend/src/Components/FacebookGetPhotos.js#L22

• from there we can pass photo IDs at {photo-ID}/?accesToken=....?fields=webp_images, more info here: https://developers.facebook.com/docs/graph-api/reference/photo/#Reading

  • significantly the Fields returned contain information available about the photo, in this case we want a link served as webp_image. More info see here: https://developers.facebook.com/docs/graph-api/reference/photo/#fields
  • see the line here: https://github.com/atnjqt/compassionapp/blob/aac8f807cdfc10a74ee1da2af73afddd4b7a997e/frontend/src/Components/FacebookPhotos.js#L22

[atnjqt]

On quick review, Microsoft Azure provides an npm package for integrating image analysis, we want cognitiveservices-face but there is a whole suite of cog services available.

• Most notably we want to review https://docs.microsoft.com/en-us/javascript/api/overview/azure/cognitive-services?view=azure-node-latest#face-api
• There is a whole quickstart guide here: Quickstart: Detect faces in an image using the REST API and JavaScript

  cd frontend
  npm install @azure/cognitiveservices-face --package-lock-only
  npm update

Microsoft has a few examples for integrating Cognitive Services into React apps as well:

• https://docs.microsoft.com/en-us/azure/cognitive-services/face/build-enrollment-app
• https://github.com/microsoft/cognitive-services-sdk-react-native-example