Getting Emotional with APIs

[ghost]

Week 2 Step 1 ⬤◯◯◯◯◯◯◯ | 🕐 Estimated completion: 5-20 minutes

Getting Emotional ~ With the Face API

Because of amazing APIs, you don't need to be an expert in machine learning and AI to take advantage of cutting edge technology. In this project, we are going to be building an API and webpage to return you a GIF when you upload a picture of yourself!

✅ Task:

Create a request in Postman to send an image of a person to the Azure Face API to return the subject's emotions

• [x] 1: Create a Face API Endpoint
• [x] 2: Create a POST request in Postman to the Face API to retrieve the emotion attributes of this image
• [x] 🚀 Place the API endpoint and subscription key in the GitHub repository secrets: API_ENDPOINT AND SUBSCRIPTION_KEY
• [x] 🚀 Comment the output of your Postman request to move on

🚧 Test your Work

You should get the following expected output if you have configured your Face API correcty, as well as sent the request with the correct parameters and body.

:white_check_mark: Expected Output

```json { "result": [ { "faceId": "d25465d6-0c38-4417-8466-cabdd908e756", "faceRectangle": { "top": 313, "left": 210, "width": 594, "height": 594 }, "faceAttributes": { "emotion": { "anger": 0, "contempt": 0, "disgust": 0, "fear": 0, "happiness": 1, "neutral": 0, "sadness": 0, "surprise": 0 } } } ] } ```

---

1. The Face API

The Face API will accept the image and return information about the face, specifically emotions. Watch this video on Microsoft Cognitive Services for an in-depth explanation: http://www.youtube.com/watch?v=2aA8OEZ1wk8

❓ How do I create and access the Face API?

1. Log into your Azure portal 2. Navigate to **Create a Resource**, the **AI + Machine Learning** tab on the left, and finally select **Face** and fill out the necessary information 3. Record and save the API endpoint and [subscription key](https://docs.microsoft.com/en-us/azure/api-management/api-management-subscriptions) 4. Place the API endpoint and subscrition key in the GitHub repository secrets: `API_ENDPOINT` AND `SUBSCRIPTION_KEY` * These keys will be used in the Azure function to give access to this API

❓ Where can I find the Face API keys?

1. Navigate to the home page on the Micrsoft Azure portal (https://portal.azure.com/#home) 2. Click on the resource you need the keys for 3. On the left menu bar, locate the Resource Management section and click on "Keys and Endpoint"

2. Using Postman to Send a Request

Now, we can test if our API is working using Postman. Make sure to pay close attention to the documentation and the API Reference

Request URL

Request URL is used when a web client makes a request to a server for a resource. Notice that the request url listed in the API reference is this:

https://{endpoint}/face/v1.0/detect[?returnFaceId]\[&returnFaceLandmarks]\[&returnFaceAttributes]\[&recognitionModel]\[&returnRecognitionModel][&detectionModel]

Parameters

Parameters are typically used in requests to APIs to specify settings or customize what YOU want to receive.

❓ What are the parameters for the request?

The Request URL has the following parameters in [ ]: * [?returnFaceId] * [&returnFaceLandmarks] * [&returnFaceAttributes] * [&recognitionModel] * [&returnRecognitionModel] * [&detectionModel] Important things to note: - All of the bracketed sections represent possible request parameters - Read through **Request Parameters** section carefully - How can we specify that we want to get the emotion data? - All of the parameters are **optional** - We can delete the parameters we don't need in our request - Your **request URL** only requres one parameter, with a specific value - Between `detect` and your parameter, add `?` - *If you had more than one parameter,* you would need to place `&` between each (but not between `detect` and your first parameter) - Since we only have one parameter, no `&` are needed :bulb: [**All of this is located in the documentation! Find this section to read more:**](https://westus.dev.cognitive.microsoft.com/docs/services/563879b61984550e40cbbe8d/operations/563879b61984550f30395236) 

Request Headers

Request Headers tell the receiving end of the request what type of data is in the body.

❓ How do I specify Request Headers?

- Go back to the Face API documentation [here](https://westus.dev.cognitive.microsoft.com/docs/services/563879b61984550e40cbbe8d/operations/563879b61984550f30395236), and find the **Request headers** section. - The `Content-Type` header should be set to`application/octet-stream`. This specifies a binary file. - The `Ocp-Apim-Subscription-Key` header should be set to one of your two keys from your Face API resource. - Request headers are **not** part of the request URL. They are specified in the Postman headers tab:

Request Body

The body of a POST request contains the data you are sending.

❓ How do I send the image in the body of the POST request?

Go to the **body** tab of your Postman request and select **binary**: Next, just upload the [image](https://images.unsplash.com/photo-1507003211169-0a1dd7228f2d?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&ixlib=rb-1.2.1&auto=format&fit=crop&w=934&q=80) and send your POST request.

[HyperPandaz]

[ { "faceId": "c1d4ddcd-7e66-478e-915d-f9d88875e99e", "faceRectangle": { "top": 240, "left": 237, "width": 397, "height": 586 }, "faceLandmarks": { "pupilLeft": { "x": 347.0, "y": 480.4 }, "pupilRight": { "x": 527.5, "y": 475.1 }, "noseTip": { "x": 442.2, "y": 607.0 }, "mouthLeft": { "x": 347.4, "y": 667.3 }, "mouthRight": { "x": 537.5, "y": 661.0 }, "eyebrowLeftOuter": { "x": 278.8, "y": 442.7 }, "eyebrowLeftInner": { "x": 390.0, "y": 438.7 }, "eyeLeftOuter": { "x": 315.2, "y": 481.0 }, "eyeLeftTop": { "x": 349.3, "y": 468.9 }, "eyeLeftBottom": { "x": 345.0, "y": 489.8 }, "eyeLeftInner": { "x": 378.3, "y": 481.9 }, "eyebrowRightInner": { "x": 482.5, "y": 437.2 }, "eyebrowRightOuter": { "x": 594.1, "y": 436.7 }, "eyeRightInner": { "x": 496.8, "y": 480.1 }, "eyeRightTop": { "x": 524.7, "y": 462.2 }, "eyeRightBottom": { "x": 529.0, "y": 484.8 }, "eyeRightOuter": { "x": 559.4, "y": 473.4 }, "noseRootLeft": { "x": 411.6, "y": 492.8 }, "noseRootRight": { "x": 463.3, "y": 492.8 }, "noseLeftAlarTop": { "x": 398.1, "y": 566.7 }, "noseRightAlarTop": { "x": 482.8, "y": 563.7 }, "noseLeftAlarOutTip": { "x": 379.2, "y": 601.5 }, "noseRightAlarOutTip": { "x": 501.6, "y": 599.8 }, "upperLipTop": { "x": 447.2, "y": 664.3 }, "upperLipBottom": { "x": 444.3, "y": 677.9 }, "underLipTop": { "x": 444.7, "y": 706.2 }, "underLipBottom": { "x": 445.0, "y": 730.9 } } } ]

[ghost]

Week 2 Step 2 ⬤⬤◯◯◯◯◯◯ | 🕐 Estimated completion: 5-20 minutes

Getting Emotional ~ With Parse-Multipart

Create an Azure function that takes in an image from a HTTP request, parses it with parse-multipart, and returns the base64 code of the image

✅ Task:

• [x] Create a new branch called emotionalgifs
• [x] 1: Create a new Azure Function called emotionalgifs with the HTTP Trigger template, install and use parse-multipart to parse an image from an HTTP request, and save its raw data in a variable
• [x] 2: Return the image data encoded in base64 in the body of the Azure Function response
• [x] 🚀 Place your function URL in a repository secret called EMOTIONAL_ENDPOINT and commit your code to emotionalgifs/index.js on your emotionalgifs branch
• [x] 🚀 Create a pull request that merges emotionalgifs to main, but do not merge it

🚧 Test your Work

Use Postman! Paste the function url and make a POST request. Remember to attach the file in Body! In the output box, you should get the output. Make sure you're using an image with a real face on it or else it won't work. Here's an example of an output I get with this image:

:white_check_mark: Expected Output

The output should be the base64 code of the inputted image, like this: ```base64 /9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQIC... ```

:question: Confused about Postman?

1. Navigate back to the Postman app and change GET to POST 2. Publish/deploy your function and copy your function url from the VS Code output like this: 3. Use the function url and any image you want to send the POST request. Remember to attach the file in Body! 

---

1. Create emotionalgifs HTTP trigger function that parses images

What is multipart request?

A HTTP multipart request is a HTTP request that HTTP clients construct to send files and data over to a HTTP Server.

💡 This is commonly used by browsers and HTTP clients to upload files to the server.

Because we want to send an image file through an HTTP Request, we need a piece of software to parse the raw data to extract the image file. Here comes the handy NPM package: parse-multipart!

The raw payload formatted as multipart/form-data will looks like this (expand)

``` ------WebKitFormBoundaryDtbT5UpPj83kllfw Content-Disposition: form-data; name="uploads[]"; filename="somebinary.dat" Content-Type: application/octet-stream some binary data...maybe the bits of a image.. ------WebKitFormBoundaryDtbT5UpPj83kllfw Content-Disposition: form-data; name="uploads[]"; filename="sometext.txt" Content-Type: text/plain hello how are you ------WebKitFormBoundaryDtbT5UpPj83kllfw-- ```

The lines above represents a raw multipart/form-data payload sent by some HTTP client via form submission containing two files. We need to extract all the files contained inside it. The multipart format allows you to send more than one file in the same payload, so that's why it is called multipart.

:package: Installing `parse-multipart`

Before you can install `parse-multipart`, you need to enter `npm init -y` into the terminal. This command allows us to set up a new npm package:

[Open up a terminal in VSCode](https://code.visualstudio.com/docs/editor/integrated-terminal) inside your function's directory, type `npm install parse-multipart`, and press enter. > :bulb: Forgot how to navigate a terminal? [Check this out.](https://computers.tutsplus.com/tutorials/navigating-the-terminal-a-gentle-introduction--mac-3855) **Note:** the text outputted by the console does not mean there was an error! The npm package has successfully been installed.

How do I use this package?

First, we need to declare the variable `multipart` outside of the async function so that we can access the NPM package: ```js var multipart = require('parse-multipart'); ``` Notice that `multipart.Parse(body, boundary)` requires two arguments, as it has two parameters. I've already gotten the boundary for you – just like the documentation example, our boundary is a string in the format `"----WebKitFormBoundary(random characters here)"`. In the `multipart.Parse()` call, you need to figure out what the body parameter should be. > :bulb: **Hint:** It should be the request body. Think about the template HTTP Trigger Azure function. How did we access the body in there? ```js // here's your boundary: var boundary = multipart.getBoundary(req.headers['content-type']); // TODO: assign the body variable the correct value var body = '' // parse the body var parts = multipart.Parse(body, boundary); ```

2. 🖼️ Receiving the image

When the POST request is made to this function with an image, we need to:

1. Get the image from the parts (parse-multipart) variable
2. Convert the image to base64
3. Store the converted image in the response body

:question: What parts of the template code do we need?

Take a look at the standard `module.exports` function code: ```js module.exports = async function (context, req) { // the code } ``` This is the function that runs **every time your HTTP trigger gets a request**. As a parameter of this function, the `req` parameter contains all the information the request was sent with. *This contains*: * Headers * The body Remove all of the content in `module.exports` except this: ```js context.res = { // status: 200, /* Defaults to 200 */ body: //LEAVE THIS BLANK }; ```

:question: How do we output the image in base64?

Next, we want to output the **base64** code of the inputted image. The parsed image data that we need to convert to base64 is is stored in index 0 of `parts` since we only sent one file, and we want the data property of this image to access the binary file. Thus, we will be converting `parts[0].data` to base64 and assigning the code to a new variable: ```javascript var convertedResult = Buffer.from(parts[0].data).toString('_____'); // FILL IN THE BLANK ``` The `Buffer` part of the code provides **temporary storage** for the binary image data as it is converted to **base64**. Now, complete the following so that the **base64** code is outputted when the function is called: ```js context.res = { // status: 200, /* Defaults to 200 */ body: //WHAT GOES HERE? }; ```

[ghost]

🎭 Getting Emotional ~ Complete!

Pat yourself on the back.