feat: multimedia request/response

[imsergiy]

PR checklist

Please tick all of the below (if a certain item is not required for this PR please tick the box anyway)

Docs

• [x] If you added new code; is it self documenting? (all the variable names clear, comments added where appropriate)
• [x] If you touched existing code; did you refactor existing naming, added comments where necessary?

Tests

• [x] If your ticket adds new functionality; have you added unit tests to verify behaviour?
• [ ] If your ticket alters user interaction with the UI; have you added integration tests?
• [ ] Do your tests check for every convievable user behabviour? (all behaviours the user can perform)

Typing

• [x] If you added new code; have you added typing?
• [x] If you touched existing code; have you improved its typing?

Styling

• [x] If new styles were created; Did you refactor/add more styles in css-in-js style?
• [x] Are all fonts, sizes, colours, margins, paddings in line with designs?
• [x] Did you perform a visual comparison of the implementation with the designs?

Added in this PR:

• request formatter
• multimedia requests
• multimedia responses

[codecov[bot]]

Codecov Report

Merging #339 (3dd1f39) into master (f3f1fe5) will increase coverage by 0.81%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #339      +/-   ##
==========================================
+ Coverage   67.17%   67.99%   +0.81%     
==========================================
  Files          62       62              
  Lines        1447     1484      +37     
  Branches      221      230       +9     
==========================================
+ Hits          972     1009      +37     
  Misses        475      475              

Impacted Files	Coverage Δ
src/helpers/format.ts	96.92% <100.00%> (+1.22%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2ccde0e...3dd1f39. Read the comment docs.

[Roshanjossey]

I like the UI :heart:

[nan-wang]

@imsergiy There is a bug when an empty Flow is given. Please run python app.py from the branch feat-empty-flow with

{"data": ["hello"], "parameters": {"mode": "audio"}}
```. 
![image](https://user-images.githubusercontent.com/4329072/121607538-ca170e80-ca82-11eb-998c-a0111da16a6b.png)

[nan-wang]

Can we support customized APIs? With 2.0, users can use customized APIs. To be specific, beside localhost/search and localhost/index, one can use localhost:45678/post/my_customized_api as well. I've created another api to check out this function. Given 127.0.0.1:45678/post/my_customized_api should return the following results

curl --request POST -d '{"data": [{"text": "hello world"}], "parameters": {"mode": "text"}}' -H 'Content-Type: application/json' http://localhost:45678/post/my_customized_api

{
  "request_id": "c02fb2f4-7273-4d7a-ad38-8e9c90abd75c",
  "data": {
    "docs": [
      {
        "id": "a5d3fc94-ca42-11eb-bb97-8c8590467aa7",
        "chunks": [
          {
            "id": "a79fd188-ca42-11eb-aab6-8c8590467aa7",
            "matches": [
              {
                "id": "a83bb33c-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_0",
                "granularity": 1,
                "score": {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              },
              {
                "id": "a83e29f0-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_1",
                "granularity": 1,
                "score": {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              },
              {
                "id": "a83e2f9a-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_2",
                "granularity": 1,
                "score": {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              },
              {
                "id": "a83e3260-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_3",
                "granularity": 1,
                "score": {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              }
            ],
            "mime_type": "text/plain",
            "text": "chunk_0",
            "granularity": 1,
            "parent_id": "a5d3fc94-ca42-11eb-bb97-8c8590467aa7",
            "content_hash": "ac17ad4d1eea81e9"
          },
          {
            "id": "a79fd778-ca42-11eb-aab6-8c8590467aa7",
            "matches": [
              {
                "id": "a83e46c4-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_0",
                "granularity": 1,
                "score": {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              },
              {
                "id": "a83e49bc-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_1",
                "granularity": 1,
                "score": {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              },
              {
                "id": "a83e4ca0-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_2",
                "granularity": 1,
                "score": {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              },
              {
                "id": "a83e4f52-ca42-11eb-8ec4-8c8590467aa7",
                "mime_type": "text/plain",
                "text": "match_3",
                "granularity": 1,
                "score": {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                "adjacency": 1
              }
            ],
            "mime_type": "text/plain",
            "text": "chunk_1",
            "granularity": 1,
            "parent_id": "a5d3fc94-ca42-11eb-bb97-8c8590467aa7",
            "content_hash": "adedb4c0da5a123e"
          }
        ],
        "matches": [
          {
            "id": "a8da85d4-ca42-11eb-b25e-8c8590467aa7",
            "mime_type": "text/plain",
            "text": "match_0",
            "score": {
              "op_name": "doc_matcher",
              "description": "score for doc",
              "operands": [
                {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                }
              ],
              "ref_id": "a5d3fc94-ca42-11eb-bb97-8c8590467aa7"
            },
            "adjacency": 1
          },
          {
            "id": "a8daa32a-ca42-11eb-b25e-8c8590467aa7",
            "mime_type": "text/plain",
            "text": "match_1",
            "score": {
              "value": 1.0,
              "op_name": "doc_matcher",
              "description": "score for doc",
              "operands": [
                {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                }
              ],
              "ref_id": "a5d3fc94-ca42-11eb-bb97-8c8590467aa7"
            },
            "adjacency": 1
          },
          {
            "id": "a8dab54a-ca42-11eb-b25e-8c8590467aa7",
            "mime_type": "text/plain",
            "text": "match_2",
            "score": {
              "value": 2.0,
              "op_name": "doc_matcher",
              "description": "score for doc",
              "operands": [
                {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd188-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.1,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.2,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                },
                {
                  "value": 0.3,
                  "op_name": "chunk_matcher",
                  "description": "score for chunk",
                  "ref_id": "a79fd778-ca42-11eb-aab6-8c8590467aa7"
                }
              ],
              "ref_id": "a5d3fc94-ca42-11eb-bb97-8c8590467aa7"
            },
            "adjacency": 1
          }
        ],
        "mime_type": "text/plain",
        "tags": {},
        "text": "what a lovely cat"
      }
    ]
  },
  "header": {
    "exec_endpoint": "my_customized_api"
  },
  "parameters": {
    "mode": "text"
  },
  "routes": [
    {
      "pod": "gateway",
      "pod_id": "4d1ab13d-1c7f-4da0-8cd4-d8c2054ef072",
      "start_time": "2021-06-10T23:21:56.375073Z",
      "end_time": "2021-06-10T23:22:01.450954Z"
    },
    {
      "pod": "pod0/ZEDRuntime",
      "pod_id": "a4b83225-7970-4e11-8909-cec355fca7cf",
      "start_time": "2021-06-10T23:21:56.376669Z",
      "end_time": "2021-06-10T23:21:56.380372Z"
    },
    {
      "pod": "pod1/ZEDRuntime",
      "pod_id": "0f36c186-903f-43cd-a70b-eacf2177909e",
      "start_time": "2021-06-10T23:21:56.381531Z",
      "end_time": "2021-06-10T23:21:59.386734Z"
    },
    {
      "pod": "pod2/ZEDRuntime",
      "pod_id": "42589ace-7fe2-4c2e-b0b9-7a51bac6607c",
      "start_time": "2021-06-10T23:21:59.388483Z",
      "end_time": "2021-06-10T23:22:00.425020Z"
    },
    {
      "pod": "pod3/ZEDRuntime",
      "pod_id": "5545ae1d-b076-43d7-a361-289e85293ddb",
      "start_time": "2021-06-10T23:22:00.434900Z",
      "end_time": "2021-06-10T23:22:01.449721Z"
    },
    {
      "pod": "gateway",
      "pod_id": "3186cf03-4f7f-401e-8974-fdaf2756ce84",
      "start_time": "2021-06-10T23:22:01.450928Z"
    }
  ],
  "status": {}
}

[imsergiy]

• refactored types through debug components
• fixed empty flow issue
• fixed mimetype issue
• added custom endpoint support

[nan-wang]

It is better to show milliseconds here.

[nan-wang]

It will be more helpful if we can show the match.score.value directly here. Meanwhile, we still use the sidebar to show the recursive structure of the match.score.operands

[nan-wang]

When formated is selected, can we allow the user to specify multiple fields? This is a critical feature for the first iteration.

Something like this, there will be two groups of fields, one is sent via data and another is parameters, i.e.

curl --request POST -d '{"data": [{"text": "hello world", "modality": "what_ever_modality_you_like"}], "parameters": {"mode": "text"}}' -H 'Content-Type: application/json' http://localhost:45678/search

[nan-wang]

Another issue is that can we infer the Response Mode automatically? This is something trivial and nice-to-have.

[imsergiy]

This is an example of entering custom fields with the updated UI: This will create the JSON:

{
    "data": [
        {
            "text": "Hello World",
            "mode": "text"
        },
        {
            "uri": "data:image/jpeg;base64,...",
            "color": "green"
        }
    ],
    "parameters": {
        "mode": "video"
    },
    "custom": "parameter"
}

You can choose where to attach your custom field: the root of the request, a data item, or the parameters field. When you select files from your machine, you will be able to see those file names as options to attach custom fields to. The value input of the custom field supports string and JSON. Using JSON, so you can create any kind of nested fields.

Another addition is anything you write in the formatted tab will now be parsed in real time an output in the raw tab. You can see a preview of the request that will be sent and event tweak it before sending it off. Keep in mind this is not bidirectional, editing the raw request will not change what is on the formatted tab and when you edit what is on the formatted tab it will overwrite whatever you changed in the raw tab.

[nan-wang]

@imsergiy Well done! Especially, I like the real-time conversion between raw and formatted. It is super helpful. Two minor comments from my side.

First, it will be great if we can display all the available fields of response.data.docs[i] instead of just id, text and uri.

Second, we would like to see the details in the score sidebar as well. We can also group the operands for better visualization.

Last, the score sidebar needs to be updated for every request. And it is better to enable the user to close it.

[nan-wang]

I found an issue when the fields to display are too long.

[nan-wang]

The score also looks wired

[Kelton8Z]

I'm getting 422 unprocessable entity error here while with the same endpoint on jinabox works 🤔

[nan-wang]

I'm getting 422 unprocessable entity error here while with the same endpoint on jinabox works

@Kelton8Z When raising issues, it is necessary to provide a minimal example or how to reproduce the bug. At least we need the error information to understand the bug.

[Kelton8Z]

I'm getting 422 unprocessable entity error here while with the same endpoint on jinabox works

@Kelton8Z When raising issues, it is necessary to provide a minimal example or how to reproduce the bug. At least we need the error information to understand the bug.

Understood. In this case, the error of Error: Error: Request failed with status code 422 came when inputting text or image to dashboard's debug tool with cross modal example's restful query endpoint localhost:45678/search