feat: executor to sagemaker custom container

Goals

The PR enables using an Executor as a sagemaker custom container for inference by implementing 2 endpoints- POST /invocations & GET /ping and accepting a serve command on port 8080.

To separate sagemaker inference, I've added a --provider [NONE|SAGEMAKER] argument to the CLI which can be expanded later to azureml, vertexai etc. When provider is sagemaker, we start a custom fastapi app that implements the GET /ping endpoint.

If /invocations route is already added to the Executor, we add it to the fastapi app.
If /invocations is not added and only one route is added to the Executor, we add an /invocations route that points to the route the Executor implements.
Else raise an error.

Sagemaker needs the image to be pushed to ECR. To use an already pushed Executor with sagemaker, one can do the following and push the image.

# Dockerfile
FROM registry.hubble.jina.ai/myimage

ENTRYPOINT [ "jina", "executor", "--uses", "config.yml", "--provider", "sagemaker" ]

During run, sagemaker adds serve as a CMD to the container entrypoint. I haven't added any arguments to the jina cli for this as serve is ignored (if we follow the above syntax in the entrypoint). While implementing other providers, I'll evaluate the need of the serve command.

End-to-end test

I've manually tested the Executor after pushing a model to S3, the Executor image to ECR & running real-time & serverless inference via sagemaker endpoints.

Pending

Support for batch-transform jobs is not added in this PR, will be followed in another PR.

[ ] check and update documentation. See guide and ask the team.

Codecov Report

Patch coverage: 29.69% and project coverage change: -3.44% :warning:

Comparison is base (d4343d0) 77.14% compared to head (812667b) 73.70%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6046 +/- ## ========================================== - Coverage 77.14% 73.70% -3.44% ========================================== Files 144 143 -1 Lines 13777 13855 +78 ========================================== - Hits 10628 10212 -416 - Misses 3149 3643 +494 ``` | [Flag](https://app.codecov.io/gh/jina-ai/jina/pull/6046/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai) | Coverage Δ | | |---|---|---| | [jina](https://app.codecov.io/gh/jina-ai/jina/pull/6046/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai) | `73.70% <29.69%> (-3.44%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files Changed](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai) | Coverage Δ | | |---|---|---| | [jina/orchestrate/flow/base.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9vcmNoZXN0cmF0ZS9mbG93L2Jhc2UucHk=) | `88.16% <ø> (+1.25%)` | :arrow_up: | | [jina/serve/runtimes/worker/http\_sagemaker\_app.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy93b3JrZXIvaHR0cF9zYWdlbWFrZXJfYXBwLnB5) | `0.00% <0.00%> (ø)` | | | [jina/serve/runtimes/worker/request\_handling.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy93b3JrZXIvcmVxdWVzdF9oYW5kbGluZy5weQ==) | `71.50% <11.76%> (-10.44%)` | :arrow_down: | | [jina/serve/executors/\_\_init\_\_.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9leGVjdXRvcnMvX19pbml0X18ucHk=) | `62.26% <35.29%> (-5.58%)` | :arrow_down: | | [jina/orchestrate/deployments/\_\_init\_\_.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9vcmNoZXN0cmF0ZS9kZXBsb3ltZW50cy9fX2luaXRfXy5weQ==) | `81.27% <42.85%> (+0.43%)` | :arrow_up: | | [jina/serve/runtimes/asyncio.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy9hc3luY2lvLnB5) | `86.95% <71.42%> (-1.66%)` | :arrow_down: | | [jina/serve/runtimes/servers/http.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy9zZXJ2ZXJzL2h0dHAucHk=) | `90.43% <71.42%> (-1.95%)` | :arrow_down: | | [jina/serve/runtimes/servers/\_\_init\_\_.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy9zZXJ2ZXJzL19faW5pdF9fLnB5) | `93.04% <83.33%> (-1.65%)` | :arrow_down: | | [jina/enums.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9lbnVtcy5weQ==) | `88.39% <100.00%> (+0.31%)` | :arrow_up: | | [jina/parsers/orchestrate/pod.py](https://app.codecov.io/gh/jina-ai/jina/pull/6046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9wYXJzZXJzL29yY2hlc3RyYXRlL3BvZC5weQ==) | `100.00% <100.00%> (ø)` | | ... and [38 files with indirect coverage changes](https://app.codecov.io/gh/jina-ai/jina/pull/6046/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jina-ai / jina