Okteto CLI - connection seems very slow, app takes 3 minutes to start

[marcindruzgala]

Describe the bug Hi!

So we wanted to use your awesome CLI tool to enable cloud development on local machines with 'okteto up' to deploy our local files to the cluster. We've managed to achieve that but it seems that everything is super slow (I have worked with okteto also in my previous company and I didn't experience anything like that).

Yaml file is very simple and looks like that:

name: identity-platform
image: okteto/dotnetcore:6
command: ["dotnet", "watch", "--project", "IdentityPlatform", "--", "run", "--configuration", "Debug", "--launch-profile", "okteto"]
workdir: /src
namespace: dev-marcin
remote: 2222
forward:
  - 8080:80
  - 5443:rds-aws:5432
  - 6380:dev-marcin-redis-master:6379
persistentVolume:
  enabled: false

Okteto is quite fast to replace the deployment and it shows the information about context, namespace and forwarded ports but then execution of the command takes 3-5 minutes. The same command takes more or less few seconds locally so it's not like the application takes so long to start :)

Any ideas what that might be?

Desktop (please complete the following information):

• OS: mac M1 Monterey 12.5 (tested also on windows 10 - same result)

[jLopezbarb]

We haven't been reported any problems with command execution times nor have we been able to reproduce it. Could you provide an example repo where there is no sensitive information and where we can reproduce the excessive execution times?

[marcindruzgala]

That's the problem, I can't really reproduce it (what I mean is that in my previous company I was using it and everything was super smooth, in the new one it isn't - I tested on different clusters on AWS and different computers with the same result..) and I don't think it is okteto cli that is to 'blame' here.

But if I could ask, what can affect the performance here? Can it be corporate network that is somehow affecting ssh connection (I have no idea if it even makes sense).

Is there any way that you suggest I debug this issue?

[jLopezbarb]

If the problem is when the command starts, it is most likely a resource issue. Could it be that there are not enough resources in the deployment(resource limits)? You can try to avoid those issues by requesting more resources using the field resources in the manifest.

[pchico83]

@marcindruzgala did you try updating the resources of your dev container?

[marcindruzgala]

Hi,

Yes I did (just now). I took the resources request and limits from the example but it didn't help :(

[pchico83]

The resources in the sample are quite small, could you try with:

resources:
  requests:
    cpu: "1"
    memory: "8Gi"
  limits:
    cpu: "2"
    memory: "12Gi"

for example? Also, check the manifest of the dev pod to see if those values are applied.

[marcindruzgala]

@pchico83 Using values your provided sped it up a lot...now it takes only 6-8 seconds. But I'm wondering...1CPU and 8gb of ram is a lot...our application is using ~30mb of memory only.

[pchico83]

@marcindruzgala those values depend on your app, adjust them as needed. Booting an application usually require more resources than running it. I would recommend you to read about kubernetes resources and how they work. Those resources are not 100% dedicated to container: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

[pchico83]

@marcindruzgala closing this one as it's related to cpu/memory dedicated to the dev container

[scheung38]

The resources in the sample are quite small, could you try with:

resources:
  requests:
    cpu: "1"
    memory: "8Gi"
  limits:
    cpu: "2"
    memory: "12Gi"

for example? Also, check the manifest of the dev pod to see if those values are applied.

Hi @pchico83 if we do not specify the resource limits in the build section then what are we getting? Every time I do okteto deploy --build --wait takes around 14 mins for a Django app with mongo

[pchico83]

@scheung38 build happens with 2CPUs and 8Gi of memory. You probably need to optimize your Dockerfile

[scheung38]

Dockerfile is coming (autogenerated using best practice from my understanding) from cookiecutter-django so it is already using multi stage.

compose/production/django/Dockerfile

ARG PYTHON_VERSION=3.10-slim-bullseye

# define an alias for the specfic python version used in this file.
FROM python:${PYTHON_VERSION} as python

# Python build stage
FROM python as python-build-stage
# FROM okteto.dev/arthur_paf_production_django:latest

ARG BUILD_ENVIRONMENT=production

# Install apt packages
RUN apt-get update && apt-get install --no-install-recommends -y \
  # dependencies for building Python packages
  build-essential \
  # psycopg2 dependencies
  libpq-dev

# Requirements are installed here to ensure they will be cached.
COPY ./requirements .

# Create Python Dependency and Sub-Dependency Wheels.
RUN pip wheel --wheel-dir /usr/src/app/wheels  \
  -r ${BUILD_ENVIRONMENT}.txt

# Python 'run' stage
FROM python as python-run-stage

ARG BUILD_ENVIRONMENT=production
ARG APP_HOME=/app

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV BUILD_ENV ${BUILD_ENVIRONMENT}

WORKDIR ${APP_HOME}

RUN addgroup --system django \
  && adduser --system --ingroup django django

# Install required system dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
  curl \
  # psycopg2 dependencies
  libpq-dev \
  # Translations dependencies
  gettext \
  # cleaning up unused files
  && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
  && rm -rf /var/lib/apt/lists/*

# devtron - install nginx
#RUN apt-get install nginx vim -y --no-install-recommends
#COPY nginx.default /etc/nginx/sites-available/default
#RUN ln -sf /dev/stdout /var/log/nginx/access.log \
#    && ln -sf /dev/stderr /var/log/nginx/error.log

# devtron- start server
#EXPOSE 8000

# All absolute dir copies ignore workdir instruction. All relative dir copies are wrt to the workdir instruction
# copy python dependency wheels from python-build-stage
COPY --from=python-build-stage /usr/src/app/wheels  /wheels/

# use wheels to install python dependencies
RUN pip install --no-cache-dir --no-index --find-links=/wheels/ /wheels/* \
  && rm -rf /wheels/

COPY --chown=django:django ./compose/production/django/entrypoint /entrypoint
RUN sed -i 's/\r$//g' /entrypoint
RUN chmod +x /entrypoint

COPY --chown=django:django ./compose/production/django/start /start
RUN sed -i 's/\r$//g' /start
RUN chmod +x /start

# copy application code to WORKDIR
COPY --chown=django:django . ${APP_HOME}

# make django owner of the WORKDIR directory as well.
RUN chown django:django ${APP_HOME}

USER django

ENTRYPOINT ["/entrypoint"]

[pchico83]

@scheung38 which steps in your build process are taking more time?

[scheung38]

Tried to add at various indentations but still?

x Invalid manifest:

• line 26: field resources not found in type model.buildInfoRaw See https://okteto.com/docs/reference/manifest/ for details

[pchico83]

@scheung38 resources is not supported in the build section

[scheung38]

We place inside k8s manifest I think, but if working with docker-compose then everytime we convert into manifests

kompose --file production.yml -o $HOME/Documents/arthur_paf/manifests convert

this resource definition would be overwritten?

[scheung38]

This takes the longest:

4/21

RUN apt-get update && apt-get install --no-install-recommends -y build-essential libpq-dev

[kylealwyn]

loading build context taking years for me as well for a turborepo based monorepo 🤔 on mac m1 monterey 12.6

**recognize context is quite large and need to optimize