spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
491 stars 43 forks source link

How to copy files in dockerfile without root build context #93

Open himat opened 3 years ago

himat commented 3 years ago

I'm trying to use spotty for the first time but I ran into an error after running spotty start and it tried to build my docker container:

 Step 2/5 : COPY requirements.txt requirements.txt
 COPY failed: file not found in build context or excluded by .dockerignore: stat requirements.txt: file does not exist

My directory structure

proj/
  docker/
    models.Dockerfile
  src/*
  requirements.txt
  spotty.yaml

Here's my dockerfile: docker/models.Dockerfile

FROM python:3.8.7-buster

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

Here's my spotty config: spotty.yaml

project:
  name: model-server
  syncFilters:
    - exclude:
        - .idea/*
        - .git/*
        - .mypy_cache/*
        - '*/__pycache__/*'
        - '*/*.swp'
containers:
  - projectDir: /workspace/project
    file: docker/models.Dockerfile
    ports:
      # Server
      - containerPort: 5000
        hostPort: 5000
    volumeMounts:
      - name: workspace
        mountPath: /workspace
instances:
  - name: model-server-inst
    provider: aws
    parameters:
      region: us-east-2
      instanceType: t2.small
      dockerDataRoot: /
      volumes:
        - name: workspace
          parameters:
            size: 50
            deletionPolicy: retain
        - name: docker
          parameters:
            size: 10
            mountDir: /docker
            deletionPolicy: retain

I saw in the docs that Note: Spotty uses the directory with the Dockerfile as its build context. So this seems like why I ran into this issue since the requirements.txt file is not present in the docker context. And indeed, the command that spotty ran was docker build -t spotty-model-server-model-server-inst-default:1616462585 -f /mnt/model-server-model-server-inst-workspace/project/docker/models.Dockerfile /mnt/ai-writer-model-server-ai-writer-model-server-inst-workspace/project/docker whereas when I build my dockerfile locally, the last argument (the context) I would give to docker build is my root directory (/project), not /project/docker.

So is there a way to get around this, or does using spotty mean that I need to keep my dockerfile in my root directory? Or is there something I'm doing wrong.

apls777 commented 3 years ago

So is there a way to get around this, or does using spotty mean that I need to keep my dockerfile in my root directory? Or is there something I'm doing wrong.`

Yes, you either should put the Dockerfile into the root directory of your project or copy the requirements.txt file inside the docker/ directory. In the first case, use a .dockerignore file to ignore potentially heavy directories that may slow down the building.

himat commented 3 years ago

Would it be possible to add a way to specify a root build context? My project has multiple components/servers each of which has their own dockerfile, which is why I put them all in a docker/ dir in the root of my project. So putting all the dockerfiles in the root isn't that pleasant, nor would copying the requirements.txt into that folder manually make much sense.

apls777 commented 3 years ago

Yes, it's possible, but I'm not sure when I will be able to have a look.

himat commented 3 years ago

Not critical actually, I did what you said of putting the dockerfile in the root of the project. Thanks!