allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
381 stars 131 forks source link

Error when starting docker #56

Open sagyHarpaz opened 4 years ago

sagyHarpaz commented 4 years ago

Hi when I tried to run "docker-compose -f docker-compose-win10.yml up" I got: Unexpected API error for trains-redis (HTTP code 502) Response body: Bad response from Docker engine

Unexpected API error for trains-webserver (HTTP code 502) Response body: Bad response from Docker engine

Unexpected API error for trains-elastic (HTTP code 502) Response body: Bad response from Docker engine

Unexpected API error for trains-apiserver (HTTP code 502) Response body: Bad response from Docker engine

Unexpected API error for trains-fileserver (HTTP code 502) Response body: Bad response from Docker engine

bmartinn commented 4 years ago

Hi @sagyHarpaz From the error codes all dockers are not running, and it seems like there is an issue with the docker service for windows setup (not specifically the trains-server dockers). Did you follow the Windows 10 instructions here ? Notice step (2) increasing the default memory for the docker service.

sagyHarpaz commented 4 years ago

I always got this message pymongo.errors.ServerSelectionTimeoutError: mongo:27017: [Errno -2] Name or service not known

jkhenning commented 4 years ago

@sagyHarpaz can you see if the trains-mongo docker is up? Assuming you're using Windows, you can open the Docker-Desktop dashboard and see the status of the container.

You can also use the docker ps command to get a list of the containers and see the status of trains-mongo - do that several times and see if the container is up, or keeps restarting.

Also, assuming there's something wrong with the container, use docker logs trains-mongo to get the full container logs - please share it so we can better understand the issue.

sagyHarpaz commented 4 years ago

When I ran docker ps command I got: image and when Iran docker logs trains-mongo I got log.txt

jkhenning commented 4 years ago

@sagyHarpaz did you check the Docker Desktop's sharing option? Check Docker Desktop\Settings\Resources\File Sharing and see if the data folder you use in the docker-compose is listed there. If not, try adding it and restart Trains Server (docker-compose down and than docker-compose up)

sagyHarpaz commented 4 years ago

image

jkhenning commented 4 years ago

Can you check what is your Docker Desktop version? In the past, we experienced issues with specific versions. Our last test was with version 2.3.0.4 (46911) which is still the latest.

In any case, I suggest trying to update to the latest version and try again 🙂

sagyHarpaz commented 4 years ago

I already have the latest version 2.3.0.4 (46911)

jkhenning commented 4 years ago

OK, this might be a volume issue, try the following:

Edit the docker-compose file, change the mongo service' volumes and add a new volume at the end of the file:

services:
  ...
  mongo:
        volumes:
        - mongodata:/data/db
        - c:/opt/trains/data/mongo/configdb:/data/configdb

...

volumes:
  mongodata:

Then do:

docker-compose down
docker volume create --name=mongodata
docker-compose up

BTW - seems like a known mongo issue on Windows: https://github.com/docker-library/mongo/issues/385

sagyHarpaz commented 4 years ago

now I get this error elasticsearch.exceptions.RequestError: RequestError(400, 'action_request_validation_exception', 'Validation Failed: 1: template is missing;')

evg-allegro commented 4 years ago

Hi @sagyHarpaz, can you please share the logs from trains-elastic and trains-apiserver dockers? I suspect that we will see some errors there. To get the elasticsearch logs into a file run: sudo docker logs trains-elastic >& elastic.logs For the apiserver: sudo docker logs trains-apiserver >& apiserver.logs