deis / workflow-cli

The CLI for Deis Workflow
http://deis.com
MIT License
31 stars 43 forks source link

Underscore in Procfile process name not allowed? #247

Closed vladi-dev closed 7 years ago

vladi-dev commented 7 years ago

After adding a new background worker in Procfile called chat_save I deployed the app with git push deis master. Tried to scale new worker with deis scale chat_save=1 and get this error. Error: 'chat_save=1' does not match the pattern 'type=num', ex: web=2

Changed chat_save to chatsave in Procfile. Now when deploying I'm getting the error below.

remote: 2016/09/16 23:43:03 Error running git receive hook [publishing release (Unknown Error (400): {"detail":"(app::deploy): (app::deploy): ('failed to create Deployment \"stage-api-chat_save\" in Namespace \"stage-api\": 422 status code 422', 'stage-api-chat_save', 'stage-api')"})]
To ssh://git@deis-builder.myapp.io:2222/stage-api.git
 ! [remote rejected] stage -> stage (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@deis-builder.myapp.io:2222/stage-api.git'

Is the underscore in worker process the reason of those errors?

helgi commented 7 years ago

Can you find the exception in the Controller? There should be more information. Looks like Deployment in Kubernetes isn't happy with the underscore, even if in the past that's been totally fine

What is your CLI version? and Workflow version

vladi-dev commented 7 years ago

deis-cli: v2.4.0 workflow: v2.5.0

vladi-dev commented 7 years ago

Controller logs. The thing is that i removed chat_save from Procfile in the repo I'm pushing.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/models/app.py", line 578, in deploy
    async_run(tasks)
  File "/app/api/utils.py", line 169, in async_run
    raise error
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/app/api/utils.py", line 182, in async_task
    yield from loop.run_in_executor(None, params)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/scheduler/__init__.py", line 175, in deploy
    namespace, name, image, entrypoint, command, **kwargs
  File "/app/scheduler/resources/deployment.py", line 118, in create
    'create Deployment "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to create Deployment "stage-api-chat_save" in Namespace "stage-api": 422 status code 422', 'stage-api-chat_save', 'stage-api')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/models/app.py", line 585, in deploy
    self.deploy(release.previous(), force_deploy=True, rollback_on_failure=False)
  File "/app/api/models/app.py", line 595, in deploy
    raise ServiceUnavailable(err) from e
api.exceptions.ServiceUnavailable: (app::deploy): ('failed to create Deployment "stage-api-chat_save" in Namespace "stage-api": 422 status code 422', 'stage-api-chat_save', 'stage-api')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/models/build.py", line 64, in create
    self.app.deploy(new_release)
  File "/app/api/models/app.py", line 595, in deploy
    raise ServiceUnavailable(err) from e
api.exceptions.ServiceUnavailable: (app::deploy): (app::deploy): ('failed to create Deployment "stage-api-chat_save" in Namespace "stage-api": 422 status code 422', 'stage-api-chat_save', 'stage-api')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 471, in dispatch
    response = handler(request, *args, **kwargs)
  File "/app/api/views.py", line 503, in create
    super(BuildHookViewSet, self).create(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/app/api/viewsets.py", line 21, in perform_create
    self.post_save(obj)
  File "/app/api/views.py", line 509, in post_save
    build.create(self.user)
  File "/app/api/models/build.py", line 71, in create
    raise DeisException(str(e)) from e
api.exceptions.DeisException: (app::deploy): (app::deploy): ('failed to create Deployment "stage-api-chat_save" in Namespace "stage-api": 422 status code 422', 'stage-api-chat_save', 'stage-api')
10.244.2.11 "POST /v2/hooks/build/ HTTP/1.1" 400 182 "deis-builder"
helgi commented 7 years ago

I'd recommend upgrading to the latest CLI - as for the logs, thanks! Will have a look through that

vladi-dev commented 7 years ago

I will thanks. Would you please recommend a way how to fix my deployment? Tried to recreate deis-controller, deis-builder but it still doesnt help. I get the same error from git receive hook.

helgi commented 7 years ago

What happens if you try to scale chat_save down to 0?

vladi-dev commented 7 years ago

$ deis scale chat_save=0 Error: 'chat_save=0' does not match the pattern 'type=num', ex: web=2

and i don't have this process when i deis ps

helgi commented 7 years ago

So only alphanumeric is allowed, previously we only allowed alpha (https://github.com/deis/controller/commit/7392eb324a84bf631e670f0128d7e6d1ae886983)

Could you delete the deployment? deis delete deployment appname-chat_save --namespace appname ?

vladi-dev commented 7 years ago

So I found a way to fix this without reinstalling deis. Basically I had to change deis database record for proccesses defined in Procfile.

  1. Login to deis database container. kubectl --namespace=deis exec -it deis-database-452r2
  2. Connect to deis postgres database. psql -U $(cat /var/run/secrets/deis/database/creds/user)
  3. Find your app structure select * from api_app; it is stored in format process_name: scale_number. It would be something like this {"web":1,"chat_save":0,"beat":0,"braintree" :1,"celery":1} and my app name is in id column -- id=stage-api.
  4. I had to remove chat_save string from structure so it became as such: {"web":1,"beat":0,"braintree" :1,"celery":1}.
  5. Run sql:UPDATE api_app SET structure='{"web":1,"beat":0,"braintree" :1,"celery":1}' WHERE id='stage-api';
  6. Same thing for api_build table. select * from api_build where procfile ilike '%save_chat%';
  7. Now git push stage-api master works. Procfile now has chatsave without underscore :)
kmala commented 7 years ago

fixed in deis/controller#1105