search

openvcloud / 0-templates

Apache License 2.0
1 stars 5 forks source link

Retry shouldn't be done for all errors.. Erros need to be filtered #118

Open john-kheir opened 6 years ago

john-kheir commented 6 years ago

After Running a blueprint to create a cloudspace with wrong prameters for instance, I'm getting these logs inside the zrobot server .. I think such kinf of errors should be filtered as i don't need to retry for a 400 bad request for example.

Zrobot server output

400 Bad Request
A resource limit should be a positive number or -1 (unlimited)., Retrying in 3 seconds...
400 Bad Request
A resource limit should be a positive number or -1 (unlimited)., Retrying in 6 seconds...
400 Bad Request
A resource limit should be a positive number or -1 (unlimited)., Retrying in 12 seconds...
400 Bad Request
A resource limit should be a positive number or -1 (unlimited)., Retrying in 24 seconds...
katia-e commented 6 years ago

Currently install actions in VM, Account and VDC templates have retry decorators catching BaseError. Opinion of @john-kheir is that we have to filter errors that can't be solved by retry. @FastGeert, @zaibon, could you give your input on it?

FastGeert commented 6 years ago

This looks like an input validation check that should not be retried, just like John says.

john-kheir commented 6 years ago

@katia-e

Blueprint

  - github.com/openvcloud/0-templates/sshkey/0.0.1__59b99453:
       name: '63b8de28'
       dir: '/root/.ssh/'
       passphrase: '59b99453'
  - github.com/openvcloud/0-templates/openvcloud/0.0.1__2b32fa7f:
      name: 'be-g8-3.demo.greenitglobe.com'
      address: 'be-g8-3.demo.greenitglobe.com'
      login: 'gig_qa_1@itsyouonline'
      token: '****'
      location: 'be-g8-3'
  - github.com/openvcloud/0-templates/vdcuser/0.0.1__49e18fba:
      openvcloud: 2b32fa7f
      email: 5ce35803@test.com
      name: 2200f103
      groups: ['user']
      provider: itsyouonline
  - github.com/openvcloud/0-templates/vdcuser/0.0.1__gig_qa_1:
      openvcloud: 2b32fa7f
      email: dina.magdy.mohammed+123@gmail.com
      name: gig_qa_1
      provider: itsyouonline

  - github.com/openvcloud/0-templates/account/0.0.1__ea21df8e:
      openvcloud: 2b32fa7f
      name: 6e31adcf

  - github.com/openvcloud/0-templates/vdc/0.0.1__0815f598:
      maxCPUCapacity: -673
      account: ea21df8e
      maxVDiskCapacity: -383
      maxNetworkPeerTransfer: -247
      maxMemoryCapacity: -231
      name: 303f11fe
      maxNumPublicIP: -122

actions:
  - template: github.com/openvcloud/0-templates/vdcuser/0.0.1
    actions: ['install']
  - template: github.com/openvcloud/0-templates/openvcloud/0.0.1
    actions: ['install']
  - template: github.com/openvcloud/0-templates/vdc/0.0.1
    actions: ['install']
  - template: github.com/openvcloud/0-templates/account/0.0.1
    actions: ['install']
katia-e commented 6 years ago

@FastGeert , after discussion with @john-kheir I understand problem as follows:

Problem: Retry on install actions is triggered by batch of errors returned by OpenvCloud. Most of these errors can't be solved by retrying and need to be reported as soon as possible. For example, OpenvCloud implements number of validations that are not necessary to repeat in 0-robot templates entirely, authorization errors also should not trigger rety.

Proposal: Run retry only on errors caused by network connectivity issues, like timeouts.