0-robot fail to create a service

rkhamis commented 6 years ago

Issue migrated from [https://api.github.com/repos/zero-os/0-robot/issues/260](), opened by @islamTaha12

Description I have a test suite to install/uninstall a DM vms and while I'm running it, 0-robot threw the following error

re-produce steps:

the most easy case is to create a for loop to install/uninstall a vm in range(200)


for _ in range(200):
    data = {
        'nodeId': self.nodeId,
        'disks': [{
            'diskType': disk_type,
            'size': disk_size,
            'mountPoint': '/mnt',
            'filesystem': disk_fs,
            'label': str(uuid.uuid4()).replace('-', '')[:10],
        }],
        'image': 'ubuntu',
        'memory': memory,
        'cpu': cpu,
        'zerotier': {'id': self.zt_network.id, 'ztClient': self.zt_client_instance},
        'configs': [{'path': '/root/.ssh/authorized_keys',
                     'content': self.ssh,
                     'name': 'sshkey'}]
    }

    try:
        self.vmservice = self.robot.services.find_or_create(self.vmtemplate, service_name=self.service_name,
                                                            data=data)
        self.vmservice.schedule_action('install').wait(die=True)
    except Exception as e:
        logging.error(traceback.format_exc())
        return



**Env**
- 0-OS 1.4.1
- js9: dev
- DM: master
- 0-robot: dev

rkhamis commented 6 years ago

commented by @zaibon I did some test that loops and create 500 services without problem. So I don't know if the problem comes from the template or the robot itself. @islamTaha12 can you post the full stack trace for both the client and the server ? Currently I'm not sure which is the one failing. Could be that we have some bug in the client code that doesn't release file description somehow

rkhamis commented 6 years ago

*commented by @islamTaha12 @zaibon I'm gonna run it in a tmux and u can access the machine cause the traceback is too long to be posted, client machine in be-ge-3 ssh root@10.101.107.91 and the host machine is

*

rkhamis commented 6 years ago

*commented by @islamTaha12 client side


[Thu12 17:22] - base.py           :209 :service-f8123400-7a58-4782-af1c-377656873653 - ERROR    - error executing action save:
Traceback (most recent call last):
  File "/opt/code/github/zero-os/0-robot/zerorobot/task/task.py", line 79, in execute
    self._result = self._func()
  File "/opt/code/github/zero-os/0-robot/zerorobot/template/base.py", line 182, in save
    'public': self._public,
  File "/opt/code/github/jumpscale/core9/JumpScale9/data/serializers/SerializerBase.py", line 11, in dump
    j.sal.fs.writeFile(filepath, data)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFSDecorators.py", line 155, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFS.py", line 1215, in writeFile
    fp = open(filename, "wb")
OSError: [Errno 24] Too many open files: '/opt/var/data/zrobot/zrobot_data/github.com/jumpscale/digital_me/vm/f80239ecb2/f8123400-7a58-4782-af1c-377656873653/service.yaml'

[Thu12 17:22] - base.py           :209 :service-e16cdbfc-d26c-49f5-abaf-d5d8a3cc1003 - ERROR    - error executing action save:
Traceback (most recent call last):
  File "/opt/code/github/zero-os/0-robot/zerorobot/task/task.py", line 79, in execute
    self._result = self._func()
  File "/opt/code/github/zero-os/0-robot/zerorobot/template/base.py", line 182, in save
    'public': self._public,
  File "/opt/code/github/jumpscale/core9/JumpScale9/data/serializers/SerializerBase.py", line 11, in dump
    j.sal.fs.writeFile(filepath, data)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFSDecorators.py", line 155, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFS.py", line 1215, in writeFile
    fp = open(filename, "wb")
OSError: [Errno 24] Too many open files: '/opt/var/data/zrobot/zrobot_data/github.com/jumpscale/digital_me/vm/7ce2df6f58/e16cdbfc-d26c-49f5-abaf-d5d8a3cc1003/service.yaml'

[Thu12 17:22] - vm.py             :57  :service-e16cdbfc-d26c-49f5-abaf-d5d8a3cc1003 - INFO     - Monitor vm 7ce2df6f58
[Thu12 17:22] - base.py           :209 :service-e16cdbfc-d26c-49f5-abaf-d5d8a3cc1003 - ERROR    - error executing action _monitor:
Traceback (most recent call last):
  File "/opt/code/github/zero-os/0-robot/zerorobot/template/state.py", line 78, in check
    state = self.get(category, tag)
  File "/opt/code/github/zero-os/0-robot/zerorobot/template/state.py", line 57, in get
    raise StateCategoryNotExistsError("category %s does not exist" % category)
zerorobot.template.state.StateCategoryNotExistsError: category actions does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/github/zero-os/0-robot/zerorobot/task/task.py", line 79, in execute
    self._result = self._func()
  File "/opt/code/github/jumpscale/digital_me/templates/vm/vm.py", line 58, in _monitor
    self.state.check('actions', 'install', 'ok')
  File "/opt/code/github/zero-os/0-robot/zerorobot/template/state.py", line 85, in check
    raise StateCheckError(err_msg)
zerorobot.template.state.StateCheckError: check for state actions:install:ok failed

[Thu12 17:22] - base.py           :209 :service-aa529441-e183-4907-b5e6-df20c6920bd7 - ERROR    - error executing action save:
Traceback (most recent call last):
  File "/opt/code/github/zero-os/0-robot/zerorobot/task/task.py", line 79, in execute
    self._result = self._func()
  File "/opt/code/github/zero-os/0-robot/zerorobot/template/base.py", line 182, in save
    'public': self._public,
  File "/opt/code/github/jumpscale/core9/JumpScale9/data/serializers/SerializerBase.py", line 11, in dump
    j.sal.fs.writeFile(filepath, data)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFSDecorators.py", line 155, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFS.py", line 1215, in writeFile
    fp = open(filename, "wb")
OSError: [Errno 24] Too many open files: '/opt/var/data/zrobot/zrobot_data/github.com/jumpscale/digital_me/vm/1c8279ebb7/aa529441-e183-4907-b5e6-df20c6920bd7/service.yaml'

    'public': self._public,
  File "/opt/code/github/jumpscale/core9/JumpScale9/data/serializers/SerializerBase.py", line 11, in dump
    j.sal.fs.writeFile(filepath, data)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFSDecorators.py", line 155, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/github/jumpscale/core9/JumpScale9/fs/SystemFS.py", line 1215, in writeFile
    fp = open(filename, "wb")
OSError: [Errno 24] Too many open files: '/opt/var/data/zrobot/zrobot_data/github.com/jumpscale/digital_me/vm/66fb43b24b/fd836490-6c6f-4670-916c-be291f581eb7/service.yaml'

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/baseserver.py", line 199, in _do_read
  File "/usr/local/lib/python3.5/dist-packages/gevent/server.py", line 164, in do_read
OSError: [Errno 24] Too many open files
2018-07-12T17:25:33Z <WSGIServer at 0x7fd6770435f8 fileno=11 address=('::', 6600, 0, 0)> failed with OSError

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/baseserver.py", line 199, in _do_read
  File "/usr/local/lib/python3.5/dist-packages/gevent/server.py", line 164, in do_read
OSError: [Errno 24] Too many open files
2018-07-12T17:25:34Z <WSGIServer at 0x7fd6770435f8 fileno=11 address=('::', 6600, 0, 0)> failed with OSError

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/baseserver.py", line 199, in _do_read
  File "/usr/local/lib/python3.5/dist-packages/gevent/server.py", line 164, in do_read
OSError: [Errno 24] Too many open files
2018-07-12T17:25:36Z <WSGIServer at 0x7fd6770435f8 fileno=11 address=('::', 6600, 0, 0)> failed with OSError

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/baseserver.py", line 199, in _do_read
  File "/usr/local/lib/python3.5/dist-packages/gevent/server.py", line 164, in do_read
OSError: [Errno 24] Too many open files
2018-07-12T17:25:37Z <WSGIServer at 0x7fd6770435f8 fileno=11 address=('::', 6600, 0, 0)> failed with OSError

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/baseserver.py", line 199, in _do_read
  File "/usr/local/lib/python3.5/dist-packages/gevent/server.py", line 164, in do_read
OSError: [Errno 24] Too many open files
2018-07-12T17:25:38Z <WSGIServer at 0x7fd6770435f8 fileno=11 address=('::', 6600, 0, 0)> failed with OSError

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gevent/baseserver.py", line 199, in _do_read
  File "/usr/local/lib/python3.5/dist-packages/gevent/server.py", line 164, in do_read
OSError: [Errno 24] Too many open files
2018-07-12T17:25:39Z <WSGIServer at 0x7fd6770435f8 fileno=11 address=('::', 6600, 0, 0)> failed with OSError

```*

rkhamis commented 6 years ago

*commented by @islamTaha12 update: When I tried to install a VM manually, The request is hanging forever.

*

zaibon commented 6 years ago

never seen the issue anymore I'll close for now

threefoldtecharchive / 0-robot

0-robot fail to create a service #6