mrackdb in inconsistent state after reusing same metadata for mrack up after mrack destroy

[pvoborni]

from my observance, it mostly fail if hosts changes between runs. E.g. metadata is modified but there is some state in DB. Simple mrack up -> mrack destroy -> mrack up usually works.

[dkarpelevich]

Using mrack up multiple times on the same host causes an exception. Workaround: rm /config/mrackdb.json

fatal: [localhost]: FAILED! => changed=true
...
    Provisioning done
    Output generation started
    cannot unpack non-iterable NoneType object
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 244, in handle
        rc = func(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 267, in run
        mrackcli(obj={})
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 60, in wrapper
        return loop.run_until_complete(f(*args, **kwargs))
      File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
        return future.result()
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 162, in up
        await generate_outputs(ctx)
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 112, in generate_outputs
        await output_action.generate_outputs()
      File "/usr/local/lib/python3.8/site-packages/mrack/actions/output.py", line 56, in generate_outputs
        ansible_o.create_output()
      File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 211, in create_output
        inventory = self.create_inventory()
      File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 191, in create_inventory
        meta_host, meta_domain = get_host_from_metadata(self._metadata, host.name)
    TypeError: cannot unpack non-iterable NoneType object
    Traceback (most recent call last):
      File "/usr/local/bin/mrack", line 21, in <module>
        run.run()
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 257, in handle
        raise exc
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 244, in handle
        rc = func(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 267, in run
        mrackcli(obj={})
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 60, in wrapper
        return loop.run_until_complete(f(*args, **kwargs))
      File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
        return future.result()
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 162, in up
        await generate_outputs(ctx)
      File "/usr/local/lib/python3.8/site-packages/mrack/run.py", line 112, in generate_outputs
        await output_action.generate_outputs()
      File "/usr/local/lib/python3.8/site-packages/mrack/actions/output.py", line 56, in generate_outputs
        ansible_o.create_output()
      File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 211, in create_output
        inventory = self.create_inventory()
      File "/usr/local/lib/python3.8/site-packages/mrack/outputs/ansible_inventory.py", line 191, in create_inventory
        meta_host, meta_domain = get_host_from_metadata(self._metadata, host.name)
    TypeError: cannot unpack non-iterable NoneType object
  stderr_lines: <omitted>
  stdout: /usr/local/bin/mrack
  stdout_lines: <omitted>

[pvoborni]

@dkarpelevich just to confirm, in these multiple times, do you e.g. change the metadata used?

[dkarpelevich]

@pvoborni Yes, I have changed the metadata file between runs.

[pvoborni]

In general, I'd recommend to use a different test working directory ($TWD, the one where you call mrack) for each use case (metadata). We can fail more gracefully, but there is an assumtion that $TWD == one testing session. The reason is that the directory serves as a context for each step - a way how a step can pass data to a following step.

It's OK to do minor changes(especially in non-provision steps) in metadata and call te multiple times with the changes. But bigger changes like changing the whole file have undefined behavior.