linux-system-roles / nbde_client

Ansible role for configuring Network Bound Disk Encryption clients (e.g. clevis)
https://linux-system-roles.github.io/nbde_client/
MIT License
14 stars 24 forks source link

Increase memory of tests #38

Closed sergio-correia closed 3 years ago

sergio-correia commented 3 years ago

We were experiencing failures in CI due to OOM errors, so let's increase the amount of memory used when running the tests.

Additionally, to help us investigate further issues, the tests now display the facts gathered during the setup phase, and run_cryptsetup() runs with --debug, so that it can display extra information in case it fails.

richm commented 3 years ago

@sergio-correia does that debug output give any clues?

sergio-correia commented 3 years ago

@richm: how can I control the amount of ram when testing with standard-test-roles? I see that when I run locally, my VM shows "memtotal_mb": 1815, while in the CI we have considerably less, e.g. 799 (https://fedorapeople.org/groups/linuxsystemroles/logs/linux-system-roles-nbde_client-pull-linux-system-roles_nbde_client-38-f9d1b81-rhel-8-y-20210309-130224/artifacts/ansible.log) and 797 (https://fedorapeople.org/groups/linuxsystemroles/logs/linux-system-roles-nbde_client-pull-linux-system-roles_nbde_client-38-f9d1b81-centos-8-20210309-125944/artifacts/ansible.log).

Perhaps I could reproduce the issue locally if I can manage to reduce the memory available to my test VM.

richm commented 3 years ago

@richm: how can I control the amount of ram when testing with standard-test-roles? I see that when I run locally, my VM shows "memtotal_mb": 1815, while in the CI we have considerably less, e.g. 799 (https://fedorapeople.org/groups/linuxsystemroles/logs/linux-system-roles-nbde_client-pull-linux-system-roles_nbde_client-38-f9d1b81-rhel-8-y-20210309-130224/artifacts/ansible.log) and 797 (https://fedorapeople.org/groups/linuxsystemroles/logs/linux-system-roles-nbde_client-pull-linux-system-roles_nbde_client-38-f9d1b81-centos-8-20210309-125944/artifacts/ansible.log).

Perhaps I could reproduce the issue locally if I can manage to reduce the memory available to my test VM.

Use fmf

cd tests
fmf init

Then create a provision.fmf like the one used by certificate - https://github.com/linux-system-roles/certificate/blob/master/tests/provision.fmf If the problem is that the nbde_client tests need more RAM, use m: 2048 in your provision.fmf

richm commented 3 years ago

@sergio-correia pass! so maybe the problem is that the test needs more ram - in which case just turn this PR into a "real" PR

sergio-correia commented 3 years ago

@sergio-correia pass! so maybe the problem is that the test needs more ram - in which case just turn this PR into a "real" PR

Yep. With --debug it gave some hints to be memory-related and by displaying the full facts I could check how much memory we had in the CI machine. After reducing the amount of memory locally, I could reproduce the issue and get an OOM error.

Finally, by using your suggestion to increase memory of the tests (thanks, btw), the issue was gone. Let me update this PR slightly and we can probably merge it.

richm commented 3 years ago

@sergio-correia please rebase this PR to resolve conflicts

sergio-correia commented 3 years ago

@richm: I have rebased it