search

pixelated / puppet-pixelated

Puppet Module to setup the Pixelated User Agent
GNU Affero General Public License v3.0
13 stars 4 forks source link

Functional tests don't remove test users and fill up disk #36

Closed varac closed 8 years ago

varac commented 8 years ago

Every hour the pixelated functional tests are run by cron. But the created testusers are not removed, and they keep filling the disk:

screenshot_2016-05-18_14-10-14

here's the disk usage graph indicating that every hour to grows:

screenshot_2016-05-18_14-14-08

(from https://unstable.pixelated-project.org/pnp4nagios/index.php/graph?host=unstable1.unstable.pixelated-project.i&srv=fs_/storage)

also, but off topic, it would be good to name the testusers starting with 'test-' or sth so you can easily spot them as testusers.

elijh commented 8 years ago

There are four kinds of users:

for users that are created and destroyed frequently, it is highly preferable to use "tmp_user_xxx" because these special users are stored in a separate couchdb database (tmp_users instead of users). This allows the tmp_users db to be regularly destroyed so that tmp users don't bloat the users db over time by accruing lots of tombstone documents for the deleted users.

there is one problem with 'tmp' users: they don't have Identity documents. This means that you cannot deliver mail to these 'tmp' users. So, for mail integration tests, users with "test_user_xxx" are needed. These users are long lived, and should not be created and destroyed each time the test is run. These "test_user" accounts can be created and destroy special by authenticating with the API using the "monitor" authentication tokens (api_monitor_auth_token in secrets.json). The monitor token is not authorized to create/destroy/update regular or admin users, just "test" users.

This silly system can go away once we are off couchdb.

cz8s commented 8 years ago

behave-testuser gets deleted now. For qa: If you want to see if the user is still in the database, use this:

/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep --color=auto 'behave'

Or use the webapp if you have an admin account

elijh commented 8 years ago

how often are these test users created and destroyed? unless the users start with 'tmpuser', they will slowly bloat the users database with tombstone records.

cz8s commented 8 years ago

okay, what would you suggest @elijh . We want to test sending mail, so tmp_test_XX can not be used. We already can delete users because we can read the netrc file. So does it make sense to change the username to test_user_XX ? If there are tombstone records left that can not be deleted, would it helpf if we ran these tests less often? maybe once a week? How bloated will the database be then? We do not want to keep the user around because we want to test a defined state. Maybe we should not run these tests on production then?

elijh commented 8 years ago

for the leap test run tests, it creates a permanent TEST_EMAIL_USER (test_user_email) that is always used for testing email delivery. This user is never destroyed, once created, but the per-user mail storage db for this user is destroyed frequently, so no db bloat.

https://github.com/leapcode/leap_platform/blob/develop/tests/white-box/mx.rb#L128

yes, i think running the tests once a week would not create noticeable db bloat. i don't remember how big the tombstone records are, but they are small. nonetheless, small adds up quickly when you run a test every five minutes.

cz8s commented 8 years ago

cool. will change the cronjob to once a week. Problem is, once they fail they will stay red for a week (or retriggered manually) So we have to make sure they don't fail :) Where get the mails fro the test_user_email user get deleted? Maybe we should have done the same in the first place. We wouldn't have to touch the tests for that ...

elijh commented 8 years ago

Where get the mails fro the test_user_email user get deleted?

it is not very elegant, but you can see it here https://github.com/leapcode/leap_platform/blob/develop/tests/white-box/couchdb.rb#L130

out of curiosity, what mail delivery tests are needed that can't use test_user_email?

cz8s commented 8 years ago

we could rename the user to test_user_email. We only need to know the password. Our tests login to the webinterface, send an email to self, receive that email and check if the labels (encrypted,signed) are correct.

cz8s commented 8 years ago

we only delete the user-database now. user does not get deleted: https://github.com/pixelated/puppet-pixelated/commit/d025c973081391f447cee4d86e76e0cb2661ceee

varac commented 8 years ago

looks like it spits out some passwords or uuids when i run it with

INVITE_CODE_ENABLED=true /usr/bin/mk-job pixelated-functional-tests /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/

I don't paste the output in case it's the pw :D

varac commented 8 years ago

What i did:

root@dev1:/srv/leap/couchdb/scripts# INVITE_CODE_ENABLED=true /usr/bin/mk-job pixelated-functional-tests /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/
Traceback (most recent call last):
  File "/usr/local/bin/behave", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/behave/__main__.py", line 111, in main
    failed = runner.run()
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 659, in run
    return self.run_with_paths()
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 664, in run_with_paths
    self.load_hooks()
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 623, in load_hooks
    exec_file(hooks_path, self.hooks)
  File "/usr/local/lib/python2.7/dist-packages/behave/runner.py", line 306, in exec_file
    execfile(filename, globals, locals)
  File "/srv/leap/tests_custom/functional-tests/environment.py", line 22, in <module>
    from page_objects import SignUpPage
  File "/srv/leap/tests_custom/functional-tests/page_objects/__init__.py", line 29, in <module>
    from leap_login_page import LeapLoginPage
ImportError: No module named leap_login_page
varac commented 8 years ago

deployed again to dev1, ran functional tests there, but the user-db of the behave-user is still there:

root@dev1:/srv/leap/couchdb/scripts# INVITE_CODE_ENABLED=true /usr/bin/mk-job pixelated-functional-tests /usr/local/bin/behave --tags @staging --tags ~@wip --no-capture -k /srv/leap/tests_custom/functional-tests/

@staging
Feature: account # ../../tests_custom/functional-tests/account.feature:17

  @register
  Scenario: user goes to signup-page  # ../../tests_custom/functional-tests/account.feature:20
    When I visit the signup-page      # ../../tests_custom/functional-tests/steps/account.py:61 0.218s
    Then I should see a signup button # ../../tests_custom/functional-tests/steps/account.py:66 0.007s

...
  @unencrypted
  Scenario: user receives an unencrypted email  # ../../tests_custom/functional-tests/send_mail.feature:29
    Given I send an unencrypted email           # ../../tests_custom/functional-tests/steps/send_mail.py:37 0.179s
    And I login as behave-testuser              # ../../tests_custom/functional-tests/steps/send_mail.py:29 17.797s
    When I open the unencrypted email           # ../../tests_custom/functional-tests/steps/send_mail.py:84 30.030s
    Then I see a unencrypted email flag         # ../../tests_custom/functional-tests/steps/send_mail.py:99 0.035s

2 features passed, 0 failed, 6 skipped
7 scenarios passed, 0 failed, 9 skipped
23 steps passed, 0 failed, 62 skipped, 0 undefined
Took 3m29.227s

root@dev1:/srv/leap/couchdb/scripts# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep behave
{"id":"85daeaa5e7d69c10d7f976f38c6e7ab7","key":"85daeaa5e7d69c10d7f976f38c6e7ab7","value":{"rev":"6-e4a8debeeaadb3901d23a9995cd826b0"},"doc":{"_id":"85daeaa5e7d69c10d7f976f38c6e7ab7","_rev":"6-e4a8debeeaadb3901d23a9995cd826b0","user_id":"85daeaa5e7d69c10d7f976f38c4e3378","address":"behave-testuser@dev.pixelated-project.org","destination":"behave-testuser@dev.pixelated-project.org","keys":{"pgp":"-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFcwPAEBEADIJ/TDv1SFy1jB+MsyTlE+vQjPrEUmJ61sMX5bRJ+YboQnfWbg\nN2JGOGhlzxwOrHtHI9Qwvxc70KyJJparOnY/ZH824o7SR6OBkDzUe5Mt3T1KQtJR\n11diwUWtzNoZ88DCmwUohng/a7mEJ3gha1uO4Ng2Kxr4JtGYmSBZKUerQfUHgISl\nGSIxshBVr6XEQ4frJojvp1eq9fmLf0Sgi3AotjSRMJ7yT6NaUZjouH892I84wkoB\npYbkOzHUxrINB7zEn5P5k1FOrVZY3Uyu2JbWl9ZJWcX70Vv3EkU8KwoTC1NKScXO\nFfw7WXJzwE9zDzW/OGszsID9DL42XP4p3FYO/y3L5LblzuYIUofOynKA0FfQEubR\nDQ+7cuLWImANVW+0bsJc/7MF+DdK99N4/5KSzDFqUeeDXmZ9JdkSBtr5/xPq4cDj\nPjg6nCCKfne6Vi+IvpoNy/VwChN50Bhkv95Q1ekyR5rgAycwFxhAVnw/v/6fTqmQ\nIThm5BDrwok3W6am795g0DiImO8zUybNKo34Nvj7ci5ndq4IaRngTHnYwzfVnFtx\nsPyFfXMjjxsyY6XbboXR+teFmePKm+SjZAmBkMKOngbjOaioJByp8bRmu5PM878Y\n+Q6X1V7BQ1CF8BH1SsZBL06X2+Y8XWNlXLLMQV+XC7JYauX8gEUX0t87+QARAQAB\ntFViZWhhdmUtdGVzdHVzZXJAZGV2LnBpeGVsYXRlZC1wcm9qZWN0Lm9yZyA8YmVo\nYXZlLXRlc3R1c2VyQGRldi5waXhlbGF0ZWQtcHJvamVjdC5vcmc+iQI+BBMBAgAo\nBQJXMDwBAhsvBQkB4K5fBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRA3CuKF\nFVrSie8tD/9nsnd0lnB25lDA0j+jez8Hl9jjS50/Cn8I0JMOmHqSJV0yzaROyhPS\nJdfG0hFOxiZwG6P7WTxmkHZjfihqMWPFdcLZ6b6ywNoKQQK+j7vBfNxj6RsH8m/M\nOgsSZpckWptCk19Qu+FzLCkS4BcHBpTCKhSfSIb3+Uxw0luBBStyilZoDPm2+ckF\ntrPk0cNHKMgs2mgVzTsk9/4pBcz0AuLLV5OYgUeoobte39mM3oGGR2ciO4P7Rb9h\nbyxsteZWmU/DmBz0gPy9XFbfeWqNX+8i2FqvihKgYl2O34TAh9gaa6dW9xh8/pcw\nCMCCSrBEUSbq2tWm4HyQt5EtDaKD/kxqmOCzkmxmwMnjLg1JKASPCW9+d98vhTFf\nL8qxpqifLTw9trx4+7fjbR3UusU8SlPNG44+moqj3tgdF2uz8I0ug9SzW3TnpsSS\nUZBo4+KPpdLfLP6EfPlxARL6yuUJNkNmyzwrsdjNtTHZpC0bA0J4euj+QHwBhLih\nzm+hpVX2Cwy3M9HXIRtPXTBWUtSqlY2mFrPqd7jQZ+GNP16Ku39pORFzRCALhDlW\nJsusM+sbwwlOAYoPaerR0az12n2Mw+1fp9KNwR0QIqsX84C+49fNqIbXd9PFVd5Y\nOw1KS84Zius+5N6TEToXfAzdd+dhhMZP18I23r/vJnv8rsp7YnWopw==\n=5lSl\n-----END PGP PUBLIC KEY BLOCK-----\n"},"cert_fingerprints":{"16:92:54:22:77:a9:71:12:8f:21:9c:4a:3e:66:d2:96:12:9f:5c:95":"2016-07-09","1e:6d:46:88:7d:6d:45:98:72:96:62:02:66:b5:6c:e5:af:95:88:5f":"2016-07-17","cb:4b:45:4d:c6:21:15:c5:8f:9c:ca:9f:57:04:34:62:54:cf:c7:9b":"2016-07-23","6b:8e:d9:30:c2:52:7e:18:64:b2:5a:1e:ca:21:ff:32:bf:66:29:f7":"2016-07-24"},"enabled":true,"type":"Identity"}},

root@dev1:/srv/leap/couchdb/scripts# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/_all_dbs' | json_pp | grep 85daeaa5e7d69c10d7f976f38c4e3378

   "user-85daeaa5e7d69c10d7f976f38c4e3378",
varac commented 8 years ago

any updates on this ?

cz8s commented 8 years ago

all envs are green and the db gets deleted.

cz8s commented 8 years ago

to qa: login to any env and check which user_id the behave-testuser has:

/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep behave

Then check if the database exists (it shouldn't):

 curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/_all_dbs' | json_pp | grep <USER_ID>
varac commented 8 years ago

on unstable the behave-userdb still exists:

root@unstable1:~# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep behave 
{"id":"ad8feb7e16b749039fe0eb3822fab002","key":"ad8feb7e16b749039fe0eb3822fab002","value":{"rev":"3-0dff17498767a69bd2e7d42a7edbca63"},"doc":{"_id":"ad8feb7e16b749039fe0eb3822fab002","_rev":"3-0dff17498767a69bd2e7d42a7edbca63","user_id":"ad8feb7e16b749039fe0eb3822dc2d27","address":"behave-testuser@unstable.pixelated-project.org","destination":"behave-testuser@unstable.pixelated-project.org","keys":{"pgp":"-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFdNZ0sBEACszIAV5CpX4CJUQ0pOnwMR6eE/y3FMiBjgXTcn6C1srlonobht\nxXDb6pjHlRx+8/N7peu+uCqUO2bp73SsqZiVV3iKIDDJYbu91tM6oOGGQEsCVVYM\ntNMlWYXt9cOzc9CnBiTYoHax9CTh1kSR6fjJORxxbHLaLmp/pAQNxf8BxWZyXqm/\nDyDlxFpAi7ujhitiLo5i+Bs3llFNOGNpGaKqUSXZ7k7cecvZBBLiuKADj3h+rTXP\n142TZyr8k9GHB3rL7iWXeQnNk2VTBTHAGFqhYkAer1XXZwZ0Kz1SC8GO7DJfLSod\nVWVXmCVz/zHdtxoZjY/NzJkgvaYqeGXhGZ5q8vyIfi7sXQM7dypcWAbnA2TFEEGG\ncusoIce5zR6hSmq3UGeb5U5gtByfwfKjLiSjh01nvYYEM5rhdAayzZXjVK5i5G+Z\nIMfzHkLx5lQCNOLMFx2ACQZXSwD2uIzA8e7XzaIjFL2Fdug8KJCGg4v56KqTl5C6\nOZHtWiZXQf8fpNz7IkGbQlXoIUw++uvhTVK/17BjUi5o/cfFVcMVJHQ9/9mPkDAQ\nBIExQr2P3/dr12a7b9Z6X7qdUh6GzANSRdMo47ET59i5CDBWEF/a78I/6tOOQH+x\nx8MZvRPQK3VqlHFi7LNFcRg4kvzvOgH75Dhvf7SOYRh6Rqm+J3tj1Fy5gQARAQAB\ntF9iZWhhdmUtdGVzdHVzZXJAdW5zdGFibGUucGl4ZWxhdGVkLXByb2plY3Qub3Jn\nIDxiZWhhdmUtdGVzdHVzZXJAdW5zdGFibGUucGl4ZWxhdGVkLXByb2plY3Qub3Jn\nPokCPgQTAQIAKAUCV01nSwIbLwUJAeCEFQYLCQgHAwIGFQgCCQoLBBYCAwECHgEC\nF4AACgkQia8HOtFppxCRIA/9GWJf7+b0yh7hDfJXHguTCvQ4MY+6M/67pVqTZe7D\n71BtkBF1PtRAVpE85t0WUTRXB5WBcBcblsX9OS53W6whY5rcbMHIrzgA7dHO5uoB\nhZPYxBEoCVoMjyItnTnaQfAQesCdy+Zeb3nq1WlwHs5Q8dHoFIuqY8GHV5hpVhr/\nZZz/Cz3mBqgkTv9Mq6jrRHEfXKTEsF68BH8E7KBnCTF2978CDMa7L8Ko8/DGFrzf\nalUB0xIwz5Joth3oXch5v8jtMcIJCG0fNGvgwknI+l3d/0s9l8eCo6/pNqGVIWWW\nZWj0kZoashuZFka+o/TwjZt3wRXwzbseC3Izbm9vsUk3JZoSp5LMWjacHJ+WqnRW\nvRPKSphYKs7Yd/HXH4EwLUur5Kl5umlK1UAfcQnvTjnxgKRxLwSQQKyMG4DeF4JV\nexUCnXC0YeeFrtcEXH+FNBno1z4DvbUTWPe/jF44kyz5jtz5fPVLPVBdBjjO0X86\npD5jI6fs9xI/XfqCvJdTHh3bSygtidAr2z4qo9dyyyziKVFFSd8/8vQDJf1uMdDI\n1i1xmVS9zi4xelUlYsJ9rYr/+i+uAFnOqhb9lxaPKzJo24Zj3nBZ+YPUTrnjs7QT\na8HVE6vIzyE5UK62a72jHEG7yNVzmjm0oRdHwy9jiNlZaqYSXTahFLPYxcv716RQ\nkao=\n=MJrH\n-----END PGP PUBLIC KEY BLOCK-----\n"},"cert_fingerprints":{"1c:1f:fe:50:d4:5f:97:ab:f4:9a:88:5e:45:07:89:89:2a:55:69:a2":"2016-07-31"},"enabled":true,"type":"Identity"}},

root@unstable1:~# /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/user-ad8feb7e16b749039fe0eb3822dc2d27' 
{"db_name":"user-ad8feb7e16b749039fe0eb3822dc2d27","doc_count":431,"doc_del_count":0,"update_seq":2134,"purge_seq":0,"compact_running":false,"disk_size":20934768,"data_size":801296,"instance_start_time":"1465288286911492","disk_format_version":6,"committed_update_seq":2134}
cz8s commented 8 years ago

they were failing on unstable because user-agent was not able to login. Restart helped. Functional tests work again

thaissiqueira commented 8 years ago

@cz8s even with failing tests, the user database should be deleted too? Because on dev worked fine. But on unstable (where tests are failing) the database was not deleted 😕

cz8s commented 8 years ago

the database can not be deleted if the login fails. so if the tests fail, the database does not get deleted.

thaissiqueira commented 8 years ago

OK, só I'll try to discover why the testa are failing on unstable. For now we can launch this issue

elijh commented 8 years ago

eventually, every soledad db without an associated user should get destroyed, but it might take a while. the script is https://leap.se/git/couchdb_scripts.git/blob_plain/HEAD:/cleanup-user-dbs

I have not confirmed that it is actually being run.

varac commented 8 years ago

I have not confirmed that it is actually being run. It is run by cron daily, see root's crontab with crontab -l:

# Puppet Name: cleanup_stale_userdbs
7 4 * * * (/bin/date; /srv/leap/couchdb/scripts/cleanup-user-dbs) >> /var/log/leap/couchdb-cleanup.log