Add support to Debian 11 (bullseye)

egypcio commented 1 year ago

Description

Motivation and Context

According to the INSTALL Documentation of the MISP Project, the installation procedures on top of Ubuntu 20.04 are being tested on a regular basic. That said, we decided to give it a shot and try running this implementation of an Ansible role on top of a Debian 11 (bullseye); it seems to be the closes system to what "Focal Fossa" offers lately (with a little more reliability and stability[0][1][2]).

$ flames >/dev/null

We also take the chance to offer an updated version of a Debian setup to folks that prefer that distribution over others -- recent documentation on the MISP Project website are covering only Debian 10 (buster).

[0] Ubuntu 20.04 currently falls under https://github.com/pyca/pyopenssl/issues/1114; [1] certbot on Ubuntu 20.04 requires that one uses snap; [2] should folks rely on node_exporter, the package for Ubuntu 20.04 is wonky - e.g.: it miscalculates NIC saturation;

How Has This Been Tested?

Fresh setup using Debian 11 (bullseye), from official project's cloud images;
Called the role directly as root (using cloud-init credentials from the vanilla image);
Opened a fresh instance of MISP, running on top of Debian 11;
Server "survives" a reboot, still offering the regular web interface and misp-modules;
Background jobs are running just fine;
Plugin credentials added to the MISP configuration file, and service restarted w/o any blocker.

Screenshots

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed including pre-commit and github actions.
[x] Used in production.

juju4 commented 1 year ago

Thanks a lot! LGTM I think that we can add tests in https://github.com/juju4/ansible-MISP/blob/main/.github/workflows/default-kitchen.yml as debian:bullseye available on https://images.linuxcontainers.org/ same for https://github.com/juju4/ansible-MISP/blob/main/.github/workflows/default.yml with https://hub.docker.com/_/debian

egypcio commented 1 year ago

Thanks a lot! LGTM I think that we can add tests in https://github.com/juju4/ansible-MISP/blob/main/.github/workflows/default-kitchen.yml as debian:bullseye available on https://images.linuxcontainers.org/ same for https://github.com/juju4/ansible-MISP/blob/main/.github/workflows/default.yml with https://hub.docker.com/_/debian

avoided to push those changed before assuming you would get these added -- as owner of the repository; thanks for allowing this PR to modify that right away.

done :+1:

PS: used "bullseye" to identify the version of the image, as shown in all matrices (LXC, and Docker) documentation references.

egypcio commented 1 year ago

ping @juju4

juju4 commented 1 year ago

Sorry @egypcio I won't have time to check PR this week. Will try following one. Thanks for your patience!

juju4 commented 1 year ago

will fix lxc/kitchen ci issue directly. Merging Thanks a lot!

juju4 commented 1 year ago

Sort out multiple role dependencies debian-11 support issues. At last, get to point where same pymisp issue observed in past, but mainly on github actions, not other setup. Not appearing on your side, I suppose?

https://github.com/juju4/ansible-MISP/actions/runs/5170219845/jobs/9312990595#step:9:3278

       TASK [juju4.misp : List current users] *****************************************
       task path: /tmp/kitchen/juju4.misp/tasks/misp-add-users.yml:72
       Sunday 04 June 2023  17:19:57 +0000 (0:00:01.042)       0:11:33.179 *********** 
       fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["/var/www/_MISP/venv/bin/python", "./users_list.py"], "delta": "0:00:00.579300", "end": "2023-06-04 17:19:58.225715", "msg": "non-zero return code", "rc": 1, "start": "2023-06-04 17:19:57.646415", "stderr": "/var/www/_MISP/venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings\n  warnings.warn(\nUnknown error: the response is not in JSON.\nSomething is broken server-side, please send us everything that follows (careful with the auth key):\nRequest headers:\n{'User-Agent': 'PyMISP 2.4.148 - Python 3.9', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'MAznv0bAt19DmArn7Ck8WUVWnxlTRMiHRoP6MUln', 'content-type': 'application/json'}\nRequest body:\nNone\nResponse (if any):\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\\/servers\\/getVersion\"}\nTraceback (most recent call last):\n  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 169, in __init__\n    response = self.recommended_pymisp_version\n  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 226, in recommended_pymisp_version\n    misp_version = self.misp_instance_version\n  File \"/usr/lib/python3.9/functools.py\", line 969, in __get__\n    val = self.func(instance)\n  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 256, in misp_instance_version\n    return self._check_json_response(response)\n  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 3454, in _check_json_response\n    r = self._check_response(response, expect_json=True)\n  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 3471, in _check_response\n    raise MISPServerError(f'Error code 500:\\n{response.text}')\npymisp.exceptions.MISPServerError: Error code 500:\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\\/servers\\/getVersion\"}\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/var/www/_MISP/MISP/PyMISP/examples/./users_list.py\", line 12, in <module>\n    misp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert)\n  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 190, in __init__\n    raise PyMISPError(f'Unable to connect to MISP ({self.root_url}). Please make sure the API key and the URL are correct (http/https is required): {e}')\npymisp.exceptions.PyMISPError: Unable to connect to MISP (https://localhost). Please make sure the API key and the URL are correct (http/https is required): Error code 500:\n{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\\/servers\\/getVersion\"}", "stderr_lines": ["/var/www/_MISP/venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings", "  warnings.warn(", "Unknown error: the response is not in JSON.", "Something is broken server-side, please send us everything that follows (careful with the auth key):", "Request headers:", "{'User-Agent': 'PyMISP 2.4.148 - Python 3.9', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'MAznv0bAt19DmArn7Ck8WUVWnxlTRMiHRoP6MUln', 'content-type': 'application/json'}", "Request body:", "None", "Response (if any):", "{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\\/servers\\/getVersion\"}", "Traceback (most recent call last):", "  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 169, in __init__", "    response = self.recommended_pymisp_version", "  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 226, in recommended_pymisp_version", "    misp_version = self.misp_instance_version", "  File \"/usr/lib/python3.9/functools.py\", line 969, in __get__", "    val = self.func(instance)", "  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 256, in misp_instance_version", "    return self._check_json_response(response)", "  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 3454, in _check_json_response", "    r = self._check_response(response, expect_json=True)", "  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 3471, in _check_response", "    raise MISPServerError(f'Error code 500:\\n{response.text}')", "pymisp.exceptions.MISPServerError: Error code 500:", "{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\\/servers\\/getVersion\"}", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/var/www/_MISP/MISP/PyMISP/examples/./users_list.py\", line 12, in <module>", "    misp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert)", "  File \"/var/www/_MISP/venv/lib/python3.9/site-packages/pymisp/api.py\", line 190, in __init__", "    raise PyMISPError(f'Unable to connect to MISP ({self.root_url}). Please make sure the API key and the URL are correct (http/https is required): {e}')", "pymisp.exceptions.PyMISPError: Unable to connect to MISP (https://localhost). Please make sure the API key and the URL are correct (http/https is required): Error code 500:", "{\"name\":\"An Internal Error Has Occurred.\",\"message\":\"An Internal Error Has Occurred.\",\"url\":\"\\/servers\\/getVersion\"}"], "stdout": "", "stdout_lines": []}

https://github.com/juju4/ansible-MISP/actions/runs/5170219845/jobs/9312990595#step:14:58

+ sudo kitchen exec default-debian-bullseye -c 'cat /var/www/_MISP/MISP/app/tmp/logs/error.log'
-----> Execute command on default-debian-bullseye.
       2023-06-04 17:19:57 Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'User.last_api_access' in 'field list'
       Request URL: /servers/getVersion
       Stack Trace:
       #0 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
       #1 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()
       #2 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): DboSource->execute()
       #3 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1226): DboSource->fetchAll()
       #4 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3053): DboSource->read()
       #5 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3025): Model->_readDataSource()
       #6 /var/www/_MISP/MISP/app/Model/Behavior/AuditLogBehavior.php(121): Model->find()
       #7 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Utility/ObjectCollection.php(129): AuditLogBehavior->beforeSave()
       #8 /var/www/_MISP/MISP/app/Lib/Tools/BetterCakeEventManager.php(23): ObjectCollection->trigger()
       #9 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1873): BetterCakeEventManager->dispatch()
       #10 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()
       #11 /var/www/_MISP/MISP/app/Model/User.php(1428): Model->save()
       #12 /var/www/_MISP/MISP/app/Controller/AppController.php(444): User->updateAPIAccessTime()
       #13 /var/www/_MISP/MISP/app/Controller/AppController.php(233): AppController->__loginByAuthKey()
       #14 /var/www/_MISP/MISP/app/Controller/ServersController.php(41): AppController->beforeFilter()
       #15 /var/www/_MISP/MISP/app/Lib/Tools/BetterCakeEventManager.php(23): ServersController->beforeFilter()
       #16 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(682): BetterCakeEventManager->dispatch()
       #17 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(189): Controller->startupProcess()
       #18 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
       #19 /var/www/_MISP/MISP/app/webroot/index.php(99): Dispatcher->dispatch()
       #20 {main}
       2023-06-04 17:19:58 Error: [PDOException] SQLSTATE[42S22]: Column not found: 1054 Unknown column 'User.last_api_access' in 'field list'
       Request URL: /servers/getVersion
       Stack Trace:
       #0 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()
       #1 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()
       #2 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(715): DboSource->execute()
       #3 /var/www/_MISP/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1226): DboSource->fetchAll()

juju4 commented 1 year ago

For now, I marked debian:bullseye as experimental for default-kitchen/lxc ci even if I believe problem is only inside github workflow.

juju4 / ansible-MISP