search

MISP / PyMISP

Python library using the MISP Rest API
Other
442 stars 278 forks source link

Error in adding organization via PyMISP #333

Closed datatern closed 5 years ago

datatern commented 5 years ago

I got the following error while running the test command:

nosetests-3.4 --with-coverage --cover-package=pymisp,tests --cover-tests tests/testlive_comprehensive.py

ERROR [aping.py:72 - _check_response() ] Something went wrong (403): {'name': 'Could not add Organisation', 'message': 'Could not add Organisation', 'url': '/admin/organisations/add', 'errors': {'name': ['An organisation with this name already exists.']}} E

ERROR: test suite for <class 'tests.testlive_comprehensive.TestComprehensive'>

Traceback (most recent call last): File "/home/ankit/venv/lib/python3.6/site-packages/nose/suite.py", line 210, in run self.setUp() File "/home/ankit/venv/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp self.setupContext(ancestor) File "/home/ankit/venv/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext try_run(context, names) File "/home/ankit/venv/lib/python3.6/site-packages/nose/util.py", line 471, in try_run return func() File "/home/ankit/PyMISP/tests/testlive_comprehensive.py", line 51, in setUpClass usr = cls.admin_misp_connector.add_user(email='testusr@user.local', org_id=cls.test_org.id, role_id=3) AttributeError: 'MISPOrganisation' object has no attribute 'id' -------------------- >> begin captured stdout << --------------------- No module named 'keys'

--------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /servers/getPyMISPVersion.json HTTP/1.1" 200 20 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /attributes/describeTypes.json HTTP/1.1" 200 17609 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /admin/organisations/add HTTP/1.1" 403 243 pymisp: ERROR: Something went wrong (403): {'name': 'Could not add Organisation', 'message': 'Could not add Organisation', 'url': '/admin/organisations/add', 'errors': {'name': ['An organisation with this name already exists.']}} --------------------- >> end captured logging << ---------------------

Name Stmts Miss Cover

pymisp/init.py 33 4 88% pymisp/abstract.py 182 31 83% pymisp/api.py 1425 804 44% pymisp/aping.py 268 230 14% pymisp/exceptions.py 36 0 100% pymisp/mispevent.py 689 156 77% pymisp/tools/init.py 18 0 100% pymisp/tools/abstractgenerator.py 43 32 26% pymisp/tools/asnobject.py 14 8 43% pymisp/tools/create_misp_object.py 74 27 64% pymisp/tools/domainipobject.py 14 8 43% pymisp/tools/elfobject.py 70 13 81% pymisp/tools/emailobject.py 48 37 23% pymisp/tools/ext_lookups.py 31 19 39% pymisp/tools/fail2banobject.py 12 6 50% pymisp/tools/fileobject.py 62 9 85% pymisp/tools/genericgenerator.py 8 5 38% pymisp/tools/geolocationobject.py 14 8 43% pymisp/tools/load_warninglists.py 15 9 40% pymisp/tools/machoobject.py 67 13 81% pymisp/tools/neo4j.py 39 27 31% pymisp/tools/openioc.py 94 81 14% pymisp/tools/peobject.py 102 21 79% pymisp/tools/sbsignatureobject.py 11 7 36% pymisp/tools/stix.py 21 15 29% pymisp/tools/vtreportobject.py 46 33 28% tests/init.py 0 0 100% tests/testlive_comprehensive.py 731 669 8%

TOTAL 4167 2272 45%

Ran 0 tests in 0.040s

FAILED (errors=1)

Running the code in a virtual environment with Python 3.6.7 (Please Help)

Rafiot commented 5 years ago

The error message is the following: ['An organisation with this name already exists.']. You cannot run the tests if there is already an organization with the same name on the system.

By default, unless they are canceled mid-way, the test cases will automatically cleanup everything it creates, but it may sometimes fails (i.e. if you do ctrl+c). In that case, you need to cleanup the remaining events, users, and orgs manually.

datatern commented 5 years ago

Thanks for the response @Rafiot , I've generated a new MISP image which resolved the above error, but still I'm unable to run this command

nosetests-3.4 --with-coverage --cover-package=pymisp,tests --cover-tests tests/testlive_comprehensive.py

Getting the following error message:

`ERROR [aping.py:72 - _check_response() ] Something went wrong (405): {'name': 'You do not have permission to use this functionality.', 'message': 'You do not have permission to use this functionality.', 'url': '/events/publish/96'} .....F.....S..ERROR [aping.py:72 - _check_response() ] Something went wrong (405): {'name': 'You do not have permission to use this functionality.', 'message': 'You do not have permission to use this functionality.', 'url': '/sightings/restSearch'} F........

FAIL: test_add_event_with_attachment (tests.testlive_comprehensive.TestComprehensive)

Traceback (most recent call last): File "/home/ankit/PyMISP/tests/testlive_comprehensive.py", line 927, in test_add_event_with_attachment self.assertEqual(len(first.objects[0].references), 1) AssertionError: 0 != 1 -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /servers/getPyMISPVersion.json HTTP/1.1" 200 20 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /attributes/describeTypes.json HTTP/1.1" 200 17609 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /admin/organisations/add HTTP/1.1" 200 484 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /admin/users/add HTTP/1.1" 200 717 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /servers/getPyMISPVersion.json HTTP/1.1" 200 20 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /attributes/describeTypes.json HTTP/1.1" 200 17609 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /admin/users/add HTTP/1.1" 200 717 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /servers/getPyMISPVersion.json HTTP/1.1" 200 20 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "GET /attributes/describeTypes.json HTTP/1.1" 200 17609 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 1719 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events/5c5abe4a-7b10-4be4-ab4d-05fc0a00020f HTTP/1.1" 200 60373 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/93 HTTP/1.1" 200 100 --------------------- >> end captured logging << ---------------------

====================================================================== FAIL: The default distributions on the VM are This community only for the events and Inherit from event for attr/obj)

Traceback (most recent call last): File "/home/ankit/PyMISP/tests/testlive_comprehensive.py", line 482, in test_default_distribution self.assertEqual(first.attributes[1].distribution, Distribution.inherit.value) AssertionError: 0 != 5 -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 3270 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events/5c5abe4b-10ec-43ed-9c7d-02d30a00020f HTTP/1.1" 200 5499 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/94 HTTP/1.1" 200 100 --------------------- >> end captured logging << ---------------------

====================================================================== FAIL: Search Tags at attributes level

Traceback (most recent call last): File "/home/ankit/PyMISP/tests/testlive_comprehensive.py", line 276, in test_search_tag_attribute self.assertEqual(len(attributes), 1) AssertionError: 3 != 1 -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 3615 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 4596 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 3907 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /attributes/restSearch HTTP/1.1" 200 3366 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /attributes/restSearch HTTP/1.1" 200 2085 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events/restSearch HTTP/1.1" 200 2272 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /attributes/restSearch HTTP/1.1" 200 2581 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /attributes/restSearch HTTP/1.1" 200 2085 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events/restSearch HTTP/1.1" 200 17 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /attributes/restSearch HTTP/1.1" 200 2085 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/108 HTTP/1.1" 200 101 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/110 HTTP/1.1" 200 101 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/109 HTTP/1.1" 200 101 --------------------- >> end captured logging << ---------------------

====================================================================== FAIL: test_sightings (tests.testlive_comprehensive.TestComprehensive)

Traceback (most recent call last): File "/home/ankit/PyMISP/tests/testlive_comprehensive.py", line 723, in test_sightings self.assertEqual(len(s), 2) AssertionError: 1 != 2 -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 1721 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /events HTTP/1.1" 200 1721 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /sightings/add/ HTTP/1.1" 200 126 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /sightings/add/ HTTP/1.1" 200 138 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "POST /sightings/restSearch HTTP/1.1" 405 162 pymisp: ERROR: Something went wrong (405): {'name': 'You do not have permission to use this functionality.', 'message': 'You do not have permission to use this functionality.', 'url': '/sightings/restSearch'} urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/130 HTTP/1.1" 200 101 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): localhost:8080 urllib3.connectionpool: DEBUG: http://localhost:8080 "DELETE /events/131 HTTP/1.1" 200 101 --------------------- >> end captured logging << ---------------------

Name Stmts Miss Cover

pymisp/init.py 33 4 88% pymisp/abstract.py 182 27 85% pymisp/api.py 1425 672 53% pymisp/aping.py 268 92 66% pymisp/exceptions.py 36 0 100% pymisp/mispevent.py 689 152 78% pymisp/tools/init.py 18 0 100% pymisp/tools/abstractgenerator.py 43 32 26% pymisp/tools/asnobject.py 14 8 43% pymisp/tools/create_misp_object.py 74 27 64% pymisp/tools/domainipobject.py 14 8 43% pymisp/tools/elfobject.py 70 13 81% pymisp/tools/emailobject.py 48 37 23% pymisp/tools/ext_lookups.py 31 19 39% pymisp/tools/fail2banobject.py 12 6 50% pymisp/tools/fileobject.py 62 9 85% pymisp/tools/genericgenerator.py 8 5 38% pymisp/tools/geolocationobject.py 14 8 43% pymisp/tools/load_warninglists.py 15 9 40% pymisp/tools/machoobject.py 67 13 81% pymisp/tools/neo4j.py 39 27 31% pymisp/tools/openioc.py 94 81 14% pymisp/tools/peobject.py 102 21 79% pymisp/tools/sbsignatureobject.py 11 7 36% pymisp/tools/stix.py 21 15 29% pymisp/tools/vtreportobject.py 46 31 33% tests/init.py 0 0 100% tests/testlive_comprehensive.py 731 51 93%

TOTAL 4167 1374 67%

Ran 28 tests in 31.209s

FAILED (SKIP=1, failures=4) `

Any idea on how to configure the permission for testlive_comprehensive.py ?

Rafiot commented 5 years ago

Permissions are cool, the test actually makes sure the permissions fail for that test.

You have lots of other issues that are probably related to the time on the server you're running the tests against. If you're running a VM, try to reboot it. The tests are very time sensitive.

Try to enter the command date on the server and on the computer you're running the tests from if there is a difference, that's the problem.

datatern commented 5 years ago

Yes, indeed I'm using a VM (MISP_2.4.96). My query is, where should I run date command (please provide path and/or filename), and also if there is a time difference, how do we rectify it?

Rafiot commented 5 years ago

Makes sense then: I discovered that sometimes, the virtual machine isn't in sync with the host system if the host system sent to sleep for example. It fixes itself if you reboot the VM (there are probably other ways, but rebooting is the easiest).

Just to be sure: did you update MISP in the VM? Because there were lots of bugfixes since 2.4.96 and the tests will definitely fail for other reasons.

To check the date, you can run date in the terminal from anywhere in the VM, and compare it with the date on your host computer. If they differ, it means your VM drifted and you need to reboot it.

datatern commented 5 years ago

Hi @Rafiot, as suggested, I have updated the MISP VM image (now set to 2.4.102 ). Also I've checked the date which is perfectly in sync. Still facing issues with the permissions, this time with test_search_csv

Got the following output:

`.....ERROR [aping.py:72 - _check_response() ] Something went wrong (405): {'name': 'You do not have permission to use this functionality.', 'message': 'You do not have permission to use this functionality.', 'url': '/events/publish/4'} F..........S...........

FAIL: test_search_csv (tests.testlive_comprehensive.TestComprehensive)

Traceback (most recent call last): File "/home/ankit/PyMISP/tests/testlive_comprehensive.py", line 813, in test_search_csv self.assertEqual(len(csv), 2) AssertionError: 0 != 2 -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events HTTP/1.1" 200 1744 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events HTTP/1.1" 200 2376 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/publish/4 HTTP/1.1" 405 159 pymisp: ERROR: Something went wrong (405): {'name': 'You do not have permission to use this functionality.', 'message': 'You do not have permission to use this functionality.', 'url': '/events/publish/4'} urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/5c5c1db1-9814-48b1-9809-03390a00020f HTTP/1.1" 200 1743 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/publish/4 HTTP/1.1" 200 77 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 284 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 284 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 284 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 130 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 284 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 130 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 511 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 284 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "POST /events/restSearch HTTP/1.1" 200 130 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "DELETE /events/4 HTTP/1.1" 200 99 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 127.0.0.1:8080 urllib3.connectionpool: DEBUG: http://127.0.0.1:8080 "DELETE /events/5 HTTP/1.1" 200 99 --------------------- >> end captured logging << ---------------------

Name Stmts Miss Cover

pymisp/init.py 33 4 88% pymisp/abstract.py 182 27 85% pymisp/api.py 1425 637 55% pymisp/aping.py 268 70 74% pymisp/exceptions.py 36 0 100% pymisp/mispevent.py 689 149 78% pymisp/tools/init.py 18 0 100% pymisp/tools/abstractgenerator.py 43 32 26% pymisp/tools/asnobject.py 14 8 43% pymisp/tools/create_misp_object.py 74 27 64% pymisp/tools/domainipobject.py 14 8 43% pymisp/tools/elfobject.py 70 13 81% pymisp/tools/emailobject.py 48 37 23% pymisp/tools/ext_lookups.py 31 19 39% pymisp/tools/fail2banobject.py 12 6 50% pymisp/tools/fileobject.py 62 9 85% pymisp/tools/genericgenerator.py 8 5 38% pymisp/tools/geolocationobject.py 14 8 43% pymisp/tools/load_warninglists.py 15 9 40% pymisp/tools/machoobject.py 67 13 81% pymisp/tools/neo4j.py 39 27 31% pymisp/tools/openioc.py 94 81 14% pymisp/tools/peobject.py 102 21 79% pymisp/tools/sbsignatureobject.py 11 7 36% pymisp/tools/stix.py 21 15 29% pymisp/tools/vtreportobject.py 46 31 33% tests/init.py 0 0 100% tests/testlive_comprehensive.py 731 22 97%

TOTAL 4167 1285 69%

Ran 28 tests in 32.050s

FAILED (SKIP=1, failures=1)

` Please help

Rafiot commented 5 years ago

As I said earlier, the permission thing isn't a test failing, but a logging message (I'll find a way to disable that, it's confusing).

I cannot reproduce the error on the CSV export so that's super weird... And you didn't have it before.