search

gcrahay / otx_misp

Imports Alienvault OTX pulses to a MISP instance
Other
52 stars 42 forks source link

Error otx-misp import with the latest MISP #37

Open V1D1AN opened 4 years ago

V1D1AN commented 4 years ago

Description

Versions

Traceback or error

When i execute: otx-misp -o xxxxxxxxxxxxxxxxxx -s https://127.0.0.1 -m xxxxxxxxxxxxxxxxxxxxxxxx

I have always the error :

Error: 'TypeError' object has no attribute 'message'

J-Robinson042519 commented 4 years ago

Dang sorry to see this. I have the same issue. I also found -d works and with -vv it shows the OTX side is working but not the commit and -vv does not show anything more when -d is removed.

Name: pymisp Version: 2.4.122

JRobinsonNS commented 4 years ago

switched to using cabby to pull from OTX and then use stix-to-misp.py to upload.

https://github.com/MISP/MISP-STIX-Converter

V1D1AN commented 4 years ago

Hi,

I'm going to test this method. I don't know cabby, so it's going to be time to test the tool.

Can Cabby export in stix format?

Thanks for your idea

J-Robinson042519 commented 4 years ago

It does stix by default. Here is what we are doing for this part.

We are also looking at contributing back to misp-stix-converter an option to add tags to the uploaded items.

V1D1AN commented 4 years ago

Hi,

I have always the same error: taxii-poll --path https://otx.alienvault.com/taxii/poll -c user_AlienVault --username MY_API_KEY --dest-dir /tmp

2020-03-04 20:25:03,098 INFO: Polling using data binding: ALL 2020-03-04 20:25:03,100 INFO: Sending Poll_Request to https://otx.alienvault.com/taxii/poll 2020-03-04 20:25:06,899 ERROR: FAILURE: There was a failure while executing the message handler

If I do this command: taxii-poll -v --host otx.alienvault.com --https -c user_AlienVault --discovery /taxii/poll --username MY_API_KEY --dest-dir /tmp

Nota: I have add the verbose option. I have this logs:

2020-03-04 20:48:51,191 cabby.cli.poll INFO: Polling using data binding: ALL 2020-03-04 20:48:51,192 cabby.dispatcher INFO: Sending Discovery_Request to https://otx.alienvault.com/taxii/poll 2020-03-04 20:48:51,193 cabby.dispatcher DEBUG: Request:

2020-03-04 20:48:51,195 urllib3.connectionpool DEBUG: Starting new HTTPS connection (1): otx.alienvault.com:443 2020-03-04 20:48:54,443 urllib3.connectionpool DEBUG: https://otx.alienvault.com:443 "POST /taxii/poll HTTP/1.1" 200 441 2020-03-04 20:48:54,444 cabby.dispatcher DEBUG: Response:

urn:taxii.mitre.org:protocol:https:1.0 https://otx.alienvault.com/taxii/poll urn:taxii.mitre.org:message:xml:1.1 OTX Taxii Polling urn:taxii.mitre.org:protocol:https:1.0 https://otx.alienvault.com/taxii/collections urn:taxii.mitre.org:message:xml:1.1 OTX Taxii Collections urn:taxii.mitre.org:protocol:https:1.0 https://otx.alienvault.com/taxii/discovery urn:taxii.mitre.org:message:xml:1.1 OTX Taxii Discovery 2020-03-04 20:48:54,449 cabby.client11.Client11 INFO: 3 services discovered 2020-03-04 20:48:54,450 cabby.dispatcher INFO: Sending Poll_Request to https://otx.alienvault.com/taxii/poll 2020-03-04 20:48:54,450 cabby.dispatcher DEBUG: Request: FULL 2020-03-04 20:48:54,452 urllib3.connectionpool DEBUG: Starting new HTTPS connection (1): otx.alienvault.com:443 2020-03-04 20:48:57,457 urllib3.connectionpool DEBUG: https://otx.alienvault.com:443 "POST /taxii/poll HTTP/1.1" 200 269 2020-03-04 20:48:57,458 cabby.dispatcher DEBUG: Response: There was a failure while executing the message handler 2020-03-04 20:48:57,459 cabby.cli.commons ERROR: FAILURE: There was a failure while executing the message handler Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/cabby/cli/commons.py", line 171, in run_client run_func(client, args.uri, args) File "/usr/local/lib/python3.6/dist-packages/cabby/cli/poll.py", line 139, in _runner for counter, block in enumerate(blocks, 1): File "/usr/local/lib/python3.6/dist-packages/cabby/client11.py", line 485, in poll service_type=const.SVC_POLL) File "/usr/local/lib/python3.6/dist-packages/cabby/abstract.py", line 190, in _execute_request timeout=self.timeout) File "/usr/local/lib/python3.6/dist-packages/cabby/dispatcher.py", line 63, in send_taxii_request raise UnsuccessfulStatusError(obj) cabby.exceptions.UnsuccessfulStatusError: FAILURE: There was a failure while executing the message handler If you have any idea ....
J-Robinson042519 commented 4 years ago

When using "taxii-poll --path https://otx.alienvault.com/taxii/poll -c user_AlienVault --username MY_API_KEY --dest-dir /tmp" the user_AlienVault does not have any alerts in it. Did you subscribe to someone and try to poll their feed? Also, the --host and --path were frustrating for me as well.

V1D1AN commented 4 years ago

Yes, I have subscribe to the AlienVault user The official of Alienvault.

I have change "taxii-poll --path https://otx.alienvault.com/taxii/poll -c AlienVault --username MY_API_KEY --dest-dir /tmp"

But i have the same error

J-Robinson042519 commented 4 years ago

something is weird with that one. I just subscribed to user jnazario and I get the same error but go back to Metadefender and get a 504 then ran it again and got a 502 then again and it worked (bottom). It might be a busy server or gremlins in the interwebs or something.

taxii-collections --path https://otx.alienvault.com/taxii/collections --username MY_KEY --password foo 2020-03-04 21:19:34,189 INFO: Sending Collection_Information_Request to https://otx.alienvault.com/taxii/collections === Data Collection Information === Collection Name: user_AlienVault Collection Type: DATA_FEED Available: True Collection Description: Data feed for user: AlienVault Supported Content: All === Polling Service Instance === Poll Protocol: urn:taxii.mitre.org:protocol:https:1.0 Poll Address: https://otx.alienvault.com/taxii/poll Message Binding: urn:taxii.mitre.org:message:xml:1.1

=== Data Collection Information === Collection Name: user_Metadefender Collection Type: DATA_FEED Available: True Collection Description: Data feed for user: Metadefender Supported Content: All === Polling Service Instance === Poll Protocol: urn:taxii.mitre.org:protocol:https:1.0 Poll Address: https://otx.alienvault.com/taxii/poll Message Binding: urn:taxii.mitre.org:message:xml:1.1

=== Data Collection Information === Collection Name: user_jnazario Collection Type: DATA_FEED Available: True Collection Description: Data feed for user: jnazario Supported Content: All === Polling Service Instance === Poll Protocol: urn:taxii.mitre.org:protocol:https:1.0 Poll Address: https://otx.alienvault.com/taxii/poll Message Binding: urn:taxii.mitre.org:message:xml:1.1

=== Data Collection Information === Collection Name: user_ Collection Type: DATA_FEED Available: True Collection Description: Data feed for user: Supported Content: All === Polling Service Instance === Poll Protocol: urn:taxii.mitre.org:protocol:https:1.0 Poll Address: https://otx.alienvault.com/taxii/poll Message Binding: urn:taxii.mitre.org:message:xml:1.1

taxii-poll --path https://otx.alienvault.com/taxii/poll --collection user_Metadefender --username --password foo -l 5 --dest-dir /tmp/otx-poll/ 2020-03-04 21:29:01,803 INFO: Polling using data binding: ALL 2020-03-04 21:29:01,804 INFO: Sending Poll_Request to https://otx.alienvault.com/taxii/poll 2020-03-04 21:29:37,822 INFO: Content block saved to /tmp/otx-poll/user_Metadefender_d3c4a5e052b7a89cfbf5b627207ced11 2020-03-04 21:29:37,999 INFO: Content block saved to /tmp/otx-poll/user_Metadefender_dae6ff33b6bcb2921129a307bb4a8333 2020-03-04 21:29:38,279 INFO: Content block saved to /tmp/otx-poll/user_Metadefender_b08ee881668b25a99a864f51c88fbd1f 2020-03-04 21:29:38,564 INFO: Content block saved to /tmp/otx-poll/user_Metadefender_9a4817a0f425daa605943a3cd6af1ba9 2020-03-04 21:29:38,725 INFO: Content block saved to /tmp/otx-poll/user_Metadefender_13bc24ef63e945c99ddb15339c89b23f 2020-03-04 21:29:38,725 INFO: 5 blocks polled

V1D1AN commented 4 years ago

I have subscribe to the user popularmalware and it's work for me... But if I test with the user_Alienvault .. I have the error .. grrrrr

perhaps a problem with the Alienvault user ..

J-Robinson042519 commented 4 years ago

Good to hear it worked for you. Maybe we need to take the debug to Alienvault.

rustybrooks commented 4 years ago

Hi,

I work on the OTX team. I'll look into this as soon as I can and get back to you.

V1D1AN commented 4 years ago

hi, can you see for otx-misp and why we can’t poll with the collection user_AlienVault .. I know that that last problem doesn't belong in this github.

rustybrooks commented 4 years ago

I don't know too much about otx-misp but it's probably the same problem that occurs with Cabby, which is that there's an error encountered when trying to render the AlienVault user's feed to STIX.

rustybrooks commented 4 years ago

OK, so, I believe that I have fixed the problem related to not being able to get TAXII results for the collection "user_AlienVault" (and possibly some other users had the sample problem, the error was related to converting some types of data to STIX and only affected users who had examples of that kind of data)

If that doesn't fix the otx-misp problems let me know and I can look deeper into it.

As an aside - if you're using Cabby for anything other than testing I would recommend limiting the results by date. The AlienVault user has a LOT of pulses/indicators, and not providing a date means that it downloads all of them, which is quite slow/taxing. Providing a date will only get pulses that are new or have been modified since that date, so if you know you fetched N hours ago, then limiting to that time frame will greatly reduce fetch time and traffic.

Such as $ taxii-poll --path https://otx.alienvault.com/taxii/poll -c user_AlienVault --username YOURKEYHERE --password foo --dest-dir /tmp --begin 2020-03-01

Ideally you'd use --begin with whatever your last fetch date was, and just keeping feeding "new" indicators into whatever you're using.

V1D1AN commented 4 years ago

yes it works :) I will keep your username if i have another problem with OTX :) Where we push this discussion for other persons ??

V1D1AN commented 4 years ago

Thanks J-Robinson042519 for misp-stix-converter

I have changed "/usr/local/lib/python3.6/dist-packages/misp_stix_converter/converters/convert.py"

I added: misp_event.add_tag('OTX') misp_event.add_tag('tlp:white') misp_event.publish()

Thanks a lot J-Robinson042519 and rustybrooks

J-Robinson042519 commented 4 years ago

@V1D1AN - did you commit this to the converter project? If not please do or make it a config parameter and add it to the project. I don't think we will get to it for a while.

Does anyone know how to limit the polling on otx-misp like @rustybrooks did for taxii-poll?

JRobinsonNS commented 4 years ago

looks like with the fix from @rustybrooks did not help otx-misp...still getting Error: 'TypeError' object has no attribute 'message'. One theory we have is it is coming from MISP with its Py 2.7 conversation issues.

J-Robinson042519 commented 4 years ago

Thanks J-Robinson042519 for misp-stix-converter

I have changed "/usr/local/lib/python3.6/dist-packages/misp_stix_converter/converters/convert.py"

I added: misp_event.add_tag('OTX') misp_event.add_tag('tlp:white') misp_event.publish()

Thanks a lot J-Robinson042519 and rustybrooks

@V1D1AN - Where in the code did you add the misp_event section? around like 210 looks about right.

davecabio commented 4 years ago

Good morning, I'm having the same problem. How did you solved it? Will the bug be resolved in otx_misp?

MISP version: 2.4 Command: otx-misp --otx "MYKEY" --server "https://localhost" --misp "MYKEY" --discover-tags -n -a --threat-level high -t "2020-03-26T19:20:30+01:00" -vv Result: Error: 'TypeError' object has no attribute 'message'

If I add -d, it works perfectly.

davecabio commented 4 years ago

OK, so, I believe that I have fixed the problem related to not being able to get TAXII results for the collection "user_AlienVault" (and possibly some other users had the sample problem, the error was related to converting some types of data to STIX and only affected users who had examples of that kind of data)

If that doesn't fix the otx-misp problems let me know and I can look deeper into it.

As an aside - if you're using Cabby for anything other than testing I would recommend limiting the results by date. The AlienVault user has a LOT of pulses/indicators, and not providing a date means that it downloads all of them, which is quite slow/taxing. Providing a date will only get pulses that are new or have been modified since that date, so if you know you fetched N hours ago, then limiting to that time frame will greatly reduce fetch time and traffic.

Such as $ taxii-poll --path https://otx.alienvault.com/taxii/poll -c user_AlienVault --username YOURKEYHERE --password foo --dest-dir /tmp --begin 2020-03-01

Ideally you'd use --begin with whatever your last fetch date was, and just keeping feeding "new" indicators into whatever you're using.

May you help us? As I'm seeing I'm not the only one with this issue.

Thanks!

rustybrooks commented 4 years ago

I've actually been playing with otx-misp today, and I have the same problem as you. I think what it comes down to is that otx-misp relies on another library called pymisp. The most recent version of pymisp has a PyMISP() object that does not take the same init parameters as it used to, so otx-misp fails.

I've tried downgrading to an older version of pymisp, and that gets past this problem, but there are other problems down the road. pymisp is handling the MISP api calls and my guess is that how these calls works has changed.

If I update to a newer pymisp and fix the initialization problems then I run into the fact that pymisp uses different functions - the "new_event" function no longer exists for example. So I think it's fixable but possibly not trivial to do so. I'm still looking into it, and other alternatives such as offering a MISP feed directly from OTX

rustybrooks commented 4 years ago

Note, a fork of otx-misp has fixed this problem I believe: https://github.com/obert01/otx_misp

I myself did not go down this road. Here's the problem as I see it

  1. MISP and pymisp (which otx-misp uses) are tightly coupled - you generally need a newish version of pymisp to interact with MISP or things don't work
  2. pymisp underwent a fairly large change to how it works a few months ago - it created enough incompatibility that it's a pretty big job to update otx-misp - look at the last commit in the repo above. It's a bit rude to have an API change this large in such a minor release of pymisp
  3. because MISP and pymisp are tightly coupled you can't even just roll back to an older version of pymisp, because it won't work with new MISP server installs.
  4. because of this, even though it's now fixed in the repo above, that repo might not work with older MISP servers.

I kind of gave up on fixing this and I'm going to instead, I think, make MISP feeds for OTX. I did the first one here, which lets you get a MISP feed from a group, as a POC https://otx.alienvault.com/api/v1/groups/GROUPID/misp_feed you can add this as a feed in MISP, you'll need to give X-OTX-API-KEY: YOURKEYHERE in the headers box.

I will probably add more MISP feed support in the near future

Note: edited to fix URL

Tubbs10 commented 4 years ago

Note, a fork of otx-misp has fixed this problem I believe: https://github.com/obert01/otx_misp

I myself did not go down this road. Here's the problem as I see it

  1. MISP and pymisp (which otx-misp uses) are tightly coupled - you generally need a newish version of pymisp to interact with MISP or things don't work
  2. pymisp underwent a fairly large change to how it works a few months ago - it created enough incompatibility that it's a pretty big job to update otx-misp - look at the last commit in the repo above. It's a bit rude to have an API change this large in such a minor release of pymisp
  3. because MISP and pymisp are tightly coupled you can't even just roll back to an older version of pymisp, because it won't work with new MISP server installs.
  4. because of this, even though it's now fixed in the repo above, that repo might not work with older MISP servers.

I kind of gave up on fixing this and I'm going to instead, I think, make MISP feeds for OTX. I did the first one here, which lets you get a MISP feed from a group, as a POC https://otx.alienvault.com/api/groups/GROUPID/misp_feed you can add this as a feed in MISP, you'll need to give X-OTX-API-KEY: YOURKEYHERE in the headers box.

I will probably add more MISP feed support in the near future

Hi,

How am I supposed to get this feed working? Is the URL really GROUPID? How about source format in MISP? Is it MISP Feed then? For what I tested it didn't work with this information.

Thank you.

rustybrooks commented 4 years ago

GROUPID should be the numeric group id of the group you want to sync. For example this is a newly created public group: https://otx.alienvault.com/group/867/pulses

It's a MISP feed, i.e. it has a /manifest.json and then a json file per pulse in the feed.

This url would be to sync a single group. We'll be adding the ability to sync a user and probably also your subscription

AntoninHL commented 4 years ago

Hi, thanks @rustybrooks for the alternative method, but I always have an Failed Job when I try to setup a MISP Feed with https://otx.alienvault.com/api/groups/GROUPID/misp_feed with X-OTX-API-KEY: YOURKEYHERE in the headers box.

On OTX Side, I created a group with 2 pulses.

Any idea?

Thanks a lot !

rustybrooks commented 4 years ago

In that URL you need to replace GROUPID with the integer group id for your group. For example if you go to a group in the UI, the url will look something like https://otx.alienvault.com/group/927/pulses and the GROUPID value would be replaced with 927 to be like https://otx.alienvault.com/api/groups/927/misp_feed

And of course YOURKEYHERE in the headers would have to be your OTX api key.

If you have access to the MISP logs I think you'll find errors in there. I'm not currently running a MISP instance but I could maybe fire one up again if necessary to try to check it out.

AntoninHL commented 4 years ago

Thanks @rustybrooks .

I had, of course, replaced all the values to be changed. ;-)

Here are logs:

`2020-05-23 18:52:53 Error: Could not get new event uuids for feed 69. [Exception] Fetching the 'https://otx.alienvault.com/api/groups/928/misp_feed/manifest.json' failed with HTTP error 404: Not Found Stack Trace:

0 /var/www/MISP/app/Model/Feed.php(194): Feed->feedGetUri(Array, 'https://otx.ali...', Object(HttpSocket), true)

1 /var/www/MISP/app/Model/Feed.php(137): Feed->downloadManifest(Array, Object(HttpSocket))

2 /var/www/MISP/app/Model/Feed.php(845): Feed->getNewEventUuids(Array, Object(HttpSocket))

3 /var/www/MISP/app/Console/Command/ServerShell.php(213): Feed->downloadFromFeedInitiator('69', Array, '4542')

4 /var/www/MISP/app/Console/Command/AppShell.php(35): ServerShell->fetchFeed()

5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()

6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()

7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))

8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')

9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')

10 {main}`

rustybrooks commented 4 years ago

Aw man sorry I had a typo in the URL, it should be https://otx.alienvault.com/api/v1/groups/69/misp_feed I'll see if I can edit my earlier post

AntoninHL commented 4 years ago

👌 Perfect!

It works like a charm!

Many thanks for this workaround @rustybrooks!

I hope we will have soon a new version of otx-misp.

seldem commented 4 years ago

Thanks for this @rustybrooks . It is very nice to have OTX pulses as MISP events. But I guess I found something to report you: In MISP there are "email-dst" and "email-src" attributes. As I understood in OTX only one attribute exist which is "email". When I checked OTX events in my MISP they are always "email-dst". Can you fix it? I believe OTX "email" should map to MISP "email-src". Thanks in advance.

rustybrooks commented 4 years ago

Yeah, I can change that, sure. It probably does make more sense for it to be email-src which I guess is probably the "from" address

martinconnarty commented 4 years ago

Thanks @rustybrooks for this well needed solution. I am quite new to using OTX, but have seen that many authors create new pulses every day, rather than updating an existing one. Unless I'm mistaken, this means that me adding that pulse to a Group is good for that day and no more. Have you managed to explore putting similar MISP manifests on either Users, or just the subscribed pulses? I have had no end of difficulties with trying to get OTX-MISP working which I acknowledge is probably mainly user error/ignorance, this solution would be invaluable to me!

rustybrooks commented 4 years ago

Yeah there are plans to add MISP support to users, and/or making a MISP feed for your "subscription" which includes:

Might come through before the end of this year but is not immediately planned

tonycaud commented 3 years ago

The error 'TypeError' object has no attribute 'message' is happening again...it started very recently for me. In fact, I was receiving an error message stating that I had to move to the latest version of MISP to continue, then after I did, I started getting these TypeError messages.

I've been trying to find a way around this and ended up here.

AFAIK, we were pulling all public pulses in the primary Alienvault feed (I believe that means I'm pulling all public pulses from user Alienvault). There doesn't appear to be a group number for that main public feed. If there is, please let me know as I would love to try getting the feeds using this other method.

We are using a config file, but running it from the manually from the CLI with the verbose switch provided no additional information.

Any help would be appreciated.

rustybrooks commented 3 years ago

@tonycaud have you tried the fork from this project I mentioned above?

tonycaud commented 3 years ago

Since I simply installed using "pip3 install otx-misp" how do I see which I have?

rustybrooks commented 3 years ago

Then you have this one, almost surely. I think you'd have to download the fork from github and install it manually.

tonycaud commented 3 years ago

So, that one throws a chunk of errors, and there isn't an issues page there...

-nevermind...with help from a friend of mine, we changed it from calling Python2.7 to Python3 and now it seems to be working.