Error when Polling Taxii Feed - Name or service not known

[jleona27]

Hey guys. I've been trying to get the newly updated version of the Taxii-service to work for a while now. I THINK I have everything configured correctly. Here are some screen shots of the config and the error I'm getting:

Sorry if these screenshots don't work as intended. Can anyone help me on getting this configured right? Thank you devs so much.

[brlogan]

Try configuring like this (specifically changes to hostname, poll/inbox paths:

I think I might add a configuration example file for this service and/or expand the details provided on the form below each of the fields.

[jleona27]

THANK YOU!! Odd because I tried just /taxii-data/ before but I left "http://" in the hostname field. I guess it doesn't like that? Seems silly that it would throw an error because you include http:// in the hostname but what do I know?! So I'm really new to Crits and have created a Campaign for Hailataxii. How would I link the threat data pulled in from this feed to the Hailataxii Campaign. Hopefully it automatically puts it under the proper source, but I'm unsure about these Campaign things. Thanks for your help! So glad you had the right answer.

*UPDATE: The poll finished and I am not seeing any data anywhere. Does the taxii_service have a default location it puts the threat indicators and data in? I am an admin and have added the Hailataxii source to the list of sources I can view. Thanks :)

[brlogan]

I finally was able to push out a number of improvements to the TAXII service. See #246.

The TAXII service now provides the HailaTAXII example configuration in the README, so hopefully that will help others who run into problems.

@jleona27: You said you did a poll and didn't see any data. Can you try that again and see if it works now? If it still doesn't, I'd be interested to get more details from you regarding exactly what you're seeing. I'm not sure that it makes sense to create a HailaTAXII campaign, but you can attribute any TLO in CRITs to a campaign by opening the TLO's details page and adding a Campaign in the Campaigns section. For the data you pulled in via TAXII, you would have to open each item and select the Campaign you are attributing to. Hope that helps!

[jleona27]

@brlogan I haven't messed with it in a bit. I'll try and pull the updates in and see what happens. Lately, the poll system has been messing up royally. It says I have 753 polls with errors, but I've only tried polling ~20 or so times. Notice the timestamps are today... I haven't told it to poll in weeks. The delete links on the right don't work either.

I have a few other projects I'm working on but I'll jump on this and see if I can't get this issue to clear up. I'm not a programmer by any means so I am just messing around with it and hope I do something right!! Thanks for putting work into this.

[jleona27]

Just updated. This is what I get when polling now.

I tried changing the dates and still got the same error.

[brlogan]

@jleona27: Wow, something clearly screwed up earlier. No idea how you would have ended up with 753 polls. Let's try clearing that out so you can start fresh and we'll see if it happens again. Please go into the Mongo DB shell, and run the following, then make sure those saved polls are gone: db.taxii.content.remove({})

Since you got a 503 when attempting to poll, I'm guessing it was a temporary issue on Hail A TAXII's end. I was just able to successfully poll the last few days of data for that same feed, so please try it again when you have a chance and let me know what happens.

[jleona27]

I didn't set up the Crits instance, but I am working with it. How would I get to the Mongo DB shell?

[brlogan]

You would have to login to the command line on the server. I've toyed with the idea of a delete all button in the Saved Polls interface. Maybe I should go ahead and add that.

[jleona27]

I have access to the crits server command line and can stop/start apache2 and mongodb. Is that the same shell or no?

[thelok]

Should just do a mongo crits or mongo and list your databases:

show dbs

To switch to a db use:

use dbnamehere

[jleona27]

Very helpful guys! Got all those errors to clear out! Polling now.

[jleona27]

@brlogan @thelok I polled dataForLast7daysOnly and specified the window of April to June, just to minimize polling time. That still took a good while. I eventually refreshed the page and it collected some 140,000+ indicators. That was still under the "Unimported" section. Trying to import them now. Where does the threat data go on the crits web interface after you import it? Thanks

[brlogan]

@jleona27: The data goes a number of places depending on what it is. Indicators, Emails, Samples, etc. are imported into their respective sections of CRITs. Incidents are imported as Events. If you have the box checked in the TAXII service configuration, an Event is created for each package based upon its header and all items within that package are related back to that Event.

The code really does need to be improved to better handle those massive data sets, but I haven't gotten around to that yet.

[jleona27]

@brlogan: Thanks for the reply! Yeah I was able to successfully pull in some data but it is hard to find within the Crits web UI. Also, with big polls, the TAXII service just times out. What is the best date range to poll to avoid this? It seems like any window over 30 a day period is out of the question. Thanks for you work so far.

[brlogan]

@jleona27: After importing STIX data into CRITs, you should be presented with a screen listing everything that was imported with links to each item. If that's not sufficient, I recommend turning on the "Pkg Header Events" service configuration option. With it enabled, you can go to the Events page in CRITs and the top items in the list should be those that were just imported.

The amount of data can vary significantly depending on the feed. The feed you are polling just happens to be massive even for a narrow date range. Until we can come up with a better way to handle huge sets of data, you might try polling a different feed.

[jleona27]

@brlogan Thanks so much for the help and advice! What do you recommend for "The maximum number of related items, of each type, that can be selected for a TAXII message."

[brlogan]

@jleona27: Well, the default value of 200 is a pretty good place to start, but it really depends on what your particular system is capable of. The greater the number, the larger the generated STIX document, and the longer it will take to process. You can experiment with some greater values and an Event with a huge number of relationships to see what happens in your case.

[brlogan]

The original reason for creation of this issue (the name or service not known error) has been addressed, so I think this issue can be closed.

[jleona27]

Yes. You have been much help. Do I need to close it?