search

crits / crits_services

CRITs Services Collection
182 stars 130 forks source link

TAXII Import Preview failures #236

Open apolkosnik-old opened 8 years ago

apolkosnik-old commented 8 years ago

I've started testing with Hailataxii feeds, and I'm seeing every other line shows a failure e.g.: Failures 1 Observable (None): No valid CybOX object_ found!

apolkosnik-old commented 8 years ago

Then on the next page nothing gets imported, and everything looks like this:

Observable (None): No valid CybOX object_ found!
Indicator (http://some.bad.com/wp-admin/badhtm/): 'StructuredText' object has no attribute 'strip'
brlogan commented 8 years ago

This is due to Hail a TAXII's extensive use of idrefs. I'll submit a PR with some changes for this soon.

jleona27 commented 8 years ago

I am also experiencing this issue. Seems like every other line is "No valid CybOX object." Is there a way to automate the polling of the Taxii feeds on a daily basis?

mgoffin commented 8 years ago

Extensive use of idrefs? Is that common or unique to them? I hope common otherwise there is more fuel for my hatred of these standards :)

There is a taxii_agent script which was originally designed to allow you to build a cronjob for polling. But I have no clue what the state is since we don't use taxii at all.

jleona27 commented 8 years ago

Interesting. What do you use? It seems like you and others have put a lot of work into the development to not use it.

mgoffin commented 8 years ago

I put the work in early on because I was forced to (prior to open source release). I put the extra work in after open sourcing to undo what I was forced to do :)

ThreatExchange is the platform of choice for sharing data now-a-days.

jleona27 commented 8 years ago

Ahh I see. Is ThreatExchange compatible with Crits via a service or feed?

mgoffin commented 8 years ago

https://github.com/crits/crits_services/tree/master/threatexchange

jleona27 commented 8 years ago

Thank you sir!

apolkosnik-old commented 8 years ago

Some more failures (replaced the original domains):

Observable (None): No valid CybOX object found! Observable (None): No valid CybOX object found! Observable (None): No valid CybOX object found! Indicator (http://something.something.something/somenting/file.php): 'StructuredText' object has no attribute 'strip' STIX Package (Failed to create STIX/CybOX from XML): 'module' object has no attribute 'version' Observable (None): No valid CybOX object found! Observable (None): No valid CybOX object found! Observable (None): No valid CybOX object found! Observable (None): No valid CybOX object_ found! Sample (file.php): ValidationError (Sample:None) (StringField only accepts string values: ['description']) Indicator (http://somethingelse.somethingelse/somethingelse/file.php): 'StructuredText' object has no attribute 'strip'

brlogan commented 8 years ago

I should have some fixes for you soon. Can you please tell me which Hail A TAXII feeds you are using and a rough time range that you are polling? I'd like to do some testing against those in particular before submitting a PR. Thanks!

apolkosnik commented 8 years ago

I was polling Abuse_ch and maybe another one for the last 24 hours at the time of testing. On Jun 24, 2016 3:57 AM, "Bradley Logan" notifications@github.com wrote:

I should have some fixes for you soon. Can you please tell me which Hail A TAXII feeds you are using and a rough time range that you are polling? I'd like to do some testing against those in particular before submitting a PR. Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/crits/crits_services/issues/236#issuecomment-228281240, or mute the thread https://github.com/notifications/unsubscribe/AA_KPp2fNYw7UpzVdTyA_f46NU5YAWjhks5qO43hgaJpZM4I6LVP .

jleona27 commented 8 years ago

After polling the dataForLast7DaysOnly feed, it kept timing out and running out of memory before the poll would finish. I may have tried 10 or so times, but not 751 times! Something must have gone wrong because this is what happens when I check the saved polls. The delete button doesn't work and I wouldn't want to have to click it 750 times anyway! Haha. Any ideas? untitled

brlogan commented 8 years ago

The updates in PR #246 should address a number of these issues, though TTPs are still not supported because I haven't figured out the best way to import that data into CRITs. I'm open to suggestions.

mgoffin commented 8 years ago

Is the issue with bajillions of poll listings fixed?

brlogan commented 8 years ago

I have been unable to reproduce the crapload of saved polls issue; I've never seen anything like it in all my testing. I believe the bigger issue is handling of feeds that contain a massive amount of data. I think we can safely close this issue as its original intent was addressed in PR #246. The remaining massive feed issue is specifically covered in #266.

jleona27 commented 8 years ago

The massive number of polls with errors may have been an anomaly. We fixed it in my case by clearing out the mongod I believe. I too am still having problems with the very large polls.