search

Graylog2 / graylog-plugin-threatintel

Graylog Processing Pipeline functions to enrich log messages with IoC information from threat intelligence databases
Other
148 stars 19 forks source link

Threat Intel plugin creates and starts data adapters without asking #55

Closed joschi closed 6 years ago

joschi commented 6 years ago

The Threat Intelligence plugin creates a battery of data adapters which access remote servers which might not be possible or might not be wanted at all.

https://github.com/Graylog2/graylog-plugin-threatintel/blob/e0e4444bff479922f5f87252e89d511231c28dbe/src/main/java/org/graylog/plugins/threatintel/ThreatIntelPluginConfiguration.java#L52-L59

https://github.com/Graylog2/graylog-plugin-threatintel/blob/e0e4444bff479922f5f87252e89d511231c28dbe/src/main/java/org/graylog/plugins/threatintel/migrations/V20170815111700_CreateThreatIntelLookupTables.java#L54-L58

https://github.com/Graylog2/graylog-plugin-threatintel/blob/e0e4444bff479922f5f87252e89d511231c28dbe/src/main/resources/org/graylog/plugins/threatintel/migrations/V20170815111700_CreateThreatIntelLookupTables-content_pack.json

2017-09-26 15:58:24,702 INFO : org.graylog2.lookup.LookupTableService - Data Adapter abuse-ch-ransomware-ip/59ca5ca848e7b9aa8938da44 [@2722d5d6] STARTING
2017-09-26 15:58:24,706 INFO : org.graylog2.lookup.LookupTableService - Data Adapter otx-ip/59ca5ca848e7b9aa8938da46 [@57abeafd] STARTING
2017-09-26 15:58:24,709 INFO : org.graylog2.lookup.LookupTableService - Data Adapter spamhaus-drop/59ca5ca848e7b9aa8938da43 [@561b68a1] STARTING
2017-09-26 15:58:24,709 INFO : org.graylog2.lookup.LookupTableService - Data Adapter whois/59ca5ca948e7b9aa8938da48 [@674672ac] STARTING
2017-09-26 15:58:24,710 INFO : org.graylog2.lookup.LookupTableService - Data Adapter whois/59ca5ca948e7b9aa8938da48 [@674672ac] RUNNING
2017-09-26 15:58:24,710 INFO : org.graylog2.lookup.LookupDataAdapterRefreshService - Adding job for <whois/59ca5ca948e7b9aa8938da48/@674672ac> [interval=3600000ms]
2017-09-26 15:58:24,710 INFO : org.graylog2.lookup.LookupTableService - Data Adapter tor-exit-node/59ca5ca948e7b9aa8938da47 [@33e5eb4e] STARTING
2017-09-26 15:58:24,713 INFO : org.graylog2.lookup.LookupTableService - Data Adapter abuse-ch-ransomware-domains/59ca5ca848e7b9aa8938da45 [@7c1224ac] STARTING
2017-09-26 15:58:24,730 INFO : org.graylog2.lookup.LookupTableService - Cache tor-exit-node-list-cache/59ca5ca848e7b9aa8938da3f [@2b8843ed] STARTING
2017-09-26 15:58:24,732 INFO : org.graylog2.lookup.LookupTableService - Cache otx-ip-cache/59ca5ca848e7b9aa8938da40 [@47ea444] STARTING
2017-09-26 15:58:24,732 INFO : org.graylog2.lookup.LookupTableService - Cache tor-exit-node-list-cache/59ca5ca848e7b9aa8938da3f [@2b8843ed] RUNNING
2017-09-26 15:58:24,733 INFO : org.graylog2.lookup.LookupTableService - Cache otx-ip-cache/59ca5ca848e7b9aa8938da40 [@47ea444] RUNNING
2017-09-26 15:58:24,734 INFO : org.graylog2.lookup.LookupTableService - Cache abuse-ch-ransomware-cache/59ca5ca848e7b9aa8938da3d [@5756c78d] STARTING
2017-09-26 15:58:24,735 INFO : org.graylog2.lookup.LookupTableService - Cache abuse-ch-ransomware-cache/59ca5ca848e7b9aa8938da3d [@5756c78d] RUNNING
2017-09-26 15:58:24,735 INFO : org.graylog2.lookup.LookupTableService - Data Adapter otx-ip/59ca5ca848e7b9aa8938da46 [@57abeafd] RUNNING
2017-09-26 15:58:24,736 INFO : org.graylog2.lookup.LookupTableService - Cache whois-cache/59ca5ca848e7b9aa8938da3e [@5ae4fa4f] STARTING
2017-09-26 15:58:24,737 INFO : org.graylog2.lookup.LookupTableService - Cache whois-cache/59ca5ca848e7b9aa8938da3e [@5ae4fa4f] RUNNING
2017-09-26 15:58:24,737 INFO : org.graylog2.lookup.LookupTableService - Cache spamhaus-e-drop-cache/59ca5ca848e7b9aa8938da42 [@657571e9] STARTING
2017-09-26 15:58:24,738 INFO : org.graylog2.lookup.LookupTableService - Cache spamhaus-e-drop-cache/59ca5ca848e7b9aa8938da42 [@657571e9] RUNNING

Errors on my development machine after a fresh installation with no manual interaction:

2017-09-26 15:58:24,844 ERROR: org.graylog2.plugin.lookup.LookupDataAdapter - Couldn't start data adapter <abuse-ch-ransomware-domains/59ca5ca848e7b9aa8938da45/@7c1224ac>
javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:1.8.0_144]
    [...]
    at org.graylog2.lookup.adapters.dsvhttp.HTTPFileRetriever.fetchFileIfNotModified(HTTPFileRetriever.java:58) ~[classes/:?]
    at org.graylog2.lookup.adapters.DSVHTTPDataAdapter.doStart(DSVHTTPDataAdapter.java:95) ~[classes/:?]
    at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [classes/:?]
    at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [guava-23.0.jar:?]
    at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [guava-23.0.jar:?]
2017-09-26 15:58:24,845 ERROR: org.graylog2.plugin.lookup.LookupDataAdapter - Couldn't start data adapter <tor-exit-node/59ca5ca948e7b9aa8938da47/@33e5eb4e>
javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
[...]
bernd commented 6 years ago

@joschi What is your suggestion? :smiley:

joschi commented 6 years ago

@bernd It would be fine to create these data adapters if they wouldn't pull data immediately. No idea if the data adapter interface and implementation allows to do that.

lennartkoopmann commented 6 years ago

I agree. Can we use this part of System/Configurations do enable/disable?

screen shot 2017-09-27 at 2 00 03 pm

kroepke commented 6 years ago

For now I'll use the flags in System configuration for those adapters/tables that actively refresh something. The others are fine, because they will trigger external requests only when they are being used explicitly.

In the case when they are disabled, the adapters will throw an exception during their start, cause the UI to display a warning icon and message on the overview pages.