ninSmith
commented
7 years ago Can you post the corresponding log from Hippocampe/core/logs/hippocampe.log please ?
Thanks
Closed bjuditt closed 7 years ago
ninSmith
commented
7 years ago Can you post the corresponding log from Hippocampe/core/logs/hippocampe.log please ?
Thanks
bjuditt
commented
7 years ago 2017-01-26 10:21:48,822 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:49,447 :: services :: INFO :: type service requested
2017-01-26 10:21:49,447 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:49,450 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:21:49,450 :: services.modules.common.ES :: INFO :: []
2017-01-26 10:21:49,457 :: services :: INFO :: sizeByType service requested
2017-01-26 10:21:49,457 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:49,459 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:21:49,460 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:21:49,464 :: services :: INFO :: sizeBySources service requested
2017-01-26 10:21:49,464 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:49,466 :: services :: INFO :: monitorSources service requested
2017-01-26 10:21:49,467 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:49,469 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:21:49,469 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:21:49,481 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:21:49,481 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:21:49,495 :: services.typeIntel :: INFO :: typeIntel.main launched
2017-01-26 10:21:49,512 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:21:49,512 :: services :: ERROR :: no data
2017-01-26 10:21:49,516 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:21:49,516 :: services :: ERROR :: no data
2017-01-26 10:21:49,516 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:21:49,517 :: services :: ERROR :: no data
2017-01-26 10:21:49,526 :: services.typeIntel :: INFO :: typeIntel.main end
2017-01-26 10:21:49,531 :: services :: INFO :: type service requested
2017-01-26 10:21:49,531 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:49,534 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:21:49,534 :: services.modules.common.ES :: INFO :: []
2017-01-26 10:21:49,541 :: services.typeIntel :: INFO :: typeIntel.main launched
2017-01-26 10:21:49,568 :: services.typeIntel :: INFO :: typeIntel.main end
2017-01-26 10:21:58,154 :: services :: INFO :: shadowbook service requested
2017-01-26 10:21:58,154 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:21:58,161 :: services.shadowbook :: INFO :: shadowbook.initJob launched
2017-01-26 10:21:58,405 :: services.shadowbook :: INFO :: number ongoing job: 1
2017-01-26 10:21:58,406 :: services.shadowbook :: ERROR :: Ongoing job already running
2017-01-26 10:21:58,406 :: services.shadowbook :: INFO :: {'error': 'Ongoing job already running'}
2017-01-26 10:21:58,406 :: services.shadowbook :: INFO :: shadowbook.initJob end
2017-01-26 10:21:58,407 :: services :: ERROR :: shadowbook failed
2017-01-26 10:22:05,177 :: services :: INFO :: jobs service requested
2017-01-26 10:22:05,177 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:05,179 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:05,179 :: services.modules.common.ES :: INFO :: ['jobsType']
2017-01-26 10:22:05,207 :: services.modules.common.ES :: INFO :: index hippocampe exists but type jobs does not
2017-01-26 10:22:05,207 :: services :: ERROR :: no data
2017-01-26 10:22:06,418 :: services :: INFO :: sizeByType service requested
2017-01-26 10:22:06,419 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:06,423 :: services :: INFO :: sizeBySources service requested
2017-01-26 10:22:06,423 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:06,440 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:06,441 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:06,445 :: services :: INFO :: monitorSources service requested
2017-01-26 10:22:06,446 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:06,458 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:06,458 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:06,460 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:06,460 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:06,469 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:06,469 :: services :: ERROR :: no data
2017-01-26 10:22:06,476 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:06,476 :: services :: ERROR :: no data
2017-01-26 10:22:06,501 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:06,501 :: services :: ERROR :: no data
2017-01-26 10:22:06,869 :: services :: INFO :: type service requested
2017-01-26 10:22:06,869 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:06,888 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:06,888 :: services.modules.common.ES :: INFO :: []
2017-01-26 10:22:06,972 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:06,973 :: services.modules.common.ES :: INFO :: []
2017-01-26 10:22:06,989 :: services.typeIntel :: INFO :: typeIntel.main launched
2017-01-26 10:22:07,012 :: services.typeIntel :: INFO :: typeIntel.main end
2017-01-26 10:22:07,515 :: services :: INFO :: sources service requested
2017-01-26 10:22:07,515 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:07,528 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:07,530 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:07,551 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:07,551 :: services :: ERROR :: no data
2017-01-26 10:22:08,593 :: services :: INFO :: hipposcore service requested
2017-01-26 10:22:08,593 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:08,606 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:08,606 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:08,614 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:08,615 :: services :: ERROR :: no data
2017-01-26 10:22:10,120 :: services :: INFO :: sizeByType service requested
2017-01-26 10:22:10,121 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:10,124 :: services :: INFO :: sizeBySources service requested
2017-01-26 10:22:10,125 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:10,141 :: services :: INFO :: monitorSources service requested
2017-01-26 10:22:10,142 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:22:10,146 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:10,146 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:10,153 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:10,153 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:10,157 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 10:22:10,158 :: services.modules.common.ES :: INFO :: ['sourceType']
2017-01-26 10:22:10,174 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:10,174 :: services :: ERROR :: no data
2017-01-26 10:22:10,176 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:10,176 :: services :: ERROR :: no data
2017-01-26 10:22:10,184 :: services.modules.common.ES :: INFO :: index hippocampe exists but type source does not
2017-01-26 10:22:10,184 :: services :: ERROR :: no data
2017-01-26 10:31:07,648 :: services :: INFO :: shadowbook service requested
2017-01-26 10:31:07,648 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:31:07,674 :: services.shadowbook :: INFO :: shadowbook.initJob launched
2017-01-26 10:31:07,759 :: services.shadowbook :: INFO :: number ongoing job: 1
2017-01-26 10:31:07,759 :: services.shadowbook :: ERROR :: Ongoing job already running
2017-01-26 10:31:07,759 :: services.shadowbook :: INFO :: {'error': 'Ongoing job already running'}
2017-01-26 10:31:07,759 :: services.shadowbook :: INFO :: shadowbook.initJob end
2017-01-26 10:31:07,759 :: services :: ERROR :: shadowbook failed
2017-01-26 10:34:59,530 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 10:35:00,183 :: services :: INFO :: type service requested
2017-01-26 10:35:00,183 :: services.modules.common.ES :: INFO :: ES.checkES launched
Shadowbook keeps saying that there's an ongoing job...
Let's try to erase all data, index the feed and have a look to elasticsearch logs.
Please execute the following:
curl -XDELETE localhost:9200/hippocampecurl -XGET localhost:5000/hippocampe/api/v1.0/shadowbookcurl -GET http://localhost:5000/hippocampe/api/v1.0/jobs/var/log/elasticsearch by defaultThanks
bjuditt
commented
7 years ago I just started a job and here to start the logs of Hippocampe
2017-01-26 13:38:08,540 :: services :: INFO :: shadowbook service requested
2017-01-26 13:38:08,540 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 13:38:08,552 :: services.shadowbook :: INFO :: shadowbook.initJob launched
2017-01-26 13:38:08,674 :: services.shadowbook :: INFO :: number ongoing job: 0
2017-01-26 13:38:08,744 :: services.shadowbook :: INFO :: {'job': {u'AVnayHojkM2RJHneGHHd': 'ongoing'}}
2017-01-26 13:38:08,745 :: services.shadowbook :: INFO :: shadowbook.initJob end
2017-01-26 13:38:08,745 :: services.shadowbook :: INFO :: shadowbook.manageJob launched
2017-01-26 13:38:08,745 :: services.shadowbook :: INFO :: shadowbook.startJob launched
2017-01-26 13:38:08,745 :: services.modules.shadowbook.createSessions :: INFO :: createSessions.createSession launched
2017-01-26 13:38:08,746 :: services.modules.shadowbook.createSessions :: INFO :: Authenticated session created for: top_level_url
2017-01-26 13:38:08,747 :: services.modules.shadowbook.createSessions :: INFO :: createSessions.createSessions end
2017-01-26 13:38:08,760 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main launched for dshield_low_DOMAIN.conf
2017-01-26 13:38:08,778 :: services.modules.shadowbook.objects.Source :: INFO :: Source.isActive starts
2017-01-26 13:38:08,784 :: services.modules.shadowbook.objects.Source :: INFO :: E scenario
2017-01-26 13:38:08,915 :: services.modules.shadowbook.objects.Source :: INFO :: Updating https://dshield.org/feeds/suspiciousdomains_Low.txt lastQuery
2017-01-26 13:38:10,195 :: services.modules.shadowbook.searchIntel :: INFO :: searchIntel.littleMsearch launched
2017-01-26 13:38:11,831 :: services.modules.shadowbook.searchIntel :: INFO :: searchIntel.littleMsearch end
2017-01-26 13:38:11,832 :: services.modules.shadowbook.processMsearch :: INFO :: processMsearch.littleSort launched
2017-01-26 13:38:11,837 :: services.modules.shadowbook.processMsearch :: INFO :: processMsearch.littleSort end
2017-01-26 13:38:11,837 :: services.modules.shadowbook.bulkOp :: INFO :: bulkOp.update launched
2017-01-26 13:38:14,928 :: services.modules.shadowbook.bulkOp :: INFO :: bulkOp.update res: (2060, [])
2017-01-26 13:38:14,929 :: services.modules.shadowbook.bulkOp :: INFO :: bulkOp.update end
2017-01-26 13:38:14,929 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main for dshield_low_DOMAIN.conf end
2017-01-26 13:38:14,937 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main launched for shunlist_IP.conf
2017-01-26 13:38:14,940 :: services.modules.shadowbook.objects.Source :: INFO :: Source.isActive starts
2017-01-26 13:38:14,940 :: services.modules.shadowbook.objects.Source :: INFO :: E scenario
2017-01-26 13:38:15,159 :: services.modules.shadowbook.objects.Source :: INFO :: Updating http://autoshun.org/files/shunlist.csv lastQuery
2017-01-26 13:38:16,558 :: services.modules.shadowbook.searchIntel :: INFO :: searchIntel.littleMsearch launched
2017-01-26 13:38:16,565 :: services.modules.shadowbook.processFeed :: ERROR :: processFeed.main failed for shunlist_IP.conf, no idea where it came from
Traceback (most recent call last):
File "/home/moi/Hippocampe/core/services/modules/shadowbook/processFeed.py", line 67, in main
resMsearch = searchIntel.littleMsearch(source.coreIntelligence, source.typeNameESIntel, parsedPage)
File "/home/moi/Hippocampe/core/services/modules/shadowbook/searchIntel.py", line 72, in littleMsearch
res = es.msearch(body = req)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 71, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 1116, in msearch
raise ValueError("Empty value passed for a required argument 'body'.")
ValueError: Empty value passed for a required argument 'body'.
2017-01-26 13:38:16,566 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main for shunlist_IP.conf end
2017-01-26 13:38:16,566 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main launched for abuse_palevo_DOMAIN.conf
2017-01-26 13:38:16,572 :: services.modules.shadowbook.objects.Source :: INFO :: Source.isActive starts
2017-01-26 13:38:16,572 :: services.modules.shadowbook.objects.Source :: INFO :: E scenario
2017-01-26 13:38:16,722 :: services.modules.shadowbook.objects.Source :: INFO :: Updating https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist lastQuery
2017-01-26 13:38:17,866 :: services.modules.shadowbook.searchIntel :: INFO :: searchIntel.littleMsearch launched
2017-01-26 13:38:17,882 :: services.modules.shadowbook.searchIntel :: INFO :: searchIntel.littleMsearch end
2017-01-26 13:38:17,882 :: services.modules.shadowbook.processMsearch :: INFO :: processMsearch.littleSort launched
2017-01-26 13:38:17,883 :: services.modules.shadowbook.processMsearch :: INFO :: processMsearch.littleSort end
2017-01-26 13:38:17,883 :: services.modules.shadowbook.bulkOp :: INFO :: bulkOp.update launched
2017-01-26 13:38:18,370 :: services.modules.shadowbook.bulkOp :: INFO :: bulkOp.update res: (14, [])
2017-01-26 13:38:18,370 :: services.modules.shadowbook.bulkOp :: INFO :: bulkOp.update endOk, it seems to work.
For the error, it is a normal behavior.
Shunlist feed now requires a free personal subscription and I guess you did not change the default configuration for this feed. That's why it raises an error.
Just wait until the end now...
Fingers crossed
bjuditt
commented
7 years ago Result ...
curl -GET http://localhost:5000/hippocampe/api/v1.0/jobs
{
"error": "no data available"
}
[2017-01-26 13:38:11,900][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [dshieldFree_lowDOMAIN]
[2017-01-26 13:38:17,904][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [abuseFree_palevotrackerDOMAIN]
[2017-01-26 13:38:20,220][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [abuseFree_zeustrackerDOMAIN]
[2017-01-26 13:38:21,572][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [malc0deFree_BlacklistIP]
[2017-01-26 13:38:29,441][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [openphishFree_feedURL]
[2017-01-26 13:38:52,113][WARN ][monitor.jvm ] [CfMa-pJ] [gc][13814] overhead, spent [552ms] collecting in the last [1s]
[2017-01-26 13:39:54,776][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [greensnowIP]
[2017-01-26 13:40:14,270][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [binarydefenseFree_banlistIP]
[2017-01-26 13:40:26,067][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [dshieldFree_highDOMAIN]
[2017-01-26 13:41:07,108][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [ciArmyListIP]
[2017-01-26 13:42:11,795][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [malwaredomainsFree_dnsbhDOMAIN]
[2017-01-26 13:42:36,768][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [abuseFree_zeustrackerIP]
[2017-01-26 13:42:39,901][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [abuseFree_feodotrackerIP]
[2017-01-26 13:42:43,667][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [etFree_compromisedIP]
[2017-01-26 13:42:49,651][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [dshieldFree_mediumDOMAIN]
[2017-01-26 13:43:16,619][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [phishtankFree_onlinevalidURL]
[2017-01-26 13:43:54,433][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [abuseFree_sslblIP]
[2017-01-26 13:43:56,227][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [abuseFree_palevotrackerIP]
[2017-01-26 13:43:56,778][INFO ][cluster.metadata ] [CfMa-pJ] [hippocampe/glfTxzAOS9qoH_4pjfPKvg] update_mapping [jobs]
Could you please copy/paste the content of Hippocampe/core/conf/hippo/hippo.conf ?
Thanks
bjuditt
commented
7 years ago Of course :
[api]
debug : False
host : 0.0.0.0
port : 5000
threaded : True
[elasticsearch]
ip : 127.0.0.1
port : 9200
#indexNameES MUST BE LOWERCASE
indexNameES : hippocampe
typeNameESSource : source
typeNameESNew : new
typeNameESJobs: jobs
[shadowbook]
nbThreadPerCPU : 2
[freshness]
#in days
threshold : 1
[schedReport]
#in hours
threshold: 12And what gives curl -GET localhost:9200/hippocampe/_mapping | python -mjson.tool ?
Thanks
bjuditt
commented
7 years ago Good reading ...
"hippocampe": {
"mappings": {
"abuseFree_feodotrackerIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"abuseFree_palevotrackerDOMAIN": {
"properties": {
"domain": {
"type": "keyword"
},
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"abuseFree_palevotrackerIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"abuseFree_sslblIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"port": {
"type": "keyword"
},
"source": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"abuseFree_zeustrackerDOMAIN": {
"properties": {
"domain": {
"type": "keyword"
},
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"abuseFree_zeustrackerIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"binarydefenseFree_banlistIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"ciArmyListIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"dshieldFree_highDOMAIN": {
"properties": {
"domain": {
"type": "keyword"
},
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"dshieldFree_lowDOMAIN": {
"properties": {
"domain": {
"type": "keyword"
},
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"dshieldFree_mediumDOMAIN": {
"properties": {
"domain": {
"type": "keyword"
},
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"etFree_compromisedIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"greensnowIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"jobs": {
"properties": {
"duration": {
"type": "float"
},
"endTime": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"report": {
"properties": {
"ET_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"abuseFeodo_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"abuseSSL_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"abuseZeus_DOMAIN": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"abuseZeus_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"abuse_palevo_DOMAIN": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"abuse_palevo_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"binarydefense_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"ciarmylist_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"dnsbh_DOMAIN": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"dshield_high_DOMAIN": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"dshield_low_DOMAIN": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"dshield_medium_DOMAIN": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"greensnow_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"malc0de_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"openbl_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"error": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"openphish_URL": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"phishtank_URL": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"shunlist_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"error": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
},
"snort_IP": {
"properties": {
"conf": {
"properties": {
"activated": {
"type": "boolean"
},
"error": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"link": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"nbFailed": {
"type": "long"
},
"nbIndex": {
"type": "long"
},
"nbNew": {
"type": "long"
},
"nbUpdate": {
"type": "long"
}
}
}
}
}
}
},
"startTime": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"status": {
"type": "keyword"
}
}
},
"malc0deFree_BlacklistIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"malwaredomainsFree_dnsbhDOMAIN": {
"properties": {
"domain": {
"type": "keyword"
},
"extra": {
"type": "keyword"
},
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"nextvalidation": {
"type": "keyword"
},
"original_reference-why_it_was_listed": {
"type": "keyword"
},
"source": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"new": {
"properties": {
"toSearch": {
"type": "keyword"
},
"type": {
"type": "keyword"
}
}
},
"openblFree_allIP": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"ip": {
"type": "ip"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
}
}
},
"openphishFree_feedURL": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"source": {
"type": "keyword"
},
"url": {
"type": "keyword"
}
}
},
"phishtankFree_onlinevalidURL": {
"properties": {
"firstAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"idSource": {
"type": "keyword"
},
"lastAppearance": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"online": {
"index": false,
"type": "keyword"
},
"phish_detail_url": {
"index": false,
"type": "keyword"
},
"phish_id": {
"type": "keyword"
},
"source": {
"type": "keyword"
},
"submission_time": {
"format": "date_time_no_millis",
"index": false,
"type": "date"
},
"target": {
"index": false,
"type": "keyword"
},
"url": {
"type": "keyword"
},
"verification_time": {
"format": "date_time_no_millis",
"index": false,
"type": "date"
},
"verified": {
"index": false,
"type": "keyword"
}
}
},
"source": {
"properties": {
"coreIntelligence": {
"type": "keyword"
},
"description": {
"type": "text"
},
"firstQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastQuery": {
"format": "basic_date_time_no_millis",
"type": "date"
},
"lastStatus": {
"type": "keyword"
},
"link": {
"type": "keyword"
},
"score": {
"type": "integer"
},
"type": {
"type": "keyword"
}
}
}
}
}
}
Hum obviously there is data...
Could you please give the Hippocampe log when you execute curl -GET http://localhost:5000/hippocampe/api/v1.0/jobs ?
Thanks
bjuditt
commented
7 years ago 2017-01-26 14:29:29,228 :: services :: INFO :: jobs service requested
2017-01-26 14:29:29,229 :: services.modules.common.ES :: INFO :: ES.checkES launched
2017-01-26 14:29:29,234 :: services.modules.common.ES :: INFO :: ES.checkData launched
2017-01-26 14:29:29,234 :: services.modules.common.ES :: INFO :: ['jobsType']
2017-01-26 14:29:29,250 :: services.modules.common.ES :: INFO :: index hippocampe exists but type jobs does not
2017-01-26 14:29:29,250 :: services :: ERROR :: no data
There is also a 500 error
127.0.0.1 - - [26/Jan/2017 14:29:29] "GET /hippocampe/api/v1.0/jobs HTTP/1.1" 500 -
ninSmith
commented
7 years ago Could you confirm us your elasticsearch version please ?
Thanks
bjuditt
commented
7 years ago 5.0.0-alpha5
ninSmith
commented
7 years ago Not sure if that will solve the problem but could you try 5.1 please ?
Thanks
bjuditt
commented
7 years ago Do you know the commands to write ?
ninSmith
commented
7 years ago Please uninstall your current and take a look at https://www.elastic.co/downloads/elasticsearch
Thanks
bjuditt
commented
7 years ago Victory !
ninSmith
commented
7 years ago Glad you did it, thanks for your tenacity ;-)
bjuditt
commented
7 years ago Thanks to you !
norgalades
commented
7 years ago Hi, I'm having some similar problems... I'm not able to retrieve the feeds. I have installed hippocampe some days ago. I have ES version 5.6.2 I've configured to have hippocampe running as a service. I start it and everything goes ok, but when I run shadowbook and then I check the results, I get these errors:
"openphish_URL.conf": {
"activated": true,
"error": [
"TransportError(400, u'illegal_argument_exception', u'Mapper for [description] conflicts with existing mapping in other types:\\n[mapper [description] is used by multiple types. Set update_all_types to true to update [fielddata] across all types.]')"
],
"link": "https://openphish.com/feed.txt",
"nbFailed": 0,
"nbIndex": 0,
"nbNew": 0,
"nbUpdate": 0
},
"phishtank_URL.conf": {
"activated": true,
"error": [
"TransportError(400, u'illegal_argument_exception', u'Mapper for [description] conflicts with existing mapping in other types:\\n[mapper [description] is used by multiple types. Set update_all_types to true to update [fielddata] across all types.]')"
],
"link": "http://data.phishtank.com/data/online-valid.csv",
"nbFailed": 0,
"nbIndex": 0,
"nbNew": 0,
"nbUpdate": 0
},for all the sources.
hippocampe's log is saying:
2017-10-17 10:57:40,063 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main for binarydefense_IP.conf end 2017-10-17 10:57:40,063 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main launched for ET_IP.conf 2017-10-17 10:57:40,065 :: services.modules.shadowbook.objects.Source :: INFO :: Source.isActive starts 2017-10-17 10:57:40,065 :: services.modules.shadowbook.objects.Source :: INFO :: E scenario 2017-10-17 10:57:40,078 :: services.modules.shadowbook.processFeed :: ERROR :: processFeed.main failed for ET_IP.conf, no idea where it came from Traceback (most recent call last): File "./services/modules/shadowbook/processFeed.py", line 50, in main source.indexSourceInES() File "./services/modules/shadowbook/objects/Source.py", line 131, in indexSourceInES indexSource.createIndexSource() File "./services/modules/shadowbook/objects/IndexSource.py", line 80, in createIndexSource self.create() File "./services/modules/shadowbook/objects/Index.py", line 49, in create indexES.put_mapping(doc_type = self.typeNameES, body = self.docMapping) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped return func(*args, params=params, **kwargs) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/indices.py", line 282, in put_mapping '_mapping', doc_type), params=params, body=body) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 312, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request self._raise_error(response.status, raw_data) File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, u'illegal_argument_exception', u'Mapper for [description] conflicts with existing mapping in other types:\n[mapper [description] is used by multiple types. Set update_all_types to true to update [fielddata] across all types.]') 2017-10-17 10:57:40,078 :: services.modules.shadowbook.processFeed :: INFO :: processFeed.main for ET_IP.conf end 2017-10-17 10:57:40,078 :: services.shadowbook :: INFO :: shadowbook.startJob end 2017-10-17 10:57:40,078 :: services.shadowbook :: INFO :: shadowbook.updateJob launched 2017-10-17 10:57:40,078 :: services.shadowbook :: INFO :: updating with status: done 2017-10-17 10:57:40,115 :: services.shadowbook :: INFO :: shadowbook.updateJob end 2017-10-17 10:57:40,115 :: services.shadowbook :: INFO :: shadowbook.manageJob end 2017-10-17 10:57:56,241 :: services :: INFO :: jobs service requested 2017-10-17 10:57:56,243 :: services.modules.common.ES :: INFO :: ES.checkES launched 2017-10-17 10:57:56,246 :: services.modules.common.ES :: INFO :: ES.checkData launched 2017-10-17 10:57:56,246 :: services.modules.common.ES :: INFO :: ['jobsType'] 2017-10-17 10:57:56,250 :: services.modules.common.ES :: INFO :: index hippocampe and type jobs exist 2017-10-17 10:57:56,250 :: services.jobs :: INFO :: jobs.main launched 2017-10-17 10:57:56,255 :: services.jobs :: INFO :: jobs.main end
It seems that ES is complaining about some conflicting mappings... How can I "Set update_all_types to true to update [fielddata] across all types." !?
Thank you :)
Hello,
After all reinstalled correctly, the error remains. I noticed error 500 int he attached logs : .