search

mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources
https://www.mlsecproject.org/
GNU General Public License v3.0
650 stars 179 forks source link

Connection Abort - Fetching IOCs #163

Closed TejaRamachandran closed 7 years ago

TejaRamachandran commented 7 years ago

Hi Folks,

I am new to this full platform. I am using Combine to push data to CRITs.

I cant able to fetch the IOC IPs from the inbound and outbound URLs. I am getting the below error. I have proxy in my network. But i have given all credentials required to connect internet in /etc/apt/apt.conf. Please help me to fix this.

It will be great if you give me the trouble shooting steps, Some document where how the exact communication will happen with internet, how to resolve this, or To communicate to the internet is combine required any specific port to fetch data to default port(80 http and 443 https)

The Error is: root@prabhu:/data/combine-master# python combine.py -t crits 2016-07-26 18:42:17,765 - combine.reaper - INFO - Fetching inbound URLs 2016-07-26 18:46:32,423 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938671f50> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,423 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938686090> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39386867d0> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938686b10> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938686e50> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39386211d0> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938621510> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938621850> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938621b90> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,424 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938621ed0> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938631250> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938631590> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39386318d0> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938631c10> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938631f50> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39386402d0> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938640610> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938640950> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,425 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938640c90> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f3938640fd0> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f393864f350> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f393864f690> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f393864f9d0> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f393864fd10> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39385de090> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39385de3d0> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39385de710> failed: ConnectionError(ProtocolError('Connection a borted.', error(101, 'Network is unreachable')),) 2016-07-26 18:46:32,426 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7f39385dea50> failed: ConnectionError(ProtocolError('Connection a borted.', error(110, 'Connection timed out')),) 2016-07-26 18:46:32,426 - combine.reaper - INFO - Fetching outbound URLs 2016-07-26 18:50:47,015 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f3934127550> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,016 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f3934127650> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,016 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f3938038a10> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,016 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f393414a490> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,017 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f393414a7d0> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,017 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f393414ab10> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,017 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f393414ae50> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,017 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f39340de1d0> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,018 - combine.reaper - ERROR - Request <grequests.AsyncRequest object at 0x7f39340de510> failed: ConnectionError(ProtocolError('Connection aborted.', error(110, 'Connection timed out')),) 2016-07-26 18:50:47,018 - combine.reaper - INFO - Storing raw feeds in harvest.json 2016-07-26 18:50:47,019 - combine.thresher - INFO - Loading raw feed data from harvest.json 2016-07-26 18:50:47,019 - combine.thresher - INFO - Storing parsed data in crop.json 2016-07-26 18:50:47,020 - combine.baler - INFO - Reading processed data from crop.json 2016-07-26 18:50:47,022 - combine.baler - INFO - Output 0 indicators to CRITs using 10 threads. Operation tool 0 seconds

alexcpsec commented 7 years ago

Hi Teja. I'd have to double check, but I am almost positive that we do not explicitly check for proxies anywhere, and specially not /etc/apt/apt.conf

Having said that, we are using the requests library, so if you export your proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY, it should work seamlessly.

Please let me know if that works for you.

TejaRamachandran commented 7 years ago

Hi Alex,

Thanks for the explanation. Now i can able to fetch URL IOCs. But i am getting below error:

root@prabhu:/data/combine-master# python combine.py -t csv 2016-07-26 21:10:23,502 - combine.reaper - INFO - Fetching inbound URLs 2016-07-26 21:10:29,538 - combine.reaper - INFO - Fetching outbound URLs 2016-07-26 21:10:36,843 - combine.reaper - ERROR - Request <grequests.AsyncReque st object at 0x7fdff4cb98d0> failed: SSLError(SSLError(SSLError('bad handshake', SysCallError(-1, 'Unexpected EOF')),),) 2016-07-26 21:10:36,846 - combine.reaper - INFO - Storing raw feeds in harvest.j son 2016-07-26 21:10:36,877 - combine.thresher - INFO - Loading raw feed data from h arvest.json 2016-07-26 21:10:36,944 - combine.thresher - INFO - Evaluating http://www.projec thoneypot.org/list_of_ips.php?rss=1 2016-07-26 21:10:36,944 - combine.thresher - INFO - Parsing feed from http://www .projecthoneypot.org/list_of_ips.php?rss=1 2016-07-26 21:10:37,015 - combine.thresher - INFO - Evaluating http://www.openbl .org/lists/base_30days.txt 2016-07-26 21:10:37,015 - combine.thresher - INFO - Parsing feed from http://www .openbl.org/lists/base_30days.txt 2016-07-26 21:10:37,027 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/ssh.txt 2016-07-26 21:10:37,027 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/ssh.txt 2016-07-26 21:10:37,037 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/apache.txt 2016-07-26 21:10:37,037 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/apache.txt 2016-07-26 21:10:37,115 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/asterisk.txt 2016-07-26 21:10:37,115 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/asterisk.txt 2016-07-26 21:10:37,116 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/bots.txt 2016-07-26 21:10:37,117 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/bots.txt 2016-07-26 21:10:37,126 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/courierimap.txt 2016-07-26 21:10:37,126 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/courierimap.txt 2016-07-26 21:10:37,133 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/courierpop3.txt 2016-07-26 21:10:37,133 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/courierpop3.txt 2016-07-26 21:10:37,140 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/email.txt 2016-07-26 21:10:37,140 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/email.txt 2016-07-26 21:10:37,308 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/ftp.txt 2016-07-26 21:10:37,308 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/ftp.txt 2016-07-26 21:10:37,311 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/imap.txt 2016-07-26 21:10:37,311 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/imap.txt 2016-07-26 21:10:37,318 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/ircbot.txt 2016-07-26 21:10:37,318 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/ircbot.txt 2016-07-26 21:10:37,318 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/pop3.txt 2016-07-26 21:10:37,318 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/pop3.txt 2016-07-26 21:10:37,326 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/postfix.txt 2016-07-26 21:10:37,326 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/postfix.txt 2016-07-26 21:10:37,485 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/proftpd.txt 2016-07-26 21:10:37,485 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/proftpd.txt 2016-07-26 21:10:37,488 - combine.thresher - INFO - Evaluating http://www.blockl ist.de/lists/sip.txt 2016-07-26 21:10:37,488 - combine.thresher - INFO - Parsing feed from http://www .blocklist.de/lists/sip.txt 2016-07-26 21:10:37,489 - combine.thresher - INFO - Evaluating http://cinsscore. com/list/ci-badguys.txt 2016-07-26 21:10:37,490 - combine.thresher - INFO - Evaluating http://reputation .alienvault.com/reputation.data 2016-07-26 21:10:37,490 - combine.thresher - INFO - Parsing feed from http://rep utation.alienvault.com/reputation.data 2016-07-26 21:10:37,619 - combine.thresher - INFO - Evaluating http://dragonrese archgroup.org/insight/sshpwauth.txt 2016-07-26 21:10:37,620 - combine.thresher - INFO - Parsing feed from http://dra gonresearchgroup.org/insight/sshpwauth.txt 2016-07-26 21:10:37,622 - combine.thresher - INFO - Evaluating http://dragonrese archgroup.org/insight/vncprobe.txt 2016-07-26 21:10:37,623 - combine.thresher - INFO - Parsing feed from http://dra gonresearchgroup.org/insight/vncprobe.txt 2016-07-26 21:10:37,623 - combine.thresher - INFO - Evaluating http://danger.rul ez.sk/projects/bruteforceblocker/blist.php 2016-07-26 21:10:37,623 - combine.thresher - INFO - Parsing feed from http://dan ger.rulez.sk/projects/bruteforceblocker/blist.php 2016-07-26 21:10:37,630 - combine.thresher - INFO - Evaluating https://isc.sans. edu/ipsascii.html 2016-07-26 21:10:37,630 - combine.thresher - INFO - Parsing feed from https://is c.sans.edu/ipsascii.html 2016-07-26 21:10:37,632 - combine.thresher - INFO - Evaluating http://www.nothin k.org/blacklist/blacklist_ssh_day.txt 2016-07-26 21:10:37,632 - combine.thresher - INFO - Parsing feed from http://www .nothink.org/blacklist/blacklist_ssh_day.txt 2016-07-26 21:10:37,633 - combine.thresher - INFO - Evaluating https://www.packe tmail.net/iprep.txt 2016-07-26 21:10:37,633 - combine.thresher - INFO - Parsing feed from https://ww w.packetmail.net/iprep.txt 2016-07-26 21:10:37,643 - combine.thresher - INFO - Evaluating https://autoshun. org/files/shunlist.csv 2016-07-26 21:10:37,643 - combine.thresher - INFO - Parsing feed from https://au toshun.org/files/shunlist.csv Traceback (most recent call last): File "combine.py", line 42, in thresh('harvest.json', 'crop.json') File "/data/combine-master/thresher.py", line 191, in thresh harvest += thresher_map[site](response[2], response[0],) File "/data/combine-master/thresher.py", line 122, in process_autoshun date = line.split(',')[1].split()[0] IndexError: list index out of range

TejaRamachandran commented 7 years ago

Hi Alex,

Please help me on this.

File "combine.py", line 40, in thresh('harvest.json', 'crop.json') File "/home/teja/combine-master/thresher.py", line 191, in thresh harvest += thresher_map[site](response[2], response[0],) File "/home/teja/combine-master/thresher.py", line 122, in process_autoshun date = line.split(',')[1].split()[0] IndexError: list index out of range

alexcpsec commented 7 years ago

@TejaRamachandran Can you please try again with the new version I just merged?

TejaRamachandran commented 7 years ago

@alexcpsec Thanks, Now its fixed. May i know what changes have you made.?

alexcpsec commented 7 years ago

One of the downloads was failing because of the lack of an API key, and the #162 PR had a workaround for that

alexcpsec commented 7 years ago

Closing this