search

SK-CERT / Taranis-NG

Taranis NG is an OSINT gathering and analysis tool for CSIRT teams and organisations. It allows team-to-team collaboration, and contains a user portal for simple self asset management. Taranis NG was developed by SK-CERT with a help from wide CSIRT community.
European Union Public License 1.2
99 stars 21 forks source link

Collector nodes' parameters are not refreshed #20

Open sebix opened 3 years ago

sebix commented 3 years ago

I think I am missing something. I suspect that the docker containers use some older code or have some caching in place that is not cleared. I noticed this as the line/change https://github.com/SK-CERT/Taranis-NG/blob/655f190efe63afef491c2f9f136c6742484b03c4/src/collectors/collectors/web_collector.py#L44 (changed in 09597f7e713ae9ccec4f32f6cc70a089a80f4c8d) doesn't seem to be effective.

First, I'm on a current version of Taranis:

$ git describe --tags 
v21.10.6-6-g843a4fe
$ git show | head -n 1
commit 843a4feadcdbf3123f626e43e2d7d69a9eabd4a3
$ grep -r 'Name of webdriver for'
src/collectors/collectors/web_collector.py:        Parameter(0, "WEBDRIVER", "Name of Webdriver", "Name of webdriver for Selenium (chrome|firefox)", ParameterType.STRING),

Let's start with building locally (I always build locally):

$ docker-compose -f docker/docker-compose.yml up --build

The build process kicks in (building all containers). Log attached, as it is long. Let's check inside the container:

$ docker exec -ti taranis-ng_collectors_1 ash
/app # grep -r 'Name of webdriver' .
./taranis-ng-collectors/collectors/web_collector.py:        Parameter(0, "WEBDRIVER", "Name of Webdriver", "Name of webdriver for Selenium (chrome|firefox)", ParameterType.STRING),

To exclude caching issues in the browser, I validated the situation with curl:

$ curl -k -H 'Authorization: Bearer ...' https://localhost:4443/api/v1/config/collectors-nodes 2>/dev/null | grep -i driver
                            "key": "WEBDRIVER",
                            "description": "Name of webdriver for selenium",
                            "name": "Name of Webdriver",

Which is before 09597f7e713ae9ccec4f32f6cc70a089a80f4c8d.

What the heck? Some caching the the core?

I also deleted the images related to the collectors (found by docker images -a | grep collect, I had 4 of them, required deletion of the container as well). But still, the result of the request is the same old/unexpected one.

Any ideas are appreciated.

milankowww commented 3 years ago

The configuration entries are only processed when you are registering the collector node. The fields are stored in the database at core. This is something we need to address in the future, don't yet know how.

sebix commented 3 years ago

Okay. If new parameters are introduced, they are not visible in the UI either, aren't they? Because that would explain another error that I'm seeing.

I tried python manage.py collector --update, but that didn't change anything.

dodancs commented 3 years ago

Hi Sebastian. I was waiting when you would find this! 😃

Yes. The management script does not refresh this. I did not want to do it, because it could disturb any existing OSINT sources you configured. Therefore this needs further investigation and implementation of a correct way of refreshing collector parameters.

sebix commented 3 years ago

Hi Sebastian. I was waiting when you would find this! smiley

Do you have a bug bounty? :D

Yes. The management script does not refresh this. I did not want to do it, because it could disturb any existing OSINT sources you configured. Therefore this needs further investigation and implementation of a correct way of refreshing collector parameters.

That's definitely an issue. For example, I'm unable to run a web collector, as I'm missing a parameter (LOAD_MORE_BUTTON_SELECTOR) in the UI causing an exception:

collectors_1  | [2021-11-15 15:18:37 +0000] [1] [DEBUG] [None] 1 data loaded                                                                                                                                                                 
collectors_1  | [2021-11-15 15:18:37 +0000] [1] [INFO] [None] COLLECTOR web/9176258a-bde6-4db8-9d87-1ab47daefa2c: Starting collector                                                                                                         
collectors_1  | [2021-11-15 15:18:37 +0000] [1] [DEBUG] [None] Traceback (most recent call last):                                                                                                                                            
collectors_1  |   File "/app/taranis-ng-collectors/collectors/base_collector.py", line 252, in refresh                                                                                                                                       
collectors_1  |     self.collect(source)                                                                                                                                                                                                     
collectors_1  |   File "/app/taranis-ng-collectors/collectors/web_collector.py", line 411, in collect                                                                                                                                        
collectors_1  |     self.__parse_settings()                                                                                                                                                                                                  
collectors_1  |   File "/app/taranis-ng-collectors/collectors/web_collector.py", line 282, in __parse_settings                                                                                                                               
collectors_1  |     self.selectors['load_more'] = self.source.parameter_values['LOAD_MORE_BUTTON_SELECTOR']                                                                                                                                  
collectors_1  | KeyError: 'LOAD_MORE_BUTTON_SELECTOR'
milankowww commented 3 years ago

Do you have a bug bounty? :D

Sure, you have a beer voucher with us when our countries properly open up again.

dodancs commented 3 years ago

It's marked on our schedule to improve this feature.