YETI analyzer does not work

[hasamba]

when trying to analyze timeline through YETI, im getting an error, it seems to me that the problem is with timesketch but if not i will forward it to YETI

Timesketch version: 20230913

To Reproduce Steps to reproduce the behavior:

1. analyze timeline with YETI

error in timesketch: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/analyzers/interface.py", line 1153, in run_wrapper result = self.run() File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/analyzers/yetiindicators.py", line 107, in run self.get_indicators("x-regex") File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/analyzers/yetiindicators.py", line 70, in get_indicators + response.json() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

error in yeti: yeti-yeti-1 | INFO:_internal:x.x.x.x - - [22/Oct/2023 07:26:20] "POST /api/indicators/filter/ HTTP/1.1" 405 - yeti-yeti-1 | ERROR:_internal:x.x.x.x - - [22/Oct/2023 07:26:20] code 400, message Bad request version ('"x-regex"}') yeti-yeti-1 | INFO:_internal:x.x.x.x - - [22/Oct/2023 07:26:20] "None /api/indicators/filter/ HTTP/0.9" HTTPStatus.BAD_REQUEST -

[jkppr]

@tomchop can you please take a look?

[tomchop]

@hasamba It looks like you're using the version of the Yeti analyzer that is bundled in the latest Timesketch release (20230913) which is not compatible with the latest version (fastapi branch of Yeti)

What version of Yeti are you using?

[jkppr]

Note: We just released 20231025 including the latest Yeti analyzer changes. Please note that an update to this version requires a database upgrade!

[hasamba]

Note: We just released 20231025 including the latest Yeti analyzer changes. Please note that an update to this version requires a database upgrade!

i just updated timesketch to 20231025 (btw the upgrade process does not work and the i couldn't open any previous timeline, had to drop-db but that's another problem :) and i get this error:

timesketch: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/analyzers/interface.py", line 1153, in run_wrapper result = self.run() File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/analyzers/yetiindicators.py", line 131, in run indicators = self.get_indicators("regex") File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/analyzers/yetiindicators.py", line 86, in get_indicators + response.json() File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

yeti: [26/Oct/2023 07:25:03] "POST /api/auth/login HTTP/1.1" 200 - [26/Oct/2023 07:25:04] "GET /api/users HTTP/1.1" 308 - [26/Oct/2023 07:25:04] "GET /api/users/ HTTP/1.1" 200 - [26/Oct/2023 07:25:04] "GET /api/users/ HTTP/1.1" 200 - [26/Oct/2023 07:25:04] "GET /favicon.ico HTTP/1.1" 304 - [26/Oct/2023 07:25:08] "GET / HTTP/1.1" 200 - [26/Oct/2023 07:25:08] "GET /css/app.c4208acc.css HTTP/1.1" 304 - [26/Oct/2023 07:25:08] "GET /js/app.e052d923.js HTTP/1.1" 304 - [26/Oct/2023 07:25:08] "GET /js/chunk-vendors.c4703d93.js HTTP/1.1" 304 - [26/Oct/2023 07:25:09] "GET /favicon.ico HTTP/1.1" 304 - [26/Oct/2023 07:26:33] "POST /api/indicators/search HTTP/1.1" 405 - [26/Oct/2023 07:26:33] code 400, message Bad request version ('"regex"}') [26/Oct/2023 07:26:33] "None /api/indicators/search HTTP/0.9" HTTPStatus.BAD_REQUEST -

Timesketch version: 20231025 i cant find YETI version, i installed it 3 days ago with latest docker (if it helps)

[tomchop]

Did you also relaunch the celery tasks in Timesketch? I see that the requests that your Yeti server is receiving are still coming to /api/indicators, which is the old API. There code in yetiindicators.py is settnig the API root to /api/v2

As for the Yeti version, you got it from https://hub.docker.com/r/yetiplatform/yeti, right?

[hasamba]

Did you also relaunch the celery tasks in Timesketch? I see that the requests that your Yeti server is receiving are still coming to /api/indicators, which is the old API. There code in yetiindicators.py is settnig the API root to /api/v2

yes, docker compose down && up

As for the Yeti version, you got it from https://hub.docker.com/r/yetiplatform/yeti, right?

i follow their instructions https://github.com/yeti-platform/yeti/tree/master/extras/docker

[tomchop]

I'm not sure about the Timesketch upgrade, but doing docker compose down and up sounds like it might just restart the old containers, you might have to docker compose pull && docker compose create --force-recreate before running up again.

i follow their instructions https://github.com/yeti-platform/yeti/tree/master/extras/docker

Ah, I see. Those instructions for Yeti are outdated, you might want to follow the ones here: https://github.com/yeti-platform/yeti-docker/blob/main/prod/README.md. The containers you're pulling are still the same though.

[hasamba]

of course i pulled the new container before the restart, i ran it again now just to be sure, still same error...

[tomchop]

Yeah but from my tests, even when you pull the new image, you have to recreate the container from that new image (hence the --force-recreate), otherwise it just uses the old container (even if you pulled the new image).

Could you paste us the output of docker images and docker ps -a please?

[hasamba]

when i said i ran it again, i meant that i ran the exact command you sent, with the --force-recreate

docker images: REPOSITORY TAG IMAGE ID CREATED SIZE us-docker.pkg.dev/osdfir-registry/timesketch/timesketch latest d4cfe4b9710b 21 hours ago 950MB us-docker.pkg.dev/osdfir-registry/timesketch/timesketch <none> 991928243d69 5 weeks ago 779MB log2timeline/plaso latest d00889bbcf2a 3 months ago 414MB opensearchproject/opensearch 2.9.0 f12cdcea030c 3 months ago 1.22GB hello-world latest 9c7a54a9a43c 5 months ago 13.3kB portainer/agent 2.16.2 dd7ce5896e9c 11 months ago 166MB redis 6.0.8-alpine 8835e6aeca99 2 years ago 31.1MB postgres 13.0-alpine 700e581c202e 3 years ago 159MB nginx 1.19.3-alpine 4efb29ff172a 3 years ago 21.8MB

docker ps -a: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2017ad5ac7b7 nginx:1.19.3-alpine "/docker-entrypoint.…" 3 hours ago Up 3 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp nginx 434b18319195 us-docker.pkg.dev/osdfir-registry/timesketch/timesketch:latest "/docker-entrypoint.…" 3 hours ago Up 3 hours timesketch-web f36713e74847 redis:6.0.8-alpine "docker-entrypoint.s…" 3 hours ago Up 3 hours 6379/tcp redis 8d1f0569523d opensearchproject/opensearch:2.9.0 "./opensearch-docker…" 3 hours ago Up 3 hours 9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp opensearch a78352194a6f postgres:13.0-alpine "docker-entrypoint.s…" 3 hours ago Up 3 hours 5432/tcp postgres 8e19ff12819e us-docker.pkg.dev/osdfir-registry/timesketch/timesketch:latest "/docker-entrypoint.…" 3 hours ago Up 3 hours timesketch-worker e85ac88b253d us-docker.pkg.dev/osdfir-registry/timesketch/timesketch:latest "/docker-entrypoint.…" 3 hours ago Up 3 hours timesketch-web-legacy 1102de54055b portainer/agent:2.16.2 "./agent" 8 hours ago Up 8 hours 0.0.0.0:9001->9001/tcp, :::9001->9001/tcp portainer_agent 1b765cd452e5 log2timeline/plaso "/usr/local/bin/plas…" 6 days ago Exited (0) 6 days ago vibrant_swirles ea0fab2fe744 log2timeline/plaso "/usr/local/bin/plas…" 6 days ago Exited (0) 6 days ago tender_murdock 7514765b766f log2timeline/plaso "/usr/local/bin/plas…" 6 days ago Exited (1) 6 days ago strange_spence 166698ec078d log2timeline/plaso "/usr/local/bin/plas…" 6 days ago Exited (1) 6 days ago practical_meitner 1b6800b689da log2timeline/plaso "/usr/local/bin/plas…" 6 days ago Exited (1) 6 days ago silly_kalam 1d3bf0f93bf4 log2timeline/plaso "/usr/local/bin/plas…" 6 days ago Exited (0) 6 days ago vibrant_babbage e90109609851 log2timeline/plaso "/usr/local/bin/plas…" 8 days ago Exited (0) 8 days ago elegant_shaw 55aacdac9ef7 log2timeline/plaso "/usr/local/bin/plas…" 4 weeks ago Exited (0) 4 weeks ago tender_moore 13cb504d5121 log2timeline/plaso "/usr/local/bin/plas…" 4 weeks ago Exited (0) 4 weeks ago objective_dhawan e3e0971363a9 log2timeline/plaso "/usr/local/bin/plas…" 4 weeks ago Exited (0) 4 weeks ago zen_chebyshev 54574535a438 log2timeline/plaso "/usr/local/bin/plas…" 4 weeks ago Exited (0) 4 weeks ago gifted_einstein eec58e215e77 hello-world "/hello" 4 weeks ago Exited (0) 4 weeks ago hardcore_spence

[tomchop]

OK, I think I know what's going on here.

1. Make sure you're running the new version of yeti, out of https://github.com/yeti-platform/yeti-docker/blob/main/dev/ (or prod if you prefer). The image in the dockerhub was updated, but the compose files in that yeti-docker repo are the ones you need to get everything up and running.

2. Your timesketch instance config probably needs to be tweaked:

YETI_API_ROOT = 'http://yeti-frontend/api/v2'

Then restart the Timesketch Celery workers, and you should be good to go.

[hasamba]

it's working! thanks alot

the only change is that the docker compose in the prod is pointing the API to port 8000 so i changed your suggestion to : YETI_API_ROOT = 'http://yeti-frontend:8000/api/v2'

thanks again!

[tomchop]

Nice, I'll adjust the Timesketch example in the config to reflect this. Just FYI, this should also work on port 80 if you have nginx acting as a reverse proxy just like the docker compose has it setup. Up to you which one to use!

[jkppr]

i just updated timesketch to 20231025 (btw the upgrade process does not work and the i couldn't open any previous timeline, had to drop-db but that's another problem :)

Now that the yeti issue is luckily solved: Did you upgrade your database after updating to the 20231025 release? (https://timesketch.org/guides/admin/upgrade/#upgrade-the-database-schema)

[hasamba]

i tried to upgrade the db before upgrading timesketch, as the guide say, "db current" didn't gave result so i used "db history" and "db stamp " (i replaced the revision with what i had) ran "db upgrade" and as far i remember it gave me the same result as "db current" did.

than i upgraded timesketch and each sketch i tried to enter gave an error... something with administrator.... so i dropped the db and created a new user and now it's seems fine i had only test sketches so i don't really care, next time i wont be able to upgrade ill open a new bug thanks for all the help !