CSE Search Entity Bad Request

[derekburgess]

The CSE Search list is very long, making it hard for me to tell which ones might work or I could be selecting known broken searches... Either way, I have tested this with 10 randomly selected searches from the list, using the same query and page request. All result in a frontend error, "There was an error fetching CSE results. Please try again."

• GEOSINTsearch (Selected because it was the default choice and I assumed it worked).
• Social Search (Sounded generic enough)
• TikTok Search Engine
• Linkedin People Search
• Linkedin
• Image Search
• Github Search +
• Twitter Historical Search
• Twitter CSE
• 300+ Social Networking Sites

This list is not a request to fix those searches, I selected them at random.

In addition to that error, the console shows:

osintbuddy-backend-1       | plugin instance:  <app.plugins.cse.CSESearchPlugin object at 0x7f82da07ee10>
osintbuddy-backend-1       | resp:  <Response [200 OK]>
osintbuddy-backend-1       | INFO:     172.18.0.1:38180 - "GET /api/v1/nodes/transforms?node_type=CSE%20Search HTTP/1.1" 200 OK
osintbuddy-microservice-1  | Crawling results for:  001794496531944888666:iyxger-cwug{"time":"2023-06-13T12:04:23.529339569Z","id":"","remote_ip":"172.18.0.2","host":"microservice:1323","method":"GET","uri":"/google-cse?query=Elon%20Musk&pages=1&id=001794496531944888666%3Aiyxger-cwug","user_agent":"python-httpx/0.24.0","status":200,"error":"","latency":3000179116,"latency_human":"3.000179116s","bytes_in":0,"bytes_out":5}
osintbuddy-microservice-1  | 2023/06/13 12:04:23 Something went wrong: Bad Request

It appears the initial request/response is successful, but microservices fails to handle the data? or the data is returned malformed?

[jerlendds]

Hey thanks for reporting this, looking into this it seems like I forgot to encode the query values before making the request in the microservice. I've implemented a fix and I'll be pushing that out soon.

I tested with the same values with Elon Musk as the query and I received results for the following:

• [x] GEOSINTsearch
• [x] Social Search
• [x] TikTok Search Engine
• [x] Linkedin People Search
• [x] Linkedin
• [x] Image Search
• [x] Github Search +
• [x] Twitter Historical Search
• [x] Twitter CSE
• [x] 300+ Social Networking Sites

[derekburgess]

Thank you! I just started learning Go, so I'm curious to see your fix.

[jerlendds]

No problem! I've just pushed the change, you can check it out here if you'd like.

commit ce8e97a

edit: Btw, you'll have to rebuild the microservice container for this update to take effect:

cd osintbuddy
docker compose build microservice
docker compose up