search

projectdiscovery / subfinder

Fast passive subdomain enumeration tool.
https://projectdiscovery.io
MIT License
9.93k stars 1.24k forks source link

[Issue] Pagination Issue with SecurityTrails API Leading to Incomplete Subdomain Retrieval #1235

Closed Shark1337 closed 4 months ago

Shark1337 commented 4 months ago

Hello Subfinder Team,

I've encountered an issue with the SecurityTrails API integration where it appears that pagination is not handled correctly, resulting in not all subdomains being retrieved. Below are the details of the problem:

Description: When making requests to the SecurityTrails API through Subfinder, it seems that the pagination mechanism isn't functioning as expected. This leads to incomplete data retrieval, specifically, subdomains are missing from the results.

Steps to Reproduce:

Execute Subfinder with SecurityTrails as the source. Observe the logs and results. Expected Behavior: All subdomains available via SecurityTrails should be retrieved and listed.

Actual Behavior: Only a subset of subdomains is retrieved. The following error message is observed in the logs when attempting to paginate:

Response for failed request against https://api.securitytrails.com/v1/scroll/edb379a8a032c11088a57fdf8c0f2e67:
{
  "endpoint": "/v1/scroll/edb379a8a032c11088a57fdf8c0f2e67",
  "message": "No data found for this request"
}
[WRN] Could not run source securitytrails: unexpected status code 404 received from https://api.securitytrails.com/v1/scroll/edb379a8a032c11088a57fdf8c0f2e67

Additional Information:

Subfinder version: v2.6.6 (latest)

I hope this information helps in diagnosing and resolving the issue. Thank you for looking into this.

Screenshots

image
GeorginaReeder commented 4 months ago

Thanks for opening this issue @Shark1337 , we'll take a look into it!

We also have a Discord server, which you’re more than welcome to join. It's a great place to connect with fellow contributors and stay updated with the latest developments!

DhiyaneshGeek commented 4 months ago

Hi @Shark1337 , i checked at my end with the following command

subfinder -d deliveroo.fr -s securitytrails -v -proxy=http://127.0.0.1:8080/

Got Around 30 domains, the total number of domains are 30.

404 verbose error is shown since there is no page to extract. Hence there is nothing to do with pagination

image

dogancanbakir commented 4 months ago

Yes, the error you are seeing is related to the API. This means that the API returns scrolling/pagination information (scrolling/pagination ID), but when we make the request, it returns a 404 error. However, you don't need to worry about this specific error as it can be ignored. I will now close this issue. Please reopen it if you have any other questions or concerns.