mendableai / firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
https://firecrawl.dev
GNU Affero General Public License v3.0
19.26k stars 1.5k forks source link

[Bug] headers don't pass to the api in crawl url #909

Open TChengZ opened 1 week ago

TChengZ commented 1 week ago

Describe the Bug Provide a clear and concise description of what the bug is. i crawl the url 'https://www.xchuxing.com/car?bid=203', in this page, it will send a request like image to get some information to show on the web,but the crawl result don't have the above message, i figure out there must have some cookie to simulate the log in state,and i use postman to check that once i add the cookie below for the request(/all-brand?bid=203), it can return the information image

and i past the cookie data to the pageOptions,but still can't crawl the full message from the website To Reproduce Steps to reproduce the issue:

  1. Configure the environment or settings with '...'
  2. Run the command '...'
  3. Observe the error or unexpected output at '...'
  4. Log output/error message

Expected Behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots or copies of the command line output to help explain the issue.

Environment (please complete the following information):

Logs If applicable, include detailed logs to help understand the problem.

Additional Context Add any other context about the problem here, such as configuration specifics, network conditions, data volumes, etc.