apify / got-scraping

HTTP client made for scraping based on got.
524 stars 39 forks source link

TypeError: fetch() URL is invalid in Bun runtime #145

Closed DobroslavR closed 9 hours ago

DobroslavR commented 21 hours ago

When using gotScraping with a proxyUrl in the Bun runtime, an "Invalid URL" error occurs. The script works correctly without the proxyUrl option, and the issue is specific to the Bun runtime.

Code Demonstrating the Issue

import { gotScraping } from "got-scraping";

const scrapeWebsite = async () => {
    const response = await gotScraping("https://www.example.com", {
        proxyUrl: "http://ip:port",
    });
    console.log(response.body);
};

scrapeWebsite();

Steps to Reproduce

  1. Save the above code in a file (e.g., index.ts)
  2. Ensure got-scraping is installed (bun add got-scraping)
  3. Run bun run index.ts
  4. Observe the "Invalid URL" error
  5. Remove or comment out the proxyUrl option
  6. Run the script again and observe that it works without error

Error Output

178 |         }
179 |         const { response, options } = this;
180 |         const attemptCount = this.retryCount + (error.name === 'RetryError' ? 0 : 1);
181 |         this._stopReading = true;
182 |         if (!(error instanceof RequestError)) {
183 |             error = new RequestError(error.message, error, this);
                          ^
RequestError: fetch() URL is invalid
 code: "ERR_INVALID_URL"

      at _beforeError (/Users/dobroslavradosavljevic/Desktop/Projects/bun-proxies/node_modules/got/dist/source/core/index.js:183:21)
      at /Users/dobroslavradosavljevic/Desktop/Projects/bun-proxies/node_modules/got/dist/source/core/index.js:203:21
      at processTicksAndRejections (native:1:1)
      at fetch (native:1:1)
      at _final (node:http:1024:9)
      at callFinal (node:stream:2906:23)
      at prefinish (node:stream:2929:23)
      at finishMaybe (node:stream:2936:25)
      at node:stream:2869:76
      at /Users/dobroslavradosavljevic/Desktop/Projects/bun-proxies/node_modules/got-scraping/dist/index.js:326:5

440 |         path: host,
441 |         // TODO: this property doesn't exist according to the types
442 |         pathname: host,
443 |         rejectUnauthorized: false
444 |       });
445 |       request.end();
      ^
TypeError: fetch() URL is invalid
 code: "ERR_INVALID_URL"

      at fetch (native:1:1)
      at _final (node:http:1024:9)
      at callFinal (node:stream:2906:23)
      at prefinish (node:stream:2929:23)
      at finishMaybe (node:stream:2936:25)
      at node:stream:2869:76
      at /Users/dobroslavradosavljevic/Desktop/Projects/bun-proxies/node_modules/got-scraping/dist/index.js:445:15

Environment

B4nan commented 9 hours ago

Please report that to bun, they should be node compatible, it's not about us being bun compatible.