[Bug] TypeError: utils.forOwn is not a function

[LeiG]

Hi, I'm running into this strange bug when using puppeteer-extra, puppeteer-extra-plugin-stealth with Typescript and the Serverless Framework. I will describe my setup and bug encountered below. I've tried a few things such as changing from puppeteer-core to puppeteer, but didn't have any luck. I'm not sure if I missed anything obvious. Thanks for the help in advance!

The full error message is below. The error throws when calling puppeteerExtra.use(StealthPlugin());. I tried other plugins such as puppeteer-extra-plugin-adblocker and the error persists so I don't think it's the issue with a specific plugin.

✖ utils.forOwn is not a function
✖ TypeError: utils.forOwn is not a function
      at cloneObjectDeep (/Users/leigong/git/friday/lambda/node_modules/merge-deep/node_modules/clone-deep/index.js:27:11)
      at cloneDeep (/Users/leigong/git/friday/lambda/node_modules/merge-deep/node_modules/clone-deep/index.js:16:14)
      at mergeDeep (/Users/leigong/git/friday/lambda/node_modules/merge-deep/index.js:19:16)
      at new PuppeteerExtraPlugin (/Users/leigong/git/friday/lambda/node_modules/puppeteer-extra-plugin/src/index.ts:77:18)
      at new StealthPlugin2 (/Users/leigong/git/friday/lambda/node_modules/puppeteer-extra-plugin-stealth/index.js:74:5)
      at defaultExport (/Users/leigong/git/friday/lambda/node_modules/puppeteer-extra-plugin-stealth/index.js:172:31)
      at Object.<anonymous> (/Users/leigong/git/friday/lambda/src/functions/hermes/handler.ts:13:20)
      at Module._compile (node:internal/modules/cjs/loader:1105:14)
      at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
      at Object.require.extensions.<computed> [as .js] (/Users/leigong/git/friday/lambda/node_modules/ts-node/src/index.ts:1587:43)
      at Module.load (node:internal/modules/cjs/loader:981:32)
      at Function.Module._load (node:internal/modules/cjs/loader:822:12)
      at Module.require (node:internal/modules/cjs/loader:1005:19)
      at require (node:internal/modules/cjs/helpers:102:18)
      at /Users/leigong/git/friday/lambda/node_modules/serverless-offline/dist/lambda/handler-runner/in-process-runner/InProcessRunner.js:166:133
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async InProcessRunner.run (/Users/leigong/git/friday/lambda/node_modules/serverless-offline/dist/lambda/handler-runner/in-process-runner/InProcessRunner.js:166:9)

Here is a simplified version of the code snippet in .ts

import { Browser } from 'puppeteer';
import { addExtra } from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
import chromium from 'chrome-aws-lambda';

const puppeteerExtra = addExtra(chromium.puppeteer as any);
puppeteerExtra.use(StealthPlugin());

browser = await puppeteerExtra.launch({
            args: chromium.args,
            defaultViewport: chromium.defaultViewport,
            executablePath: await chromium.executablePath,
            headless: chromium.headless,
            ignoreHTTPSErrors: true,
        })

        const page = await browser.newPage();
        // other scraping logics

The relevant version of the libraries are below

"puppeteer": "^10.1.0",
"puppeteer-extra": "^3.3.4",
"puppeteer-extra-plugin-adblocker": "^2.13.4",
"puppeteer-extra-plugin-stealth": "^2.11.0",

[zkschmitz]

+1 (same problem)

[LeiG]

Not sure if this is the root cause - the "clone-deep" package in one of the dependencies is at a pretty old version 0.2.4 and the error seems to be pointing to this line here https://github.com/jonschlinkert/clone-deep/blob/8ab50fb93db4eb9288a6a9b6042dbe3e9a1ac7fb/utils.js#L14, which looks like should be require('for-own', 'forOwn');?

As a js n00b, I'm not sure how to fix or if it's something Typescript related.

[cdotte]

(same problem) any update?

[Randdalf]

I worked around this by overriding the version of "clone-deep" in my package.json

  "overrides": {
    "clone-deep": "^4.0.1"
  }

[ledesmablt]

^^ The above fix works for me, though had to delete node_modules & package-lock.json

[gwak2837]

I worked around this by overriding the version of "clone-deep" in my package.json when using yarn berry

{
  "resolutions": {
    "clone-deep": "^4.0.1"
  }
}

[dbjpanda]

+1 Need a fix here

[AdrKacz]

Adding "clone-deep": "^4.0.1" to my resolution didn't fix the issue on my side, anyone to help?

I am using yarn with workspaces, I added the resolutions, deleted the node_modules and yarn.lock, but I still have the error TypeError: utils.forOwn is not a function

[jeanhdev]

So I fixed this issue - what you need to do is force the resolutions of the clone-deep and merge-deep latest versions cc @AdrKacz like below in your package.json

"resolutions": { "merge-deep": "^3.0.3", "clone-deep": "^4.0.1" }

However, I'm still getting the error below; A plugin listed 'puppeteer-extra-plugin-stealth/evasions/chrome.app' as dependency, which is currently missing.

If you have any idea on how to solve my issue, I'd gladly take your recommendation. Cheers

[AdrKacz]

Thank you for your help @jeanhdev, unfortunately using "resolutions": { "merge-deep": "^3.0.3", "clone-deep": "^4.0.1" } I still have the exact same error.

Probably has an impact, I have to use puppeteer-core version 19.4.0 (so not the latest).

Here is what in my yarn.lock

clone-deep@^0.2.4, clone-deep@^4.0.1:
  version "4.0.1"
  resolved "https://registry.yarnpkg.com/clone-deep/-/clone-deep-4.0.1.tgz#c19fd9bdbbf85942b4fd979c84dcf7d5f07c2387"
  integrity sha512-neHB9xuzh/wk0dIHweyAXv2aPGZIVk3pLMe+/RNzINf17fe0OG96QroktYAUm7SM1PBnzTabaLboqqxDyMU+SQ==
  dependencies:
    is-plain-object "^2.0.4"
    kind-of "^6.0.2"
    shallow-clone "^3.0.0"

...

merge-deep@^3.0.1, merge-deep@^3.0.3:
  version "3.0.3"
  resolved "https://registry.yarnpkg.com/merge-deep/-/merge-deep-3.0.3.tgz#1a2b2ae926da8b2ae93a0ac15d90cd1922766003"
  integrity sha512-qtmzAS6t6grwEkNrunqTBdn0qKwFgNWvlxUbAV8es9M7Ot1EbyApytCnvE0jALPa46ZpKDUo527kKiaWplmlFA==
  dependencies:
    arr-union "^3.1.0"
    clone-deep "^0.2.4"
    kind-of "^3.0.2"

[sh1kxrv]

just add to your package.json in workspace repository

  "dependencies": {,
      "puppeteer": "^10.1.0",
      "puppeteer-extra": "^3.3.4",
      "puppeteer-extra-plugin-adblocker": "^2.13.4",
      "puppeteer-extra-plugin-stealth": "^2.11.0",
  },

[shakkky]

hey @AdrKacz, did it manage to figure it out? I'm running into the same issue. Also using puppeteer-core

[AdrKacz]

No, I had to build my solution without puppeteer-extra unfortunately 😞

[Juaoie]

如果你使用的是yarn，你在yarn.lock中就可以找到，是哪个包依赖了clone-deep，我这边显示的是merge-deep，你也可以通过查看源码看到依赖关系，这里依赖的clone-deep是0.2.4版本，你要修复这个bug，就只用把clone-deep升级就可以了，而不是安装两个clone-deep版本

能理解我意思吗？

最后分析一下产生这个问题的原因，我自己尝试使用官方的demo去执行，发现不会这个bug，但是只要上ts就会有问题，很明显是使用node执行的js代码有问题，也就是ts打包以后有问题，我这里项目使用esbuild打包的，和tsc、webpack打包同理，也可以直接在源码中找到其中clone-deep@0.2.4引用forOwn有问题，这里commonjs 导入编译成module导入是有问题的，所以会报utils.forOwn is not a function，但是查看clone-deep@4.0.1源码就会发现它不再依赖forOwn包了，所以这里不会报错

另外，如果你用的是pnpm运行的项目，那这里就有个大坑了，因为现在很多项目对依赖管理使用的是npm或者yarn，如果对人为对依赖管理不是很清晰，就会导致很多幽灵依赖的问题出现，如果你使用pnpm安装这个工具，你就会发现需要额外安装kind-of和is-plain-object，明明你没有主动依赖这两个插件，也需要你主动去安装这个插件

最后贴两张clone-deep@0.2.4和clone-deep@4.0.1源码图片

[Juaoie]

clone-deep包是来自puppeteer-extra-plugin-stealth的依赖

[Juaoie]

在clone-deep@0.2.4中使用了lazy-cache，但是lazy-cache是专门为 CommonJS 模块系统设计的 npm 插件，所以在es模块中无法直接使用

[Juaoie]

pnpm解决方案：在package.json文件中加入下面代码

npm解决方案：在package.json文件中加入下面代码

yarn解决方案也是类似，覆盖clone-deep为最新版本就可以了

[TobiasPankner]

I fixed this by adding below to package.json

"overrides": {
    "clone-deep": "^4.0.1",
    "merge-deep": {
      "clone-deep": "^4.0.1"
    }
  }