Failed to deserialize response: expected value at line 1 column 1

mpeyfuss commented 1 year ago

Component

Forge

Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?

forge 0.2.0 (8307d6d 2023-04-20T00:05:51.425008000Z)

What command(s) is the bug in?

forge verify-contract

Operating System

Linux

Describe the bug

I am running forge in an AWS EC2 instance and on verification keep getting a 403 error from Etherscan which results in the Failed to deserialize response: expected value at line 1 column 1 error as it returns html with a captcha rather than json.

I've reached out to Etherscan and this was their response:

We are aware of our CloudFlare protections being particularly sensitive to unknown automated scripts, and returning 403 responses as a result. We're looking into whitelisting this at the moment, however a quick workaround for this would be to make your request more identifiable by attaching a User Agent string

My understanding is that reqwest doesn't set a default User-Agent header and so CloudFlare decides to block my request.

Does anyone know of a way to add a User-Agent header to outgoing requests on forge?

This seems like it currently affects just a small subset of users, although it would future proof everyone if a default user agent was set by forge/foundry. For example, curl automatically applies a default user agent so it would make sense for forge/foundry to do the same.

mattsse commented 1 year ago

Does anyone know of a way to add a User-Agent header to outgoing requests on forge?

we already set this

https://github.com/foundry-rs/foundry/blob/e15e33a07c0920189fc336391f538c3dad53da73/config/src/etherscan.rs#L276-L277

we check for various cloudflare errors, but this one we're missing apparently, do you know what kind of error is returned?

mpeyfuss commented 1 year ago

ah thanks for pointing me to that. I was on an older build but just now built again from source on the EC2. about to test

Not 100% sure the kind of error returned as it doesn't seem like a regular 403 error. But here is the html body that foundry couldn't deserialize.

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/captcha/nojs/transparent.gif?ray=7c08c85288277fb7')"></div>
        <form id="challenge-form" action="/api/?__cf_chl_f_tk=Q7F3hkX.VOk8KSj37RUZQAEw34ohIhGvc9K2pEp9d2c-1682952056-0-gaNycGzNMDs" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="REAALLYY LONG VALUE OMITTED HERE">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cZone: 'api-goerli.etherscan.io',
            cType: 'interactive',
            cNounce: '11195',
            cRay: '7c08c85288277fb7',
            cHash: '98433f73c9870d8',
            cUPMDTk: "\\/api\\/?__cf_chl_tk=Q7F3hkX.VOk8KSj37RUZQAEw34ohIhGvc9K2pEp9d2c-1682952056-0-gaNycGzNMDs",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cMTimeMs: '0',
            cTplV: 5,
            cTplB: 'cf',
            cK: "",
            cRq: {
                ru: 'aHR0cHM6Ly9hcGktZ29lcmxpLmV0aGVyc2Nhbi5pby9hcGkv',
                ra: 'Tk9fVUE=',
                rm: 'UE9TVA==',
                d: 'HyOwClHdzKb0QWhXxeflQnZhLi9iZ7fHD9C+JBVClbvYfmbSMr0eTICiMY/XSloNj3BuVgnWI+iu9KVE7ISpl/t5XVS9v+jUUDQib5iMmaaTHcfWA/+cod0Jr95xHjnrv35LHfPkoZPgjKL2zuAlSahaeIRqYat5vw+1SQ9hoApJoTqZ77JRRqrrFodhQdFYRQPT2pl5PrbMb6YrksC26y6IDqUF/3pkVm+A0Qfs0u4Qaxltqm2RvbO7Ibx0MoYHxZyeZhTEa52lwtRNoDvN3vqJWYHlA3A0x8cOQ2YPE1YiuGtF/fjrqTSw9GOXasLgU0UTZXp2xY674bry0b/Cirzb3ElR612wVlOvyo7cfwsljp1lZf5I6OmhQXw/3lTG8lfcsyTYRk7Cvs0vigejlt6VsQXhYhgpXf0OwpPNIsEyjNg4X+7F6AA6zqPwfcSpHIdvmkOc6G+7rH49Ag2L0g9wWGpVGTmWvW9s+Zb/eDogqy3pJrkIVnHBi73V6r7BKMl/LgJoIqolCUlg0DoqdZRIju9ijDlPQYYaYKycLVT033HyzJdDWrJDGKwokRkGIwbZ7+P/eqDpXoUBrr3YzcKfUL5P6aMKrPg66gYrr99etaJ+xnoZ/GmS2/NjqwFZ',
                t: 'MTY4Mjk1MjA1Ni43MjkwMDA=',
                m: '43Q+P85nrusAlMAenGBEOgWlw5AZs3ny6hjIRkI0ew0=',
                i1: 'LNiT0NzLFeZ2oBCfHihVjQ==',
                i2: 'asxnmYxvJS85O7q/1ITnUQ==',
                zh: 'qb4aFuGlbJn/rUOkKXjUqElKDKE10jDqu5PE014OTwk=',
                uh: 'DV4j3Tmrbi5Rs1q3ahwVS6SgbPbI7np5884QO1u1Cgg=',
                hh: 'Ax949TKiHbaXasTISC7ryL1/i3VsF1So3LziNEbpSQM=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=7c08c85288277fb7');
        trkjs.setAttribute('alt', '');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1?ray=7c08c85288277fb7';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\\/api\\/?__cf_chl_rt_tk=Q7F3hkX.VOk8KSj37RUZQAEw34ohIhGvc9K2pEp9d2c-1682952056-0-gaNycGzNMDs" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>

</body>

</html>

mpeyfuss commented 1 year ago

Ah, unfortunately still getting the same response even with the latest build.

pythonberg1997 commented 1 year ago

Same issue

ratankaliani commented 1 year ago

Getting the same issue, any idea what it could be?

Is this a goerli specific issue?

mpeyfuss commented 1 year ago

Latest comms from the Etherscan API team made it seem like this is an issue they are aware of and their suggested User Agent fix doesn't quite work from my testing. No response back about that. Seems to also be affecting Aribtrum Goerli (which uses etherscan).

I have yet to try on mainnet fully but my initial testing went a lot better on mainnet so it seems restricted to goerli at least.

kuncle commented 1 year ago

I have been suspecting that there was an issue with my script or with my local environment until I spent nearly an hour and finally discovered this thread...

Latest comms from the Etherscan API team made it seem like this is an issue they are aware of and their suggested User Agent fix doesn't quite work from my testing. No response back about that. Seems to also be affecting Aribtrum Goerli (which uses etherscan).

I have yet to try on mainnet fully but my initial testing went a lot better on mainnet so it seems restricted to goerli at least.

I have been suspecting that there was an issue with my script or with my local environment until I spent nearly an hour and finally discovered this comment...

ratankaliani commented 1 year ago

Looks like it's broken on Gnosis as well - @mpeyfuss did you find a fix for Arbitrum Goerli?

ratankaliani commented 1 year ago

@mattsse Would it be possible to take a look at this? Contract verification via the CLI for chains outside of mainnet seems to be broken

Evalir commented 1 year ago

Hey @ratankaliani — we have a fix for contract verification on ethers-rs, we just need to upstream. It'll be done in the upcoming days

kuncle commented 1 year ago

Hey @ratankaliani — we have a fix for contract verification on ethers-rs, we just need to upstream. It'll be done in the upcoming days

See you again. lol. any update on this issue?

mpeyfuss commented 1 year ago

@Evalir can you link the upstream fix on ethers-rs so we can monitor and test?

Evalir commented 1 year ago

Hmmm these fixes might have not circumvented this particular issue @mpeyfuss, but we do have the user agent string in place. We'll look into this to ensure we're not overlooking anything, even though it's particularly hard to test if this only happens from a remote instance—i cannot reproduce locally.

Is everyone who has commented on this issue running forge on a remote instance? Or is someone getting this locally? If so, it might be a different issue.

ratankaliani commented 1 year ago

Issue has resolved for me - I was able to deploy & verify contracts on Optimism Goerli earlier today.

mpeyfuss commented 1 year ago

@Evalir yeah totally understood. It seems like the user agent string has not worked and I suspect this has to do with Cloudflare blocking the AWS IP range or something to that extent. The issue still persists with the latest build.

Definitely hard to test locally 😂

Is the foundry cloud build on ubuntu working with verification? Can't remember if contract verification to etherscan is tested in the github action off the top of my head.

Evalir commented 1 year ago

Ah I think it was tested on the live tests, but we recently removed them, so not anymore :(

ratankaliani commented 1 year ago

@mpeyfuss Ah, working with Goerli today & yeah seems like Cloudflare is still blocking it even with the latest build.

mpeyfuss commented 1 year ago

I can confirm this issue does not exist for mainnet eth, only testnets it seems. Not sure there is much we can do but we should keep this open until the Etherscan team gives an update.

pedrobergamini commented 1 year ago

Same issue here

mpeyfuss commented 1 year ago

@Evalir just wanted to bump this thread again

ShackAmeri commented 1 year ago

Same error with forge script broadcast on Sepolia

chen4903 commented 1 year ago

the same here in goerli:

   [⠰] Compiling...
   [⠔] Compiling 3 files with 0.8.20
   [⠑] Solc 0.8.20 finished in 2.13s
   Compiler run successful with warnings:
   Warning (2072): Unused local variable.
     --> script/deploy.sol:11:9:
      |
   11 |         Core c = new Core();
      |         ^^^^^^

   Deployer: 0xd3E65149C212902749D49011B6ab24bba30D97c6
   Deployed to: 0xB43573c1d6E5A6c42A4FC1240Aa6d4D7bC5120C5
   Transaction hash: 0x81d80351e5f477c84bbef6143958e4d363e406405c2bfedd817543364c68d6f3
   Starting contract verification...
   Waiting for etherscan to detect contract deployment...
   Start verifying contract `0xb43573c1d6e5a6c42a4fc1240aa6d4d7bc5120c5` deployed on goerli
   2023-10-09T07:58:57.426391Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res="              <!DOCTYPE html><html lang=\"en-US\"

mattsse commented 1 year ago

2023-10-09T07:58:57.426391Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res=" <!DOCTYPE html><html lang=\"en-US\"

looks like the etherscan API response with an error or cloudflare page...

do you have the full response?

chen4903 commented 1 year ago

This is the full response:

[⠰] Compiling...
[⠒] Compiling 3 files with 0.8.20
[⠘] Solc 0.8.20 finished in 2.28s
Compiler run successful with warnings:
Warning (2072): Unused local variable.
--> script/deploy.sol:11:9:
|
11 |         Core c = new Core();
|         ^^^^^^

Deployer: 0xd3E65149C212902749D49011B6ab24bba30D97c6
Deployed to: 0x9C05899ed01E57Dad8BEca7e3137AD04BBf5f2B2
Transaction hash: 0x5bc0d34d2b1827d52e30cc8d1663a366e6d516ceffbe6ca65f54c705ae66150f
Starting contract verification...
Waiting for etherscan to detect contract deployment...
Start verifying contract `0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2` deployed on goerli
2023-10-09T08:12:46.795512Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res="

<!DOCTYPE html>
<html lang="\&quot;en-US\&quot;">
<head>
<title>Just a moment...</title>
<meta http-equiv="\&quot;Content-Type\&quot;" content="\&quot;text/html;" charset="UTF-8\&quot;" />
<meta http-equiv="\&quot;X-UA-Compatible\&quot;" content="\&quot;IE=Edge\&quot;" />
<meta name="\&quot;robots\&quot;" content="\&quot;noindex,nofollow\&quot;" />
<meta name="\&quot;viewport\&quot;" content="\&quot;width=device-width,initial-scale=1\&quot;" />
<link href="\&quot;/cdn-cgi/styles/challenges.css\&quot;" rel="\&quot;stylesheet\&quot;" />
</head>
<body class="\&quot;no-js\&quot;">
<div class="\&quot;main-wrapper\&quot;" role="\&quot;main\&quot;">
<div class="\&quot;main-content\&quot;">
    <noscript>
    <div id="\&quot;challenge-error-title\&quot;">
    <div class="\&quot;h2\&quot;">
    <span class="\&quot;icon-wrapper\&quot;">
        <div class="\&quot;heading-icon" warning-icon\"=""></div></span>
    <span id="\&quot;challenge-error-text\&quot;">Enable JavaScript and cookies to continue</span>
    </div>
    </div>
    </noscript>
</div>
</div>
<script>(function(){window._cf_chl_opt={cvId: '2',cZone: \"api-goerli.etherscan.io\",cType: 'interactive',cNounce: '5086',cRay: '813529177d3bc09b',cHash: 'c548661e7180c05',cUPMDTk: \"\\/api\\/?apikey=CQWS36SV2WKNHU1RH7XY6QSWQK9MKJ54CY&module=contract&action=getabi&address=0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2&__cf_chl_tk=moDC8fLFgM9LR9i6AXJTnSiiNVrTWmrcW.qiQdQ05x0-1696839166-0-gaNycGzNCtA\",cFPWv: 'g',cTTimeMs: '1000',cMTimeMs: '0',cTplV: 5,cTplB: 'cf',cK: \"visitor-time\",fa: \"\\/api\\/?apikey=CQWS36SV2WKNHU1RH7XY6QSWQK9MKJ54CY&module=contract&action=getabi&address=0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2&__cf_chl_f_tk=moDC8fLFgM9LR9i6AXJTnSiiNVrTWmrcW.qiQdQ05x0-1696839166-0-gaNycGzNCtA\",md: \"bo20uCQgmDtvj7Ns.nEadYkobArkRYlH3Jof77jKKfs-1696839166-0-AUrlrDvdVzpROmpOcrQEir1Ts4Unha7FoPLyVc1n3V74R9MBSZmc3lE12vohpP2zcVU6y4Oi9Dp7DDkscd3oEoyOduA_s5zBFdfWKg2_Mk6EOzKmW2I1RRW50kbPnWoBFiJFNZaCoJh5jVI_zAjvPwbxz_TfggbKUHDYmSQ7E4ndzQvVNqYtWrnnoVDi48Q2DG9UjwHK3XOXEm8ZL0fL4BfI-9SJKCHcjXcTMgKWiHnej9VdIun73L-2rqCrdhPZMVletlWc8QP6mgfa-vz7KO9YcXzh3a-Pf8oGAHTepk8mS02Qz4hulrMsM1vpjkec3Fmt2TsUpV3oY4cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=813529177d3bc09b';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, \"\\/api\\/?apikey=CQWS36SV2WKNHU1RH7XY6QSWQK9MKJ54CY&module=contract&action=getabi&address=0x9c05899ed01e57dad8beca7e3137ad04bbf5f2b2&__cf_chl_rt_tk=moDC8fLFgM9LR9i6AXJTnSiiNVrTWmrcW.qiQdQ05x0-1696839166-0-gaNycGzNCtA\" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(cpo);}());</script>
</body>
</html>
"
Error:
expected value at line 1 column 1

ratankaliani commented 1 year ago

Running into this same issue today as well on Goerli even after running foundryup:

Failed to deserialize response: expected value at line 1 column 1 res="<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\"><meta name=\"robots\" content=\"noindex,nofollow\"><meta name=\"viewport\" content=\"width=device-width,initial-scale=1\"><link href=\"/cdn-cgi/styles/challenges.css\" rel=\"stylesheet\"></head><body class=\"no-js\"><div class=\"main-wrapper\" role=\"main\"><div class=\"main-content\"><noscript><div id=\"challenge-error-title\"><div class=\"h2\"><span class=\"icon-wrapper\"><div class=\"heading-icon warning-icon\"></div></span><span id=\"challenge-error-text\">Enable JavaScript and cookies to continue</span></div></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: '2',cZone: \"api-goerli.etherscan.io\",cType: 'interactive',cNounce: '33797',cRay: '813aa27faa900822',cHash: '49d497c7c50c4a0',cUPMDTk: \"\\/api\\/

Forge scripts on Goerli are failing too.

ratankaliani commented 1 year ago

@mattsse Is there a release of foundry with the above PR merged?

mattsse commented 1 year ago

next nightly

I also think we should consider calling a random etherscan API request first before deploying so we can check if there are issues

wdyt?

ratankaliani commented 1 year ago

Yeah I agree, would be good to add to these tests or a similar nightly suite to detect when there are Cloudflare errors.

GuiFlam commented 8 months ago

I am having a similar issue with Sepolia. Does anyone have a fix for this?

magnetto90 commented 8 months ago

I am having a similar issue with Sepolia. Does anyone have a fix for this?

If you are using Infura change to Alchemy, it solved my issue.

RodionMilman commented 5 months ago

I am having a similar issue with Sepolia. Does anyone have a fix for this?

If you are using Infura change to Alchemy, it solved my issue.

using Alchemy, have the same issue

GarethJamesLarkan commented 4 months ago

I am still having this issue on Alchemy when deploying upgradeable UUPS contracts, any ideas?

mpeyfuss commented 4 months ago

The RPC provider shouldn't affect the verification request to Etherscan, which is what this issue calls out. If you are trying to send Etherscan requests from share IP ranges, such as on AWS, their API will likely detect and try to mitigate a bot, especially on testnets. Verifying from a personal computer should pretty much always work.

hernandp commented 3 weeks ago

Having the same issue today.

difof commented 6 days ago

Same issue on Binance Smart Chain

forge verify-contract --chain $CHAIN_ID --num-of-optimizations 200 --compiler-version v0.8.23+commit.f704f362 -e $API_KEY --watch $ADDRESS $SCRIPT

Out:

2024-11-21T14:50:54.306326Z ERROR etherscan: Failed to deserialize response: expected value at line 1 column 1 res="<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"
2024-11-21T14:50:54.310954Z ERROR forge_verify::etherscan: Failed to submit verification args=<HUGE JSON OF SMART CONTRACT CODE>
Warning: Failed to deserialize content: expected value at line 1 column 1
<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>
 (4 tries remaining)

foundry-rs / foundry