davidhq
commented
3 years ago Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]: {
body: Gunzip {
_writeState: [Uint32Array],
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 5,
_maxListeners: undefined,
_writableState: [WritableState],
allowHalfOpen: true,
bytesWritten: 0,
_handle: [Zlib],
_outBuffer: <Buffer 00 00 00 00 00 00 00 00 e0 00 00 00 00 00 00 00 00 82 00 ff ea 7f 00 00 44 00 04 00 04 00 04 00 e0 e5 05 ff ea 7f 00 00 20 e0 05 ff ea 7f 00 00 00 00 ... 16334 more bytes>,
_outOffset: 0,
_chunkSize: 16384,
_defaultFlushFlag: 2,
_finishFlushFlag: 2,
_defaultFullFlushFlag: 3,
_info: undefined,
_maxOutputLength: 4294967296,
_level: -1,
_strategy: 0,
[Symbol(kCapture)]: false,
[Symbol(kCallback)]: null,
[Symbol(kError)]: null
},
disturbed: false,
error: null
},
[Symbol(Response internals)]: {
url: 'https://www.amazon.com/gp/product/1732265178/ref=ox_sc_act_image_1',
status: 200,
statusText: 'OK',
headers: Headers { [Symbol(map)]: [Object: null prototype] },
counter: 0
}
}
<!DOCTYPE html>
<!--[if lt IE 7]> <html lang="en-us" class="a-no-js a-lt-ie9 a-lt-ie8 a-lt-ie7"> <![endif]-->
<!--[if IE 7]> <html lang="en-us" class="a-no-js a-lt-ie9 a-lt-ie8"> <![endif]-->
<!--[if IE 8]> <html lang="en-us" class="a-no-js a-lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!-->
<html class="a-no-js" lang="en-us"><!--<![endif]--><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title dir="ltr">Amazon.com</title>
<meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="https://images-na.ssl-images-amazon.com/images/G/01/AUIClients/AmazonUI-3c913031596ca78a3768f4e934b1cc02ce238101.secure.min._V1_.css">
<script>
if (true === true) {
var ue_t0 = (+ new Date()),
ue_csm = window,
ue = { t0: ue_t0, d: function() { return (+new Date() - ue_t0); } },
ue_furl = "fls-na.amazon.com",
ue_mid = "ATVPDKIKX0DER",
ue_sid = (document.cookie.match(/session-id=([0-9-]+)/) || [])[1],
ue_sn = "opfcaptcha.amazon.com",
ue_id = 'RAQNHNVQQJA9RJJ72HGH';
}
</script>
</head>
<body>
<!--
To discuss automated access to Amazon data please contact api-services-support@amazon.com.
For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_c_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html/ref=rm_c_ac for advertising use cases.
-->
<!--
Correios.DoNotSend
-->
<div class="a-container a-padding-double-large" style="min-width:350px;padding:44px 0 !important">
<div class="a-row a-spacing-double-large" style="width: 350px; margin: 0 auto">
<div class="a-row a-spacing-medium a-text-center"><i class="a-icon a-logo"></i></div>
<div class="a-box a-alert a-alert-info a-spacing-base">
<div class="a-box-inner">
<i class="a-icon a-icon-alert"></i>
<h4>Enter the characters you see below</h4>
<p class="a-last">Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.</p>
</div>
</div>
<div class="a-section">
<div class="a-box a-color-offset-background">
<div class="a-box-inner a-padding-extra-large">
<form method="get" action="/errors/validateCaptcha" name="">
<input type=hidden name="amzn" value="4KRs5TchISf0cuUZ5onZdw==" /><input type=hidden name="amzn-r" value="/gp/product/1732265178/ref=ox_sc_act_image_1" />
<div class="a-row a-spacing-large">
<div class="a-box">
<div class="a-box-inner">
<h4>Type the characters you see in this image:</h4>
<div class="a-row a-text-center">
<img src="https://images-na.ssl-images-amazon.com/captcha/bcxmjlko/Captcha_ygofoltlqs.jpg">
</div>
<div class="a-row a-spacing-base">
<div class="a-row">
<div class="a-column a-span6">
</div>
<div class="a-column a-span6 a-span-last a-text-right">
<a onclick="window.location.reload()">Try different image</a>
</div>
</div>
<input autocomplete="off" spellcheck="false" placeholder="Type characters" id="captchacharacters" name="field-keywords" class="a-span12" autocapitalize="off" autocorrect="off" type="text">
</div>
</div>
</div>
</div>
<div class="a-section a-spacing-extra-large">
<div class="a-row">
<span class="a-button a-button-primary a-span12">
<span class="a-button-inner">
<button type="submit" class="a-button-text">Continue shopping</button>
</span>
</span>
</div>
</div>
</form>
</div>
</div>
</div>
</div>
<div class="a-divider a-divider-section"><div class="a-divider-inner"></div></div>
<div class="a-text-center a-spacing-small a-size-mini">
<a href="https://www.amazon.com/gp/help/customer/display.html/ref=footer_cou?ie=UTF8&nodeId=508088">Conditions of Use</a>
<span class="a-letter-space"></span>
<span class="a-letter-space"></span>
<span class="a-letter-space"></span>
<span class="a-letter-space"></span>
<a href="https://www.amazon.com/gp/help/customer/display.html/ref=footer_privacy?ie=UTF8&nodeId=468496">Privacy Policy</a>
</div>
<div class="a-text-center a-size-mini a-color-secondary">
© 1996-2014, Amazon.com, Inc. or its affiliates
<script>
if (true === true) {
document.write('<img src="https://fls-na.amaz'+'on.com/'+'1/oc-csi/1/OP/requestId=RAQNHNVQQJA9RJJ72HGH&js=1" />');
};
</script>
<noscript>
<img src="https://fls-na.amazon.com/1/oc-csi/1/OP/requestId=RAQNHNVQQJA9RJJ72HGH&js=0" />
</noscript>
</div>
</div>
<script>
if (true === true) {
var head = document.getElementsByTagName('head')[0],
prefix = "https://images-na.ssl-images-amazon.com/images/G/01/csminstrumentation/",
elem = document.createElement("script");
elem.src = prefix + "csm-captcha-instrumentation.min.js";
head.appendChild(elem);
elem = document.createElement("script");
elem.src = prefix + "rd-script-6d68177fa6061598e9509dc4b5bdd08d.js";
head.appendChild(elem);
}
</script>
</body></html>
{ title: 'Amazon.com', favicon: 'https://www.amazon.com/favicon.ico' }Looks like amazon is sometimes serving this (or "Something went wrong etc." instead of the real page ..
Investigating further...
Maybe I need to try with some browser agent ....
jacktuck
Hello again!
unfurl.jsreturnedat one occassion when scraping https://www.amazon.com/gp/product/1732265178/ref=ox_sc_act_image_1
now it works better and returns correct metadate
My question is: did all of that come from Amazon? I don't think that title is something that is coming from unfurl or any associated libraries... so if that's the case, nothing can be done except to retry, right?
The problem is that I cannot know which data to retry (automatically)... is there a good way to detect such cases based on something else, perhaps possible HTTP code coming from the server ?
I'd like to check for that and not save any metadata like this example above to the database.
thank you
UPDATE:
Amazon returns 503 with HTML which contains above title "Sorry! ..."