Closed dicortazar closed 9 years ago
jgbarah
commented
9 years ago 410 is page gone, should be dealt with as 404 is, for example.
However, I cannot reproduce: which one is the offending url? I'm trying https://bugs.launchpad.net/nova/+bug/1323541 but it works fine...
iganchev
commented
9 years ago Hello,
I encountered the same issue with the launchpad backend. This can be reproduced with the terminator repository on launchpad.
bicho --db-user-out=user --db-password-out=password --db-database-out=database -d 10 -n 10 -b lp -u https://bugs.launchpad.net/terminator
After receiving 232 issues the program hangs with the following error message :
Checking URL: https://bugs.launchpad.net
Running Bicho with delay of 10 seconds
/usr/lib64/python2.7/site-packages/storm/database.py:454: Warning: Data truncated for column 'linked_branches' at row 1
return function(*args, **kwargs)
Error in function analyzeBug with URL: ' 'https://bugs.launchpad.net/terminator and Bug: https://api.launchpad.net/1.0/terminator/+bug/516724
Traceback (most recent call last):
File "/bin/bicho", line 25, in <module>
retval = bicho.main.main()
File "/usr/lib/python2.7/site-packages/bicho/main.py", line 56, in main
backend.run()
File "/usr/lib/python2.7/site-packages/bicho/backends/lp.py", line 1022, in run
issue_data = self.analyze_bug(bug)
File "/usr/lib/python2.7/site-packages/bicho/backends/lp.py", line 887, in analyze_bug
by = self.__get_people_from_uri(entry['person_link'])
File "/usr/lib/python2.7/site-packages/bicho/backends/lp.py", line 934, in __get_people_from_uri
people_lp = self.lp.people[self._get_nickname_from_uri(uri)]
File "/usr/lib/python2.7/site-packages/lazr/restfulclient/resource.py", line 987, in __getitem__
shim_resource._ensure_representation()
File "/usr/lib/python2.7/site-packages/lazr/restfulclient/resource.py", line 381, in _ensure_representation
representation = self._root._browser.get(self._wadl_resource)
File "/usr/lib/python2.7/site-packages/lazr/restfulclient/_browser.py", line 434, in get
response, content = self._request(url, extra_headers=headers)
File "/usr/lib/python2.7/site-packages/lazr/restfulclient/_browser.py", line 424, in _request
raise error
lazr.restfulclient.errors.ClientError: HTTP Error 410: Gone
Response headers:
---
-content-encoding: gzip
content-length: 9645
content-type: text/html;charset=utf-8
date: Fri, 17 Apr 2015 13:05:34 GMT
server: zope.server.http (HTTP)
status: 410
vary: Accept-Encoding
x-pad: avoid browser bug
x-powered-by: Zope (www.zope.org), Python (www.python.org)
---
Response body:
---
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
lang="en" dir="ltr">
<head>
<meta charset="UTF-8" />
<title>Error: Page gone</title>
<link rel="shortcut icon" href="/@@/launchpad.png" />
<link type="text/css" rel="stylesheet"
media="screen, print"
href="/+icing/rev17430/combo.css" />
<script type="text/javascript">
var LP = {
cache: {},
links: {}
};
</script>
<script type="text/javascript">var cookie_scope = '; Path=/; Secure; Domain=.launchpad.net';</script>
<script type="text/javascript"
src="/+combo/rev17430/?yui/yui/yui-min.js&lp/meta.js&yui/loader/loader-min.js"></script>
<script type="text/javascript">
var raw = null;
if (LP.devmode) {
raw = 'raw';
}
YUI.GlobalConfig = {
combine: true,
comboBase: '/+combo/rev17430/?',
root: 'yui/',
filter: raw,
debug: false,
fetchCSS: false,
maxURLLength: 2000,
groups: {
lp: {
combine: true,
base: '/+combo/rev17430/?lp/',
comboBase: '/+combo/rev17430/?',
root: 'lp/',
// comes from including lp/meta.js
modules: LP_MODULES,
fetchCSS: false
}
}
}</script>
<script type="text/javascript">
// we need this to create a single YUI instance all events and code
// talks across. All instances of YUI().use should be based off of
// LPJS instead.
LPJS = new YUI();
</script>
<script id="base-layout-load-scripts" type="text/javascript">
//<![CDATA[
LPJS.use('base', 'node', 'console', 'event',
'oop', 'lp', 'lp.app.foldables','lp.app.sorttable',
'lp.app.inlinehelp', 'lp.app.links', 'lp.app.longpoll',
'lp.bugs.bugtask_index', 'lp.bugs.subscribers',
'lp.app.ellipsis', 'lp.code.branchmergeproposal.diff',
'lp.views.global',
function(Y) {
Y.on("domready", function () {
var global_view = new Y.lp.views.Global();
global_view.render();
Y.lp.app.sorttable.SortTable.init();
Y.lp.app.inlinehelp.init_help();
Y.lp.activate_collapsibles();
Y.lp.app.foldables.activate();
Y.lp.app.links.check_valid_lp_links();
// Longpolling will only start if
// LP.cache.longpoll is populated.
// We use Y.later to work around a Safari/Chrome 'feature':
// The mouse cursor stays 'busy' until all the requests started during
// page load are finished. Hence we want the long poll request to start
// right *after* the page has loaded.
Y.later(0, Y.lp.app.longpoll, Y.lp.app.longpoll.setupLongPollManager);
});
Y.on('lp:context:web_link:changed', function(e) {
window.location = e.new_value;
});
});
//]]>
</script>
<script id="base-helper-functions" type="text/javascript">
//<![CDATA[
// This code is pulled from lp.js that needs to be available on every
// request. Pulling here to get it outside the scope of the YUI block.
function setFocusByName(name) {
// Focus the first element matching the given name which can be focused.
var nodes = document.getElementsByName(name);
var i, node;
for (i = 0; i < nodes.length; i++) {
node = nodes[i];
if (node.focus) {
try {
// Trying to focus a hidden element throws an error in IE8.
if (node.offsetHeight !== 0) {
node.focus();
}
} catch (e) {
LPJS.use('console', function(Y) {
Y.log('In setFocusByName(<' +
node.tagName + ' type=' + node.type + '>): ' + e);
});
}
break;
}
}
}
function selectWidget(widget_name, event) {
if (event && (event.keyCode === 9 || event.keyCode === 13)) {
// Avoid firing if user is tabbing through or simply pressing
// enter to submit the form.
return;
}
document.getElementById(widget_name).checked = true;
}
//]]>
</script>
</head>
<body id="document" itemscope=""
itemtype="http://schema.org/WebPage"
class="tab-unknown
main_only
public
yui3-skin-sam">
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-12833497-1']);
_gaq.push(['_setDomainName', '.launchpad.net']);
_gaq.push(['_setAllowHash', false]);
_gaq.push(['_trackPageview']);
</script>
<script type="text/javascript"
src="https://launchpad.net/+icing/rev17430/google-analytics/ga.js"></script>
<div class="yui-d0">
<div id="locationbar" class="login-logout">
<div id="logincontrol">
<form action="/+logout" method="post">
<input type="hidden" name="loggingout" value="1" />
<a href="/~iganchev" class="sprite person">iganchev (iganchev)</a> •
<input type="submit" name="logout" value="Log Out" />
</form>
</div>
</div><!--id="locationbar"-->
<div id="watermark" class="watermark-apps-portlet">
<div>
<img alt="" width="64" height="64" src="/@@/launchpad-logo" />
</div>
<div class="wide">
<h2 id="watermark-heading"><span>Launchpad.net</span></h2>
</div>
<!-- Application Menu -->
<ul class="facetmenu">
</ul>
</div>
<div id="maincontent" class="yui-main">
<div class="yui-b" dir="ltr">
<div class="context-publication">
<div id="registration" class="registering">
</div>
</div>
<div id="request-notifications">
</div>
<div class="top-portlet">
<h1>Page gone</h1>
<p>There’s no page with this address in Launchpad.</p>
<p>
Check that you entered the address correctly, or search for it:
</p>
<form method="get" accept-charset="UTF-8"
style="margin: 1em 0 1em 0"
action="https://launchpad.net/+search">
<img src="/@@/search.png" />
<input id="text" type="text" name="field.text" size="50" />
<input type="submit" value="Search Launchpad" />
</form>
<div style="margin-bottom: 1em;">
</div>
<ul>
<li>
<a href="https://launchpad.net/">Return to the Launchpad front page</a>
</li>
</ul>
</div>
</div><!-- yui-b -->
</div><!-- yui-main -->
<!-- yui-b side -->
<!-- yui-t4 -->
<div id="footer" class="footer">
<div class="lp-arcana">
<div class="lp-branding">
<a href="https://launchpad.net/"><img src="/@@/launchpad-logo-and-name-hierarchy.png" alt="Launchpad" /></a>
•
<a href="https://launchpad.net/+tour">Take the tour</a>
•
<a href="https://help.launchpad.net/">Read the guide</a>
<form id="globalsearch" method="get"
accept-charset="UTF-8"
action="https://launchpad.net/+search">
<input type="search" id="search-text" name="field.text" />
<input type="image" src="/@@/search" style="vertical-align:5%" alt="Search Launchpad" />
</form>
</div>
</div>
<div class="colophon">
© 2004-2015
<a href="http://canonical.com/">Canonical Ltd.</a>
•
<a href="https://launchpad.net/legal">Terms of use</a>
•
<a href="/support">Contact Launchpad Support</a>
•
<a href="http://blog.launchpad.net/">Blog</a>
•
<a href="http://www.canonical.com/about-canonical/careers">Careers</a>
•
<a href="http://identi.ca/launchpadstatus">System status</a>
<span id="lp-version">
•
r17430
(<a href="https://dev.launchpad.net/">Get the code!</a>)
</span>
</div>
</div>
</div><!-- yui-d0-->
<script id="json-cache-script">LP.cache = {"related_features": {}};</script>
</body>
<!--
Facet name: unknown
Page type: main_only
Has global search: True
Has application tabs: True
Has side portlets: False
At least 4 queries/external actions issued in 0.04 seconds
Features: {'app.mainsite_only.canonical_url': None, 'js.yui_version': None, 'visible_render_time': None, 'baselayout.careers_link.disabled': None}
r17430
-->
</html>
---The page on which it hangs is accessible through web browser. The lazr.restfulclient doesn't like in fact 4xx return codes and raises an error when such is seen. Is this a normal behaviour?
iganchev
commented
9 years ago After some more time spent on that bug here is what it seems happens. The backend calls lazr.restful in order to get the names related to a bug. For some reason the page returns 410. Lazr sees that and raises an error.
In my opinion, lazr shouldn't raise an error, but rather inform the calling function for the return status and it should decide what to do. In our example, we have couple hundred pages to analyze and a 410 (or any other 4XX return status) make the program crash.
What should be the fix for this misbehaviour? Change lazr code? Use other library to get the code?
Thanks,
sduenas
commented
9 years ago This is the same error we fixed in commit 80f742e but in other section of the code. The problem is raised when the information from identities is retrieved. As far as I remember, the launchpad library that we used to retrieve the info has bug and it haven't been fixed yet. I'm afraid it won't never be fixed because the library seems to be abandoned.
@iganchev download the new changes and try it again. It should work now.
If a 410 error arrives, it seems that Bicho does not know how to deal with this and crashes.
This example comes from the OpenStack analysis.
This database can be used as the example to continue with the analysis: http://activity.openstack.org/dash/browser/data/db/tickets.mysql.7z