search

openhatch / oh-bugimporters

Bug importers for the OpenHatch project oh-mainline
https://oh-bugimporters.readthedocs.org/
GNU Affero General Public License v3.0
12 stars 28 forks source link

Google Code bug importer API changes break our importer #74

Closed ghost closed 10 years ago

ghost commented 10 years ago

Comment by paulproteus:

Per https://code.google.com/p/support/wiki/IssueTrackerAPI and http://googleblog.blogspot.com/2012/12/winter-cleaning.html , the code.google.com APIs that we rely on in oh-bugimporters to download bug data from Google Code now give HTTP status 404 for all requests.

We need, therefore, to reimplement the functionality of the bug importers via either scraping or the undocumented "issueupdates basic" API.

The issueupdates API can be found by visiting a bug list like https://code.google.com/p/jmonkeyengine/issues/list?can=2&q=label%3AProduct-jME3 and then viewing HTML source, and finding this tag:

<link type="application/atom+xml" rel="alternate" href="/feeds/p/jmonkeyengine/issueupdates/basic">

I believe (but have not verified at all) that everything we do with the old Google Code Issues API works the same with issueupdates/basic; it's just that we'd have to adjust all the URLs that we request data from.

Luckily, this is something that anyone can do! http://oh- bugimporters.readthedocs.org/ has reasonably good documentation, and you could follow the example of the current google code import code. Your best bet for understanding how to make this change is to start by reading the existing code within https://github.com/openhatch/oh-bugimporters that makes the Google Code importer work.

If you choose to do this, make sure to claim this bug by marking it as "Assigned To" yourself.

Comment by ehashman:

Claiming this for a couple hours.

Comment by ehashman:

Fix is not so easy. Appears that /feeds/p/service/issueupdates/basic is also deprecated, e.g. http://code.google.com/feeds/p/jmonkeyengine/issueupdates/basic/53 gives a 404.

We will have to scrape the data.

Comment by ehashman:

Scraper for the google bugs will be based off the model in trac.py. HTML is not too bad in terms of structure.

Just need to get a chance to work on it. Hopefully should finish development in the next couple weeks.

Comment by shaunduncan:

ehashman turned this over to me

Comment by shaunduncan:

I've been looking around a bit, and I'm going to be reworking some of the internals of the Google bugimporter so that it uses successive calls to pull CSV data directly from the issue tracker. I think this makes the most sense as the CSV contents can be controlled to give us most of the data we need, with one exception: the issue description. That will need to be scraped from the issue detail page.

Also, CSV will let us avoid dealing with potentially messy feed parsing for issue feeds that have little, if any, documentation.

Comment by shaunduncan:

I've submitted PRs for oh-mainline (https://github.com/openhatch/oh- mainline/pull/246) and oh-bugimporters (https://github.com/openhatch/oh- bugimporters/pull/50) to fix this.

Comment by ehashman:

Opened new PR https://github.com/openhatch/oh-bugimporters/pull/55 in order to get this merged ASAP with code review feedback

Comment by ehashman:

Finished epically long pair-code review, merged code. There are still some outlying problems but this is essentially resolved.

Asheesh will review the oh-mainline commits, do a deploy, and that will close this issue.

Comment by ehashman:

Also merged the counterpart https://github.com/openhatch/oh-mainline/pull/246 in mainline. Unless this breaks, marking as closed.

Status: resolved Nosy List: ehashman, paulproteus, shaunduncan, sunu Priority: urgent Imported from roundup ID: 871 (view archived page) Last modified: 2014-07-01.05:00:34