search

openhatch / oh-bugimporters

Bug importers for the OpenHatch project oh-mainline
https://oh-bugimporters.readthedocs.org/
GNU Affero General Public License v3.0
12 stars 28 forks source link

Fixed simple typo for optional argument of process_bugs #105

Closed dirkbaechle closed 9 years ago

dirkbaechle commented 9 years ago

This patch fixes a little typo (bugs -> bug) in the "google" bugimporter. It seems to be the main reason for the currently failing bug scraping process on the OH server!

ehashman commented 9 years ago

Nice catch! Curious how long this has been in here---perhaps since the redesign of this scrape.

Super happy to merge this, but I get the feeling this won't be solving all our problems. I still can't explain why the new OpenHatch GitHub tracker is not being scraped, e.g., and I'm not sure what's happening with SCons.

dirkbaechle commented 9 years ago

The wrong attribute name leads to a TypeError exception for "OpenStates", which doesn't seem to get catched...so the scraping stops before the following projects (like OpenHatch and SCons) can get processed. I tested the scraping locally on my machine (not for the full data, but a part of it), and am rather confident that this patch solves the problem...but we'll see.

ehashman commented 9 years ago

Interesting! Thanks for the explanation. Fingers crossed.

dirkbaechle commented 9 years ago

Hi Elana,

sorry for having to bother you with this, ... again. ;) A few days ago I sent the message below to Asheesh, and got no response so far. I guess he's super-swamped with work as always, that's okay. I just wanted you to have the same information, for the case that you'd want to have another look at the problem.

I think it would be good if someone (you, Asheesh,...) could repeat my experiment, and "bug scrape" everything on another machine...not the server. If this should work successfully, comparing the versions of installed software and packages (mysql, django,...) between the machines should then show differences...at least that's the only explanation I have left for this error. What do you think?

Best regards,

Dirk

Hi Asheesh,

after Elana merged my pull request #105 (for oh-bugimporters), I expected the scraping process to work. But it still doesn't, as the latest log on http://inside.openhatch.org/crawl-logs/ shows. No bugs listed for SCons, yet. :(

Is the current "master", for both oh-mainline and oh-bugimporters, not active yet?

Yesterday, I successfully did a complete "scrape" on my local machine...so it doesn't seem to be a data error. If the problem persists, it appears to be a server problem (other Django version, or somethin'?). However, the problem is I can't really help anymore in this realm. All I can offer at the moment is, to upload my scrape results (and log, both about 15MB each) to an Owncloud and give you the link for downloading it. Then you could try to import the JSON data directly, just to confirm that it's in fact no data problem, and to have a somewhat updated state in your database.

Sorry, for having to step on your foot a little about this...but I think the bug scraping is one of the major vital services that OH offers to people, so let's get it back to life soon.

Best regards,

Dirk

ehashman commented 9 years ago

Yeah, I'll have to look into it with Asheesh. I'm in crunch time right now with school so if I take a deep look at this it will probably be mid-Dec. I continue to be mystified at the failures.

paulproteus commented 9 years ago

I agree that it'll be great to get it back to life soon. Thanks for both your work, @dirkbaechle and @ehashman on this. I've carved out some time on my calendar on Tuesday to make sure I don't forget to look into that.