Bicho does not retrieve all components of a given product

[iKuba]

While trying to retrieve all bugs for the Mozilla Firefox project a few components do not seem to be exist in the Bicho database as they were not retrieved.. Examples of these components are as follows: Developer Tools Net Monitor, Developer Tools Responsive Mode, Developer Tools Command Line and Toolbar, Developer Tools Source Editor, Webapp Runtime, WinQual Reports, Web Apps and Downloads Panel.

The following command was used to retreive bugs from the Mozilla Bugzilla site. bicho --db-user-out=user --db-password-out=password --db-database-out=firefox -d 0 bg -u https://bugzilla.mozilla.org/buglist.cgi?product=Firefox

[canasdiaz]

You're right. Those fields need to be added to the extended table.

Patches/contributions are more than welcome ;)

[iKuba]

I'm sorry, I don't quite understand.. what extended table?

[jgbarah]

sanacl, my guess is that iKuba is talking about components whose tickets are not retrieved by Bicho, not fields that are not retrieved for an specific ticket. Am I right, iKuba?

[iKuba]

That's correct jgbarah, my apologies if I was being unclear!

[iKuba]

Hello, I'd just like to follow up on this ticket as I'd like to have this issue resolved ASAP. I have been unable to identify myself what could be the issue and was wondering if you guys could potentially point me in the right direction?

[sduenas]

@iKuba, may you tell us how many issues are stored in your database?

[iKuba]

@sduenas there are 278664 stored in my database.

[canasdiaz]

I have a database with 100K bugs from Firefox and I do have some of the components you miss. I found them using the following queries:

select distinct(component) from issues_ext_bugzilla where component LIKE '%Developer%' select distinct(component) from issues_ext_bugzilla where component LIKE '%Webapp%' select distinct(component) from issues_ext_bugzilla where component LIKE '%Web Apps%'

Could you please let us know the version of the tool you are using? I guess you just cloned it from master

[iKuba]

Bicho --version tells me I am using 0.9, this was a clone of the master retrieved approximately 15 days ago.

I have tried executing the queries you have specified but I am still coming up with missing components.

[canasdiaz]

@iKuba I'm testing it in our machines with the latest version of master (May 03, 2013). I'll let you know in a few hours if I could reproduce the problem.

[canasdiaz]

First 32K bugs downloaded with 0 rows returned for the queries above.

[canasdiaz]

Downloaded stopped with last issues cached modified on: 2008-01-24

Download restarted. So far no results with the queries I put above (39K bugs). I've been having a look at my old database and the first issue to be downloaded with one of these three components was modified in Dec 2009. So, until we reach that date we can't say anything about the issue

These are the queries I used (remember, we use the delta_ts field to download them by modification date)

select distinct(component), MIN(delta_ts) from issues, issues_ext_bugzilla where issues.id = issues_ext_bugzilla.issue_id AND component LIKE '%Developer%';
select distinct(component), MIN(delta_ts) from issues, issues_ext_bugzilla where issues.id = issues_ext_bugzilla.issue_id AND component LIKE '%Webapp%';
select distinct(component), MIN(delta_ts) from issues, issues_ext_bugzilla where issues.id = issues_ext_bugzilla.issue_id AND component LIKE '%Web Apps%';

[canasdiaz]

@iKuba some results, I have 91K bugs downloaded so far, the latest modification time is '2011-05-31 16:58:25' and the queries return results for the component 'Developer Tools'.

Don't you find this component in your database? I could provide you a dump with the DB I'm getting.

[iKuba]

@sanacl Yes, I do find Developer Tools but this is not quite one of the missing components, the query you provided should also return Developer Tools Net Monitor, Developer Tools Responsive Mode, Developer Tools Command Line and Toolbar and Developer Tools Source Editor which are all their own specific component.

The following are links to these components: https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Netmonitor https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Inspector https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Profiler https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Responsive%20Mode https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Scratchpad https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Source%20Editor https://bugzilla.mozilla.org/buglist.cgi?product=Firefox&component=Developer%20Tools%3A%20Style%20Editor

For the record; I have 278664 bugs in my db the latest modification time being 2013-06-05 06:42:57.

[canasdiaz]

This is really weird. I have now 109623 issues downloaded and I do find the following components:

"Developer Tools",1259
"Developer Tools: Console",287
"Developer Tools: Debugger",196
"Developer Tools: Framework",2
"Developer Tools: Graphic Command",2
"Developer Tools: Inspector",242
"Developer Tools: Netmonitor",8
"Developer Tools: Profiler",5
"Developer Tools: Responsive Mode",1
"Developer Tools: Scratchpad",41
"Developer Tools: Source Editor",32
"Developer Tools: Style Editor",71

The query I used was:

select distinct(component), COUNT(*) as total FROM issues_ext_bugzilla WHERE component LIKE '%Developer%' GROUP BY component;

[iKuba]

That is strange, I will try to regenerate my database and let you know if I pick up the missing components.

[canasdiaz]

@iKuba I can provide you the dump I got. Do you want it?

[iKuba]

Yes please, that would be very appreciated!

[canasdiaz]

@iKuba I've just sent you the dump in a private message. I'm closing the issue because it was not possible to reproduce the error. If you see this strange behaviour again please let me know and re-open the issue.