search

GothenburgBitFactory / bugwarrior

Pull github, bitbucket, and trac issues into taskwarrior
http://pypi.python.org/pypi/bugwarrior
GNU General Public License v3.0
732 stars 209 forks source link

Gitlab pull - tasks recreated #950

Closed marcoskv closed 1 year ago

marcoskv commented 1 year ago

Hi, I recently started using taskwarrior with bugwarrior and I found this combo very helpful! Thanks!

While configuring bugwarrior with one private gitlab instance, I encountered an issue. I configured 4 repositories. For 3 of them almost no issues, a part two tasks that were updated every time. I managed to fix it by adding static_fields = gitlabdescription.

The major issue is in the 4th repository. The first pull works fine. All subsequent pulls are recreating the tasks. As an example, see below 1st and 2st pull.

[marco@sampei ansible]$ bugwarrior-pull 
INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 1 workers.
INFO:bugwarrior.services:Working on [gitlab_pm]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/infrastructure%2Fproject-management?simple=true HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=1&per_page=100 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=2&per_page=100 HTTP/1.1" 200 None
DEBUG:bugwarrior.services.gitlab: Found 123 issues.
DEBUG:bugwarrior.services.gitlab: Pruned down to 41 issues.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/merge_requests?state=opened&page=1&per_page=100 HTTP/1.1" 200 2
DEBUG:bugwarrior.services.gitlab: Found 0 merge requests.
DEBUG:bugwarrior.services.gitlab: Pruned down to 0 merge requests.
INFO:bugwarrior.services:Done with [gitlab_pm] in 2.222076s
INFO:bugwarrior.services:Done aggregating remote issues.
INFO:bugwarrior.db:Adding 41 tasks
INFO:bugwarrior.db:Adding task KVM hosts + VMs
INFO:bugwarrior.db:Adding task Welcome Kry
INFO:bugwarrior.db:Adding task Provision kvm-003
...
INFO:bugwarrior.db:Adding task Setup a Foreman Proxy in each VLAN
INFO:bugwarrior.db:Adding task Bonding configuration
INFO:bugwarrior.db:Adding task oVirt cluster PoC
INFO:bugwarrior.db:Updating 0 tasks
INFO:bugwarrior.db:Closing 0 tasks

[marco@sampei ansible]$ bugwarrior-pull 
INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 1 workers.
INFO:bugwarrior.services:Working on [gitlab_pm]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/infrastructure%2Fproject-management?simple=true HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=1&per_page=100 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=2&per_page=100 HTTP/1.1" 200 None
DEBUG:bugwarrior.services.gitlab: Found 123 issues.
DEBUG:bugwarrior.services.gitlab: Pruned down to 41 issues.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/merge_requests?state=opened&page=1&per_page=100 HTTP/1.1" 200 2
DEBUG:bugwarrior.services.gitlab: Found 0 merge requests.
DEBUG:bugwarrior.services.gitlab: Pruned down to 0 merge requests.
INFO:bugwarrior.services:Done with [gitlab_pm] in 2.360802s
INFO:bugwarrior.services:Done aggregating remote issues.
INFO:bugwarrior.db:Adding 41 tasks
INFO:bugwarrior.db:Adding task KVM hosts + VMs
INFO:bugwarrior.db:Adding task Welcome Kry
INFO:bugwarrior.db:Adding task Provision kvm-003
...
INFO:bugwarrior.db:Adding task Setup a Foreman Proxy in each VLAN
INFO:bugwarrior.db:Adding task Bonding configuration
INFO:bugwarrior.db:Adding task oVirt cluster PoC
INFO:bugwarrior.db:Updating 0 tasks
INFO:bugwarrior.db:Closing 41 tasks
INFO:bugwarrior.db:Completing task 9474ceea-44d0-44c0-b9b1-0590d17f81f5 Bonding configuration
INFO:bugwarrior.db:Completing task ec9299ff-166f-4812-b1db-c47428744d60 Setup a Foreman Proxy in each VLAN
INFO:bugwarrior.db:Completing task 5008929f-41ef-478f-9f07-e18054a60723 Provision kvm-003
...
INFO:bugwarrior.db:Completing task e772a477-42d2-4596-84b7-9910cb21f048 oVirt cluster PoC
INFO:bugwarrior.db:Completing task d2fd5d53-2116-459f-8201-ff00877d6a92 Welcome Kry
INFO:bugwarrior.db:Completing task 2737d806-9d40-4d63-9993-e85bffa075c8 KVM hosts + VMs

All subsequent pulls are exactly like the last one, and all tasks are recreated.

I tried to play with the different options, add more static_fields, set as static_fields all Datetime gitlab fields, set custom project template and description... but the result is always the same.

I have seen some other tickets with similar issues for Gitlab, due to IDs or DateTime fields, but I could not find a workaround or solution to my issue.

At the end taskwarrior gets filled with the same completed tasks. Example

[marco@sampei ansible]$ task completed Welcome

ID UUID     Created    Completed  Age   P Project Due        Description      
 - d2fd5d53 2022-07-19 2022-07-29 10d   M pm      2022-07-29 Welcome Kry
 - f3f63ae8 2022-07-19 2022-07-29 10d   M pm      2022-07-29 Welcome Kry
 - 11a223c4 2022-07-19 2022-07-29 10d   M pm      2022-07-29 Welcome Kry

3 tasks

And here is the task itself

[marco@sampei ansible]$ task 23
No command specified - assuming 'information'.

Name                 Value                                                                                              
ID                   23                                                                                                 
Description          Welcome Kry                                                                                  
Status               Pending                                                                                            
Project              pm
Entered              2022-07-19 14:16:47 (10d)                                                                          
Due                  2022-07-29 02:00:00
Last modified        2022-07-29 15:58:24 (2min)                                                                         
Virtual tags         DUE DUETODAY MONTH OVERDUE PENDING PRIORITY PROJECT QUARTER READY TODAY UDA UNBLOCKED WEEK YEAR
UUID                 65cc68b2-dccc-4614-89c5-6dd5856f31e4                                                               
Urgency              4.022
Gitlab Assignee      marco                                                                                              
Gitlab Author        marco
Gitlab Created       1658233007                                                                                         
Gitlab Description   I open this ticket for planning the first activities of Kry...
Gitlab Downvotes     0                                                                                                  
Gitlab Due Date      1659052800
Gitlab Namespace     infrastructure                                                                              
Gitlab Issue/MR      511
Gitlab Repo Slug     -management                                                                                        
Gitlab Issue/MR      opened
State
Gitlab Title         Welcome Kry                                                                                  
Gitlab Type          issue
Gitlab Updated       1659097803                                                                                         
Gitlab Upvotes       0
Gitlab URL           https://gitlab-host/gitlab-url/infrastructure/project-management/issues/511    
Gitlab MR            0
Work-In-Progress
Flag
Priority             M                                                                                                  

    project             1 *    1 =      1
    due             0.756 *   12 =   9.07
    age             0.027 *    2 =  0.055
    UDA priority.M      1 *  3.9 =    3.9
    PROJECT pm          1 *  -10 =    -10
                                   ------
                                    4.022

The only weird thing I have noticed on this repository, is that, since the name is project-management, it gets truncated to -management. Thus I used a configuration like

[gitlab_pm]
service = gitlab
gitlab.login = marco
gitlab.token = xxx
gitlab.host = gitlab-host/gitlab-url
gitlab.include_repos = infrastructure/project-management
gitlab.only_if_assigned = marco
gitlab.description_template = {{ gitlabtitle }}
gitlab.project_template = pm

To have the tasks under project = pm.

Any hint or suggestion to try? Thanks!

ryneeverett commented 1 year ago

Nothing comes to mind. How did you install bugwarrior? Are you using a released version or the latest development version?

marcoskv commented 1 year ago

I'm on Fedora 36. Since there was not an official package for bugwarrior, I created a virtualenv with pipenv (based on python 3.10.5), and installed bugwarrrior inside.

((bugwarrior) ) [marco@sampei bugwarrior]$ pip list
Package            Version
------------------ ---------
bugwarrior         1.8.0
certifi            2022.6.15
charset-normalizer 2.0.12
click              8.1.3
decorator          5.1.1
dogpile.cache      1.1.6
future             0.18.2
idna               3.3
Jinja2             3.1.2
kitchen            1.2.6
lockfile           0.12.2
MarkupSafe         2.1.1
pbr                5.9.0
pip                21.3.1
python-dateutil    2.8.2
pytz               2022.1
requests           2.28.0
setuptools         59.6.0
six                1.16.0
stevedore          3.5.0
taskw              2.0.0
urllib3            1.26.9
wheel              0.37.1
ryneeverett commented 1 year ago

Thanks. I think your intuition that "project" was getting removed at insertion was on track. This is probably a similar case to #733 in which a taskwarrior keyword ("project") at the beginning of a string is being interpreted as a separate argument.

Similarly to #733 this might be fixed by upgrading taskwarrior. (What version are you using?) Also similarly, the solution probably needs to be implemented upstream in either taskwarrior or taskw.

marcoskv commented 1 year ago

I was pretty sure to have installed the latest version of taskwarrior, and instead I'm on 2.5.3. I will give it a try tomorrow with v2.6.2 and report back.

If the upgrade does not solve this issue, you suggest to open a ticket in the taskwarrrior repository, with a link to this issue?

ryneeverett commented 1 year ago

If the upgrade does not solve this issue, you suggest to open a ticket in the taskwarrrior repository, with a link to this issue?

No, it needs to remain a bugwarrior issue until we've narrowed it down into a more specific upstream bug. We're calling taskw which in turn makes a call to taskwarrior. We would need to identify what call we're making is exhibiting inappropriate behavior in order to report an actionable issue upstream.

marcoskv commented 1 year ago

I uninstalled the fedora package of taskwarrior 2.5.3, and installed taskwarrior v2.6.2 from source.

Re-executed a few times bugwarrior-pull. At the first execution, the tasks are still recreated. In all subsequent executions, everything seems to work fine, no more recreated tasks.

[marco@sampei ~]$ bugwarrior-pull 
INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 1 workers.
INFO:bugwarrior.services:Working on [gitlab_pm]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/infrastructure%2Fproject-management?simple=true HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=1&per_page=100 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=2&per_page=100 HTTP/1.1" 200 None
DEBUG:bugwarrior.services.gitlab: Found 121 issues.
DEBUG:bugwarrior.services.gitlab: Pruned down to 40 issues.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/merge_requests?state=opened&page=1&per_page=100 HTTP/1.1" 200 2
DEBUG:bugwarrior.services.gitlab: Found 0 merge requests.
DEBUG:bugwarrior.services.gitlab: Pruned down to 0 merge requests.
INFO:bugwarrior.services:Done with [gitlab_pm] in 2.298777s
INFO:bugwarrior.services:Done aggregating remote issues.
INFO:bugwarrior.db:Adding 40 tasks
INFO:bugwarrior.db:Adding task Welcome Kry
...
INFO:bugwarrior.db:Adding task oVirt cluster PoC
INFO:bugwarrior.db:Updating 0 tasks
INFO:bugwarrior.db:Closing 41 tasks
INFO:bugwarrior.db:Completing task 65cc68b2-dccc-4614-89c5-6dd5856f31e4 Welcome Kry
...
INFO:bugwarrior.db:Completing task 5f648f3e-d81f-4391-ba0a-1bbfe4a3e302 oVirt cluster PoC

[marco@sampei ~]$ bugwarrior-pull 
INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 1 workers.
INFO:bugwarrior.services:Working on [gitlab_pm]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/infrastructure%2Fproject-management?simple=true HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=1&per_page=100 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=2&per_page=100 HTTP/1.1" 200 None
DEBUG:bugwarrior.services.gitlab: Found 121 issues.
DEBUG:bugwarrior.services.gitlab: Pruned down to 40 issues.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/merge_requests?state=opened&page=1&per_page=100 HTTP/1.1" 200 2
DEBUG:bugwarrior.services.gitlab: Found 0 merge requests.
DEBUG:bugwarrior.services.gitlab: Pruned down to 0 merge requests.
INFO:bugwarrior.services:Done with [gitlab_pm] in 2.408595s
INFO:bugwarrior.services:Done aggregating remote issues.
INFO:bugwarrior.db:Adding 0 tasks
INFO:bugwarrior.db:Updating 0 tasks
INFO:bugwarrior.db:Closing 0 tasks

[marco@sampei ~]$ bugwarrior-pull 
INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 1 workers.
INFO:bugwarrior.services:Working on [gitlab_pm]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/infrastructure%2Fproject-management?simple=true HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=1&per_page=100 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/issues?state=opened&page=2&per_page=100 HTTP/1.1" 200 None
DEBUG:bugwarrior.services.gitlab: Found 121 issues.
DEBUG:bugwarrior.services.gitlab: Pruned down to 40 issues.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): gitlab-host:443
DEBUG:urllib3.connectionpool:https://gitlab-host:443 "GET /gitlab-url/api/v4/projects/197/merge_requests?state=opened&page=1&per_page=100 HTTP/1.1" 200 2
DEBUG:bugwarrior.services.gitlab: Found 0 merge requests.
DEBUG:bugwarrior.services.gitlab: Pruned down to 0 merge requests.
INFO:bugwarrior.services:Done with [gitlab_pm] in 2.349686s
INFO:bugwarrior.services:Done aggregating remote issues.
INFO:bugwarrior.db:Adding 0 tasks
INFO:bugwarrior.db:Updating 0 tasks
INFO:bugwarrior.db:Closing 0 tasks

Many thanks for your help, I did not realize I was on a old taskwarrior version.