Add Azure Devops Service (with tests)

[cam-barts]

I screwed up 745's git history, and I will be more productive if I just nuke the branch and start over. Initial commit with comments from @ryneeverett from old PR. Currently working on creating tests. Closes #745

[cam-barts]

Looks like build is failing because Rust isn't there for Arch: ppc64le, which is required for cryptography package. See PR here https://github.com/pyca/cryptography/issues/5771

Unsure how you folks want to proceed with that. Opened #809

[ryneeverett]

A rebase on develop should fix the build.

[cam-barts]

I think the rebase is what screwed it up in the first place. This is a whole new branch, with just the relevant code on there.

[cam-barts]

I'm unsure how to kick the travis build again though if the issue got closed.

[ryneeverett]

I'd suggest thinking of a branch as a history of commits. You can see it here. In order to use the new (fixed) CI you just need to rewrite your branch's history so that your new commits follow the latest upstream commit. So:

$ git clone https://github.com/cam-barts/bugwarrior.git
Cloning into 'bugwarrior'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 6695 (delta 4), reused 11 (delta 4), pack-reused 6682
Receiving objects: 100% (6695/6695), 1.54 MiB | 1.96 MiB/s, done.
Resolving deltas: 100% (4916/4916), done.

$ cd bugwarrior

$ git remote add upstream https://github.com/ralphbean/bugwarrior.git

$ git remote -v
origin  https://github.com/cam-barts/bugwarrior.git (fetch)
origin  https://github.com/cam-barts/bugwarrior.git (push)
upstream    https://github.com/ralphbean/bugwarrior.git (fetch)
upstream    https://github.com/ralphbean/bugwarrior.git (push)

$ git fetch upstream
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 14 (delta 8), reused 11 (delta 8), pack-reused 2
Unpacking objects: 100% (14/14), 2.31 KiB | 1.16 MiB/s, done.
From https://github.com/ralphbean/bugwarrior
 * [new branch]      bitbucket-oracle                -> upstream/bitbucket-oracle
 * [new branch]      config-parser-no-option-error   -> upstream/config-parser-no-option-error
 * [new branch]      develop                         -> upstream/develop
 * [new branch]      feature/annotations             -> upstream/feature/annotations
 * [new branch]      feature/bugzilla                -> upstream/feature/bugzilla
 * [new branch]      feature/cache-for-bitly         -> upstream/feature/cache-for-bitly
 * [new branch]      feature/comments-as-annotations -> upstream/feature/comments-as-annotations
 * [new branch]      feature/jira-sprints            -> upstream/feature/jira-sprints
 * [new branch]      feature/kill-debug              -> upstream/feature/kill-debug
 * [new branch]      feature/modern-requests         -> upstream/feature/modern-requests
 * [new branch]      feature/multiprocessing         -> upstream/feature/multiprocessing
 * [new branch]      feature/pep8                    -> upstream/feature/pep8
 * [new branch]      feature/pesky-asbool            -> upstream/feature/pesky-asbool
 * [new branch]      feature/py3                     -> upstream/feature/py3
 * [new branch]      feature/pynotify                -> upstream/feature/pynotify
 * [new branch]      feature/taiga-error-handling    -> upstream/feature/taiga-error-handling
 * [new branch]      master                          -> upstream/master
 * [new branch]      refactor_bugwarrior             -> upstream/refactor_bugwarrior
 * [new branch]      revert-406-python3              -> upstream/revert-406-python3
 * [new branch]      task-merge                      -> upstream/task-merge
 * [new tag]         1.8.0                           -> 1.8.0

$ git checkout azure_devops_with_test
Branch 'azure_devops_with_test' set up to track remote branch 'azure_devops_with_test' from 'origin'.
Switched to a new branch 'azure_devops_with_test'

$ git rebase upstream/develop
Successfully rebased and updated refs/heads/azure_devops_with_test.

$ git log -3
commit 7a2565a8506c55475e0ba7a367aa64a058598673 (HEAD -> azure_devops_with_test)
Author: Cam Barts <CameronD.Barts@gmail.com>
Date:   Sun Mar 28 15:40:29 2021 -0400

    Added some tests

commit 46dc986c3b7e5cd40ddc980adec952d3519c0272
Author: Cam Barts <CameronD.Barts@gmail.com>
Date:   Sat Mar 27 22:08:05 2021 -0400

    Add Azure Devops Service

commit ea2c04ed7d218e8d82aaf5081a7e042a02f73540 (upstream/develop)
Author: ryneeverett <ryneeverett@gmail.com>
Date:   Tue Nov 10 15:53:19 2020 +0000

$ git push -f

[ryneeverett]

Hmm. You've only got two commits so let's try a simpler tactic to save this branch:

$ git checkout azure_devops_with_test 
Switched to branch 'azure_devops_with_test'

$ git checkout -b azure_devops_with_test_backup
Switched to a new branch 'azure_devops_with_test_backup'

$ git branch -d azure_devops_with_test 
Deleted branch cam-barts/azure_devops_with_test (was ef7cbc3).

$ git checkout upstream/develop
Note: switching to 'upstream/develop'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at ea2c04e travis -> github actions

$ git checkout -b azure_devops_with_test
Switched to a new branch 'azure_devops_with_test'

$ git cherry-pick 9c89f0ff330886e692c05ec2367163cd352236df
[azure_devops_with_test 372b125] Add Azure Devops Service
 Author: Cam Barts <CameronD.Barts@gmail.com>
 Date: Sat Mar 27 22:08:05 2021 -0400
 3 files changed, 209 insertions(+), 6 deletions(-)
 create mode 100644 bugwarrior/services/azuredevops.py

$ git cherry-pick 09ec1b2e2af86fe6a8f77572a969c1f2376d113a
[azure_devops_with_test 6490e59] Added some tests
 Author: Cam Barts <CameronD.Barts@gmail.com>
 Date: Sun Mar 28 15:40:29 2021 -0400
 2 files changed, 219 insertions(+), 4 deletions(-)
 create mode 100644 tests/test_azuredevops.py

$ git log -3
commit 6490e59ae447d51f1d861ae4d1b0b5d47c24e793 (HEAD -> azure_devops_with_test)
Author: Cam Barts <CameronD.Barts@gmail.com>
Date:   Sun Mar 28 15:40:29 2021 -0400

    Added some tests

commit 372b1250c18e934afcda8efdc052f5fc8d03d0b0
Author: Cam Barts <CameronD.Barts@gmail.com>
Date:   Sat Mar 27 22:08:05 2021 -0400

    Add Azure Devops Service

commit ea2c04ed7d218e8d82aaf5081a7e042a02f73540 (upstream/develop)
Author: ryneeverett <ryneeverett@gmail.com>
Date:   Tue Nov 10 15:53:19 2020 +0000

    travis -> github actions

[aquaherd]

Could you please also add the documentation on how to configure? I would love to test drive this.

[cam-barts]

Hey @aquaherd

I've needed to come back to this for awhile for both tests and documentation. So you aren't waiting on me having time to put that effort in, I can give you the config items really quick so you can try it out. I'd love another set of eyes.

[my-azure-devops]
# Required
service = azuredevops
ado.organization = My_Company
ado.project = My_Project
ado.PAT = abc123 # This is a Personal Access Token with 'Work Items' scope of Read

# Extra
ado.host = dev.azure.com # If you have a different ado host than dev.azure.com
ado.wiql_filter = [System.WorkItemType] = "Bug" # This will append a wiql query to the default query ("SELECT [System.Id] FROM workitems WHERE [System.AssignedTo] = @me")

[aquaherd]

If an issue #4711 is available from the following URL: https://some.company.com/tfs/CV.TPC/CV/_queries/edit/4711

Then I am doing this:

[me]
17 service = azuredevops
18 ado.host = some.company.com
19 ado.PAT = abcdef...
20 ado.project = CV
21 ado.organization = CV.TPC

And I get this:

ERROR:bugwarrior.services:Worker for [me] failed: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/services/__init__.py", line 499, in _aggregate_issues
    for issue in service.issues():
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/services/azuredevops.py", line 189, in issues
    issue_ids = self.get_query()
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/services/azuredevops.py", line 168, in get_query
    list_of_items = self.client.get_work_items_from_query(default_query)
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/services/azuredevops.py", line 55, in get_work_items_from_query
    return [workitem['id'] for workitem in resp.json()["workItems"]]
  File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
INFO:bugwarrior.services:Done with [me] in 0.227448s
INFO:bugwarrior.services:Terminating workers
ERROR:bugwarrior.command:Aborted (critical error in target 'me')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/command.py", line 73, in pull
    synchronize(issue_generator, config, main_section, dry_run)
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/db.py", line 368, in synchronize
    for issue in issue_generator:
  File "/usr/local/lib/python3.7/dist-packages/bugwarrior-1.8.0-py3.7.egg/bugwarrior/services/__init__.py", line 568, in aggregate_issues
    "critical error in target '{}'".format(target))
RuntimeError: critical error in target 'me'

What am I doing wrong?

[cam-barts]

Based on your url, it looks like you are on Team Foundation Server, which uses a different version of the api, and I am not completely sure if its fully compatible.

You could try changing the API version from 6.0-preview.2 to 4.1 based on this document and give it a spin.

diff --git a/bugwarrior/services/azuredevops.py b/bugwarrior/services/azuredevops.py
index cae7a79..60d0266 100644
--- a/bugwarrior/services/azuredevops.py
+++ b/bugwarrior/services/azuredevops.py
@@ -39,7 +39,7 @@ class AzureDevopsClient(ServiceClient):
             "content-type": "application/json",
         }
         self.session = requests.Session()
-        self.params = {"api-version": "6.0-preview.2"}
+        self.params = {"api-version": "4.1"}
         for k, v in self.headers.items():
             self.session.headers[k] = v

I'd love to know if it works, and if so, I'll add a config item for the rest api version.

[ryneeverett]

Now that you've removed the assignment filter from the default query it occurs to me that this service probably doesn't support the common service configuration option only_if_assigned which looks like would require at least a get_owner method.

[ryneeverett]

After digging into this a bit more I realized the get_owner method is only called if the service's issue method uses the base class' include method to filter issues. It's more performant to implement the common service configuration options via the api though (pivotaltracker, redmine, and trello services do this) so I'd recommend removing the get_owner method and just appending the necessary filters to the query if only_if_assigned or also_unassigned are toggled on (assuming it's possible to query for unassigned issues).