Migration failed #14

Closed fromage9747 closed 1 year ago

If I run the queries that are being used in the script using Postman, I receive data.

I am testing this out on a small project in Jira that has only one issue. Just to see how things map across in GitLab before going onto my more involved projects.

[INFO] #1/1 Migrating Jira issue ACT-1 ...   Traceback (most recent call last):
  File "/home/jared/jira2gitlab-main/./jira2gitlab.py", line 914, in <module>
    migrate_project(jira_project, gitlab_project)
  File "/home/jared/jira2gitlab-main/./jira2gitlab.py", line 507, in migrate_project
    ).json()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Migration failed

Resetting user privileges..

That would be this request:

        # Epic name to label
        if JIRA_EPIC_FIELD in issue['fields'] and issue['fields'][JIRA_EPIC_FIELD]:
            epic_info = requests.get(
                f"{JIRA_API}/issue/{issue['fields'][JIRA_EPIC_FIELD]}/?fields=summary",
                auth = HTTPBasicAuth(*JIRA_ACCOUNT),
                verify = VERIFY_SSL_CERTIFICATE,
                headers = {'Content-Type': 'application/json'}
            ).json()

I have to admit I haven't used much the JIRA_EPIC_FIELD myself.

So if you try this request with Podman you get data and this data is valid JSON? Can you post the answer you get?

Even if there is no epic information, I'd expect the answer to still be valid JSON.

Sure, here is the JSON response from running the query through Postman:

{
    "expand": "names,schema",
    "startAt": 0,
    "maxResults": 100,
    "total": 1,
    "issues": [
        {
            "expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
            "id": "12101",
            "self": "https://URL_REDACTED/rest/api/2/issue/12101",
            "key": "ACT-1",
            "fields": {
                "issuetype": {
                    "self": "https://URL_REDACTED/rest/api/2/issuetype/10000",
                    "id": "10000",
                    "description": "Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down.",
                    "iconUrl": "https://URL_REDACTED/images/icons/issuetypes/epic.svg",
                    "name": "Epic",
                    "subtask": false
                },
                "components": [],
                "timespent": null,
                "timeoriginalestimate": 144000,
                "description": "DESCRIPTION_REDACTED",
                "project": {
                    "self": "https://URL_REDACTED/rest/api/2/project/10300",
                    "id": "10300",
                    "key": "ACT",
                    "name": "PROJECT_NAME_REDACTED",
                    "projectTypeKey": "software",
                    "avatarUrls": {
                        "48x48": "https://URL_REDACTED/secure/projectavatar?pid=10300&avatarId=10900",
                        "24x24": "https://URL_REDACTED/secure/projectavatar?size=small&pid=10300&avatarId=10900",
                        "16x16": "https://URL_REDACTED/secure/projectavatar?size=xsmall&pid=10300&avatarId=10900",
                        "32x32": "https://URL_REDACTED/secure/projectavatar?size=medium&pid=10300&avatarId=10900"
                    }
                },
                "fixVersions": [],
                "aggregatetimespent": null,
                "resolution": null,
                "customfield_10104": "Initial Functionality",
                "customfield_10105": "ghx-label-2",
                "customfield_10106": null,
                "attachment": [
                    {
                        "self": "https://URL_REDACTED/rest/api/2/attachment/12100",
                        "id": "12100",
                        "filename": "image-2019-04-27-08-56-53-235.png",
                        "author": {
                            "self": "https://URL_REDACTED/rest/api/2/user?username=fromage9747",
                            "name": "fromage9747",
                            "key": "fromage9747",
                            "emailAddress": "EMAIL_REDACTED",
                            "avatarUrls": {
                                "48x48": "https://URL_REDACTED/secure/useravatar?ownerId=fromage9747&avatarId=10800",
                                "24x24": "https://URL_REDACTED/secure/useravatar?size=small&ownerId=fromage9747&avatarId=10800",
                                "16x16": "https://URL_REDACTED/secure/useravatar?size=xsmall&ownerId=fromage9747&avatarId=10800",
                                "32x32": "https://URL_REDACTED/secure/useravatar?size=medium&ownerId=fromage9747&avatarId=10800"
                            },
                            "displayName": "DISPLAY_NAME_REDACTED",
                            "active": true,
                            "timeZone": "TIMEZONE_REDACTED"
                        },
                        "created": "2019-04-27T01:56:53.000+0000",
                        "size": 170695,
                        "mimeType": "image/png",
                        "content": "https://URL_REDACTED/secure/attachment/12100/image-2019-04-27-08-56-53-235.png",
                        "thumbnail": "https://URL_REDACTED/secure/thumbnail/12100/_thumb_12100.png"
                    },
                    {
                        "self": "https://URL_REDACTED/rest/api/2/attachment/12101",
                        "id": "12101",
                        "filename": "FILENAME_REDACTED",
                        "author": {
                            "self": "https://URL_REDACTED/rest/api/2/user?username=fromage9747",
                            "name": "fromage9747",
                            "key": "fromage9747",
                            "emailAddress": "EMAIL_REDACTED",
                            "avatarUrls": {
                                "48x48": "https://URL_REDACTED/secure/useravatar?ownerId=fromage9747&avatarId=10800",
                                "24x24": "https://URL_REDACTED/secure/useravatar?size=small&ownerId=fromage9747&avatarId=10800",
                                "16x16": "https://URL_REDACTED/secure/useravatar?size=xsmall&ownerId=fromage9747&avatarId=10800",
                                "32x32": "https://URL_REDACTED/secure/useravatar?size=medium&ownerId=fromage9747&avatarId=10800"
                            },
                            "displayName": "DISPLAY_NAME_REDACTED",
                            "active": true,
                            "timeZone": "TIMEZONE_REDACTED"
                        },
                        "created": "2019-04-27T02:08:31.000+0000",
                        "size": 59784,
                        "mimeType": "application/zip",
                        "content": "https://URL_REDACTED/secure/attachment/12101/FILENAME_REDACTED"
                    }
                ],
                "aggregatetimeestimate": 144000,
                "resolutiondate": null,
                "workratio": 0,
                "summary": "SUMMARY_REDACTED",
                "lastViewed": "2023-03-25T10:51:56.318+0000",
                "watches": {
                    "self": "https://URL_REDACTED/rest/api/2/issue/ACT-1/watchers",
                    "watchCount": 1,
                    "isWatching": true
                },
                "creator": {
                    "self": "https://URL_REDACTED/rest/api/2/user?username=fromage9747",
                    "name": "fromage9747",
                    "key": "fromage9747",
                    "emailAddress": "EMAIL_REDACTED",
                    "avatarUrls": {
                        "48x48": "https://URL_REDACTED/secure/useravatar?ownerId=fromage9747&avatarId=10800",
                        "24x24": "https://URL_REDACTED/secure/useravatar?size=small&ownerId=fromage9747&avatarId=10800",
                        "16x16": "https://URL_REDACTED/secure/useravatar?size=xsmall&ownerId=fromage9747&avatarId=10800",
                        "32x32": "https://URL_REDACTED/secure/useravatar?size=medium&ownerId=fromage9747&avatarId=10800"
                    },
                    "displayName": "DISPLAY_NAME_REDACTED",
                    "active": true,
                    "timeZone": "TIMEZONE_REDACTED"
                },
                "subtasks": [],
                "created": "2019-04-27T01:56:57.000+0000",
                "reporter": {
                    "self": "https://URL_REDACTED/rest/api/2/user?username=fromage9747",
                    "name": "fromage9747",
                    "key": "fromage9747",
                    "emailAddress": "EMAIL_REDACTED",
                    "avatarUrls": {
                        "48x48": "https://URL_REDACTED/secure/useravatar?ownerId=fromage9747&avatarId=10800",
                        "24x24": "https://URL_REDACTED/secure/useravatar?size=small&ownerId=fromage9747&avatarId=10800",
                        "16x16": "https://URL_REDACTED/secure/useravatar?size=xsmall&ownerId=fromage9747&avatarId=10800",
                        "32x32": "https://URL_REDACTED/secure/useravatar?size=medium&ownerId=fromage9747&avatarId=10800"
                    },
                    "displayName": "DISPLAY_NAME_REDACTED",
                    "active": true,
                    "timeZone": "TIMEZONE_REDACTED"
                },
                "customfield_10000": "{summaryBean=com.atlassian.jira.plugin.devstatus.rest.SummaryBean@4c3a0421[summary={pullrequest=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@6a744b5b[overall=PullRequestOverallBean{stateCount=0, state='OPEN', details=PullRequestOverallDetails{openCount=0, mergedCount=0, declinedCount=0}},byInstanceType={}], build=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@7c39afbd[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BuildOverallBean@34df0df2[failedBuildCount=0,successfulBuildCount=0,unknownBuildCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], review=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@175e1130[overall=com.atlassian.jira.plugin.devstatus.summary.beans.ReviewsOverallBean@2510550[stateCount=0,state=<null>,dueDate=<null>,overDue=false,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], deployment-environment=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@332416e0[overall=com.atlassian.jira.plugin.devstatus.summary.beans.DeploymentOverallBean@7f582deb[topEnvironments=[],showProjects=false,successfulCount=0,count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], repository=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@18e864a1[overall=com.atlassian.jira.plugin.devstatus.summary.beans.CommitOverallBean@12999969[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}], branch=com.atlassian.jira.plugin.devstatus.rest.SummaryItemBean@1991ecc[overall=com.atlassian.jira.plugin.devstatus.summary.beans.BranchOverallBean@3f104c61[count=0,lastUpdated=<null>,lastUpdatedTimestamp=<null>],byInstanceType={}]},errors=[],configErrors=[]], devSummaryJson={\"cachedValue\":{\"errors\":[],\"configErrors\":[],\"summary\":{\"pullrequest\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":\"OPEN\",\"details\":{\"openCount\":0,\"mergedCount\":0,\"declinedCount\":0,\"total\":0},\"open\":true},\"byInstanceType\":{}},\"build\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"failedBuildCount\":0,\"successfulBuildCount\":0,\"unknownBuildCount\":0},\"byInstanceType\":{}},\"review\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"stateCount\":0,\"state\":null,\"dueDate\":null,\"overDue\":false,\"completed\":false},\"byInstanceType\":{}},\"deployment-environment\":{\"overall\":{\"count\":0,\"lastUpdated\":null,\"topEnvironments\":[],\"showProjects\":false,\"successfulCount\":0},\"byInstanceType\":{}},\"repository\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}},\"branch\":{\"overall\":{\"count\":0,\"lastUpdated\":null},\"byInstanceType\":{}}}},\"isStale\":false}}",
                "aggregateprogress": {
                    "progress": 0,
                    "total": 144000,
                    "percent": 0
                },
                "priority": {
                    "self": "https://URL_REDACTED/rest/api/2/priority/3",
                    "iconUrl": "https://URL_REDACTED/images/icons/priorities/medium.svg",
                    "name": "Medium",
                    "id": "3"
                },
                "customfield_10100": "0|i001mn:",
                "customfield_10101": null,
                "customfield_10102": null,
                "labels": [
                    "Feature_Request"
                ],
                "customfield_10103": {
                    "self": "https://URL_REDACTED/rest/api/2/customFieldOption/10000",
                    "value": "To Do",
                    "id": "10000"
                },
                "environment": null,
                "timeestimate": 144000,
                "aggregatetimeoriginalestimate": 144000,
                "versions": [],
                "duedate": null,
                "progress": {
                    "progress": 0,
                    "total": 144000,
                    "percent": 0
                },
                "comment": {
                    "comments": [],
                    "maxResults": 0,
                    "total": 0,
                    "startAt": 0
                },
                "issuelinks": [],
                "votes": {
                    "self": "https://URL_REDACTED/rest/api/2/issue/ACT-1/votes",
                    "votes": 0,
                    "hasVoted": false
                },
                "worklog": {
                    "startAt": 0,
                    "maxResults": 20,
                    "total": 0,
                    "worklogs": []
                },
                "assignee": null,
                "updated": "2019-04-27T02:08:33.000+0000",
                "status": {
                    "self": "https://URL_REDACTED/rest/api/2/status/10000",
                    "description": "",
                    "iconUrl": "https://URL_REDACTED/",
                    "name": "To Do",
                    "id": "10000",
                    "statusCategory": {
                        "self": "https://URL_REDACTED/rest/api/2/statuscategory/2",
                        "id": 2,
                        "key": "new",
                        "colorName": "blue-gray",
                        "name": "To Do"
                    }
                }
            }
        }
    ]
}

At the moment I don't have a setup to test it, but looking at the code and at you last post, I think this bit:

        # Epic name to label
        if JIRA_EPIC_FIELD in issue['fields'] and issue['fields'][JIRA_EPIC_FIELD]:
            epic_info = requests.get(
                f"{JIRA_API}/issue/{issue['fields'][JIRA_EPIC_FIELD]}/?fields=summary",
                auth = HTTPBasicAuth(*JIRA_ACCOUNT),
                verify = VERIFY_SSL_CERTIFICATE,
                headers = {'Content-Type': 'application/json'}
            ).json()

was intended to be (add missing ['id']):

        # Epic name to label
        if JIRA_EPIC_FIELD in issue['fields'] and issue['fields'][JIRA_EPIC_FIELD]:
            epic_info = requests.get(
                f"{JIRA_API}/issue/{issue['fields'][JIRA_EPIC_FIELD]['id']}/?fields=summary",
                auth = HTTPBasicAuth(*JIRA_ACCOUNT),
                verify = VERIFY_SSL_CERTIFICATE,
                headers = {'Content-Type': 'application/json'}
            ).json()

Could you try this?

@swingbit

One step closer. That has now progressed, but onto another problem:

user@docker-vm:~/jira2gitlab-main$ ./jira2gitlab.py

Migrating ACT to user/ACT
[INFO] Loading Jira issues from project ACT ... 1

[INFO] #1/1 Migrating Jira issue ACT-1 ...   [WARN] Unable to migrate attachment: https://REDACTED/secure/attachment/15601/image-2019-04-27-08-56-53-235.png ...
[WARN] Unable to migrate attachment: https://REDACTED/secure/attachment/15602/file.zip ...
data: {'created_at': '2019-04-27T01:56:57.000+0000', 'assignee_ids': None, 'title': '[ACT-1] [Feature] - Nodejs Backend', 'description': 'REDACTED.  \n!image-2019-04-27-08-56-53-235.png|width=1032,height=515!  \n^file.zip\n\n___\n\n**Imported from Jira issue [ACT-1](https://REDACTED/browse/ACT-1)**\n\n**Original creator of the issue: Jira user fromage9747**\n\n', 'milestone_id': None, 'labels': 'jira-import, Feature_Request, T::epic, P::normal, As an Agile team, I\'d like to learn about Scrum >> Click the "TDP-1" link at the left of this row to see detail in the Description tab on the right'} ...
Traceback (most recent call last):
  File "/home/user/jira2gitlab-main/./jira2gitlab.py", line 575, in migrate_project
    gl_issue.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://192.168.1.177:58443/api/v4/projects/7/issues

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/jira2gitlab-main/./jira2gitlab.py", line 914, in <module>
    migrate_project(jira_project, gitlab_project)
  File "/home/user/jira2gitlab-main/./jira2gitlab.py", line 578, in migrate_project
    raise Exception(f"Unable to create Gitlab issue for Jira issue {issue['key']}\n{e}")
Exception: Unable to create Gitlab issue for Jira issue ACT-1
403 Client Error: Forbidden for url: https://192.168.1.177:58443/api/v4/projects/7/issues

Migration failed

Resetting user privileges..

I have tried removing the attachments from the Jira issue, but the issue persists. I have used True and False for the self-signed SSL. The issue persists. I have tried using the full SSL DNS for the GitLab server as well as the direct IP address. The issue persists.

If I navigate to https://192.168.1.177:58443/api/v4/projects/7/issues directly, I receive {"message":"404 Project Not Found"}. However, if I use the normal name URL of https://gitlab.mydomain.com/api/v4/projects/7/issues I get an empty array.

The ProjectID in GitLab is 7, so I am not sure what is going on here?

@swingbit Hey man, do I need to raise a new request for the next issue?

When you get an empty array, it works correctly (you don't have any issues yet). It works because you are already authenticated with that URL. If you change it to the internal IP, you are not authenticated for it, so you get an error. That is all fine.

The script fails because the authentication isn't correct (you get a 403 ). Connectivity is ok, otherwise you wouldn't get a 403. .It's difficult for me to know why that happens, but it's most likely a misconfiguration.

Are you using a (non-expired) token with admin privileges for your gitlab authentication?

If this doesn't work, then you're not using a valid token:

GITLAB_TOKEN=<your token>
GITLAB_API="https://gitlab.mydomain.com/api/v4"
curl -H 'Content-Type: application/json' -H "Authorization: Bearer ${GITLAB_TOKEN}" "${GITLAB_API}/projects/7/issues"

@swingbit Thank you for pointing me in the right direction. The access token was valid and not expired, but I had only given it rights to the API. I created a new one and just gave it rights to everything. Progress has been made!

@swingbit Excellent. Thanks a million. This works beautifully.

swingbit / jira2gitlab

Migration failed #14