Attachments fail to download from Jira

[nickn00]

Describe the problem When I run the latest Jira export tool, it fails to download Attachments within issues on Jira (please see attached screenshots). I've found that the last version to download them successfully is 3.0.85, so I think it may be due to this previous bug fix: "Bugfix/do not try to migrate deleted attachments #618"

To Reproduce Steps to reproduce the behavior:

1. Use the config-agile file below and run jira-export on a project with an issue that contains an attachment
2. Check the "Attachments" folder in the workspace
3. No attachments exist in the folder.
4. Run the same process using version 3.0.85. Attachments will download into the folder.

Tool version

• jira-azuredevops-migrator-3.0.186

Attachments

Please attach the following files:

config-agile.json: { "source-project": "BAU RC200", "target-project": "BAU IQT3", "query": "project=BAURC ORDER BY Rank ASC", "using-jira-cloud": false, "workspace": "C:\Temp\JiraExport\", "epic-link-field": "Epic Link", "sprint-field": "Sprint", "download-options": 2, "batch-size": 20, "log-level": "Debug", "attachment-folder": "Attachments", "user-mapping-file": "users.txt", "base-area-path": "Migrated", "base-iteration-path": "Migrated", "ignore-failed-links": true, "include-link-comments": false, "process-template": "Agile", "link-map": { "link": [ { "source": "Epic", "target": "System.LinkTypes.Hierarchy-Reverse" }, { "source": "Parent", "target": "System.LinkTypes.Hierarchy-Reverse" }, { "source": "Relates", "target": "System.LinkTypes.Related" }, { "source": "Duplicate", "target": "System.LinkTypes.Duplicate-Forward" }, { "source": "Child", "target": "System.LinkTypes.Hierarchy-Forward" } ] }, "type-map": { "type": [ { "source": "Epic", "target": "Epic" }, { "source": "Story", "target": "Feature" }, { "source": "Bug", "target": "Bug" }, { "source": "Story", "target": "User Story" }, { "source": "Sub-task", "target": "Task" }, { "source": "Certification Task", "target": "Task" }, { "source": "Task", "target": "Task" }, { "source": "Lab Request", "target": "Task" }, { "source": "Review", "target": "Task" }, { "source": "Test Request", "target": "Task" } ] }, "field-map": { "field": [ { "source": "summary", "target": "System.Title", "mapper": "MapTitle" }, { "source": "assignee", "target": "System.AssignedTo", "mapper": "MapUser" }, { "source": "description", "target": "System.Description", "mapper": "MapRendered" }, { "source": "priority", "target": "Microsoft.VSTS.Common.Priority", "for": "Feature,Epic,User Story,Bug,Task", "mapping": { "values": [ { "source": "Blocker", "target": "1" }, { "source": "Critical", "target": "2" }, { "source": "Major", "target": "3" }, { "source": "Must have", "target": "3" }, { "source": "Trivial", "target": "4" }, { "source": "Minor", "target": "4" } ] } }, { "source": "labels", "target": "System.Tags", "mapper": "MapTags" }, { "source": "comment", "target": "System.History", "mapper": "MapRendered" }, { "source": "status", "target": "System.State", "for": "Feature,Epic,User Story,Bug,Task", "mapping": { "values": [ { "source": "Backlog", "target": "New" }, { "source": "Selected for Work", "target": "New" }, { "source": "In Progress", "target": "Active" }, { "source": "On Hold", "target": "Active" }, { "source": "Done", "target": "Closed" }, { "source": "To Do", "target": "New" }, { "source": "Removed", "target": "Removed" } ] } }, { "source": "Story Points", "source-type": "name", "target": "Microsoft.VSTS.Scheduling.StoryPoints", "not-for": "Task" }, { "source": "timeestimate", "target": "Microsoft.VSTS.Scheduling.RemainingWork", "mapper": "MapRemainingWork", "for": "Bug,Task" }, { "source": "description", "target": "Microsoft.VSTS.TCM.ReproSteps", "for": "Bug" } ] } }

Screenshots If applicable, add screenshots to help explain your problem.

Thank you for looking into the bug. Hopefully it's replicatable! Let me know if you need any further details

Cheers

Nick

[RPainter8West]

Same problem here. Version 3.0.85 is a long ways back, so using it isn't a workaround option since it had other problems.

[D][11:53:25]    Downloaded issue: SE-67 changelog.
[D][11:53:25]    No attachment to undo for 'Added 59400/BarryWhite.jpg'.
[D][11:53:25]    Attachment BarryWhite.jpg cannot be migrated because it was deleted.
[D][11:53:25]    No attachment to undo for 'Added 54019/image-2022-08-19-15-09-02-155.png'.
[D][11:53:25]    Attachment image-2022-08-19-15-09-02-155.png cannot be migrated because it was deleted.

In this case, one image was inline in the description, and the other was dragged into JIRA as a file attachment.

[nickn00]

Ah, glad it's not just me! Thanks for the added info @RPainter8West

[Alexander-Hjelm]

Check the History of the Jira Issue. Do you notice anything out of the ordinary? For example, was the attachment removed before it was added, or anything like that?

[RPainter8West]

No these are very simple image additions. In my example, one was pasted into the description field, and the other was dragged on as an attachment. I did this just before doing the export. Probably 10% of my items have at least one image attached. I am not getting any images exported.

[nickn00]

Hi @Alexander-Hjelm, I have a range of attachments including word docs and outlook messages and these haven't been removed and re-added either. None of them will download successfully from a project.

But they do using 3.0.85

[nickn00]

I've just modified the logic and it seems to work as intended:

private static void HandleAttachmentChange(JiraChangeItem item, List<RevisionAction<JiraAttachment>> attachmentChanges, List<JiraAttachment> attachments)
{
    var attachmentChange = TransformAttachmentChange(item);
    if (attachmentChange == null)
        return;

    if (UndoAttachmentChange(attachmentChange, attachments))
    {
        Logger.Log(LogLevel.Debug, $"Attachment {item.ToString ?? item.FromString} cannot be migrated because it was deleted.");
    }
    else
    {
       attachmentChanges.Add(attachmentChange);
    }
}

private static bool UndoAttachmentChange(RevisionAction<JiraAttachment> attachmentChange, List<JiraAttachment> attachments)
{
    if (attachmentChange.ChangeType == RevisionChangeType.Removed)
    {
        Logger.Log(LogLevel.Debug, $"Skipping undo for attachment '{attachmentChange.ToString()}'.");
        return RemoveAttachment(attachmentChange, attachments);
    }
    return false;
}

Now if I add an attachment, it will download it successfully and if I delete the attachment, it will add the relevant comment to the log and not try to download it

[Alexander-Hjelm]

Perfect! Happy to hear that you managed to resolve your problem. I will review these changes as soon as I find the time, and then look at incorporating them into the main line. Feel free to raise a PR if you want to be included as a contributor.

[RPainter8West]

@nickn00 are you putting in a PR for this fix? Or @Alexander-Hjelm do you have an idea when a fix will be available?

[nickn00]

@Alexander-Hjelm @RPainter8West

I've just created PR #867 for review

[Alexander-Hjelm]

Perfect. I will try and find the time to review and test your change this week!

[RPainter8West]

Thanks @nickn00 ! I downloaded your fork, and it solved my problem.

[Alexander-Hjelm]

For me, this fix causes duplicate attachments to appear in the imported item.

I will need to address this separately before merging the PR

[Alexander-Hjelm]

@C-F-Palmer reported issue separately in #952

Have you noticed anything particular in the issue itself in Jira? If you check the history of the concerned issue, does it seem like the attackment has been added in any unconventional way? For example, maybe if you check the order of the history, it could say that the attachment was simply added, swapped for another file, deleted and then re-added, or anything else? I'm trying to think of any edge cases that we possibly haven't covered.

[C-F-Palmer]

Hi Alexander,

There is nothing obvious in the history audit trail that suggest that the attachments were added or re-added in a peculiar way. There are 76 attachments in the issue I am trying to export which have been added between 2008 and 2023 and not one single attachment exported.

I did the export again using version 3.0.85 which I had used for a previous migration and the attachments exported as expected. It seems something has changed between build 85 and 328?

The version of Jira I am migrating is 7.11.

[C-F-Palmer]

@C-F-Palmer reported issue separately in #952

Have you noticed anything particular in the issue itself in Jira? If you check the history of the concerned issue, does it seem like the attackment has been added in any unconventional way? For example, maybe if you check the order of the history, it could say that the attachment was simply added, swapped for another file, deleted and then re-added, or anything else? I'm trying to think of any edge cases that we possibly haven't covered.

Hi ALexander,

Just thinking about this some more and I was wondering if the Attachment Path in Jira could be causing this issue?

In our Jira instance the Attachment Path is set to "Use Custom Directory" instead of the "Use Default Directory" option.

[Alexander-Hjelm]

@C-F-Palmer, that sounds like a plausible culprit, but it would be completely bonkers if changing the Attachment Path changes the behavior of the /issue API.... :/

Could you download Visual Studio (https://visualstudio.microsoft.com/vs/community/) and build the source code? If so, let us schedule a Teams meeting for remote debugging. Maybe we can better identify the problem that way.

You can email me at alexander.hjelm@solidify.dev with some suggestions for meeting times. My time zone is CET.

[C-F-Palmer]

Hi Alexander,

I am going to do one more test before moving onto your suggestion above to debug the problem. As you see from the images below our settings for attachments seem to vary from screen to screen in Jira: one more test before moving onto your suggestion above to debug the problem. As you see from the images below our settings for attachments seem to vary from screen to screen in Jira:

From the System screen in Jira:

You can see that the attachment path states "C:\Program Files\JIRA_attachments" which I believe at some point in time our IT Team created when upgrading or migrating servers.

When you "Edit Settings" on this screen you get the following dialog opening:

As you see the "Attachment Path" is set to "Use Custom Directory" and is pointing to a completely different path. It is also disabled and can't be amended. The other option "Use Default Directory" is enabled but this path does not match the path we are using as shown in the first attachment.

So I will populate the "Default Directory" path with attachments and see if that solves the problem.

[C-F-Palmer]

Hi Alexander,

What I have stated above has not worked. This kind of makes sense as the attachments do come across in the previous version 3.0.85. Below are extracts from the Export.json files from each version and the Export Jira Logs:

LIC-1.json (version 3.0.85) works

{ "Author": "licensing@ikonscience.com", "Time": "2009-04-27T16:19:20+01:00", "Index": 14, "Fields": [], "Links": [], "Attachments": [ { "Change": 0, "FileName": "ISL_1104.xls", "Comment": "Imported from Jira", "AttOriginId": "10273", "FilePath": "C:\Jira Export 3.0.85\Attachments\10273\ISL_1104.xls" } ], "AttachmentReferences": true },

JIra Export Log extract:

[D][11:15:12] Link with description 'This issue Is Replaced by LIC-221' is either not found or this issue (LIC-1) is not inward issue. [D][11:15:12] No attachment to undo for 'Added 10273/ISL_1104.xls'. [D][11:15:12] No attachment to undo for 'Added 10035/Xxxxxxx Quote.29-06-2008.pdf'. [D][11:15:12] Link with description 'This issue Is Replaced by LIC-2' is either not found or this issue (LIC-1) is not inward issue.

LIC-1.json (version 3.0.334) doesn't work

{ "Author": "licensing@ikonscience.com", "Time": "2009-04-27T16:19:20+01:00", "Index": 14, "Fields": [], "Links": [], "Attachments": [], "Commit": null, "AttachmentReferences": false },

Jira Export Log extract:

[D][11:07:28] Link with description 'This issue Is Replaced by LIC-221' is either not found or this issue (LIC-1) is not inward issue. [D][11:07:28] No attachment to undo for 'Added 10273/ISL_1104.xls'. [D][11:07:28] Attachment ISL_1104.xls cannot be migrated because it was deleted. [D][11:07:28] No attachment to undo for 'Added 10035/Xxxxxxx Quote.29-06-2008.pdf'. [D][11:07:28] Attachment Xxxxxxx Quote.29-06-2008.pdf cannot be migrated because it was deleted. [D][11:07:28] Link with description 'This issue Is Replaced by LIC-2' is either not found or this issue (LIC-1) is not inward issue.

[Alexander-Hjelm]

@C-F-Palmer Ok! That's really interesting. It seems that the recommendation nowadays is to always use the default path. Not sure why, if it has something to do with managing folder permissions and ensuring that no Jira Server data is written to disk outside of the default folder, where user file permissions could be less restricted...

Try it and let me know how it goes.

[C-F-Palmer]

.

Hi Alexander,

What I have stated above has not worked. This kind of makes sense as the attachments do come across in the previous version 3.0.85. Below are extracts from the Export.json files from each version and the Export Jira Logs:

LIC-1.json (version 3.0.85) works

{ "Author": "licensing@ikonscience.com", "Time": "2009-04-27T16:19:20+01:00", "Index": 14, "Fields": [], "Links": [], "Attachments": [ { "Change": 0, "FileName": "ISL_1104.xls", "Comment": "Imported from Jira", "AttOriginId": "10273", "FilePath": "C:\Jira Export 3.0.85\Attachments\10273\ISL_1104.xls" } ], "AttachmentReferences": true },

JIra Export Log extract:

[D][11:15:12] Link with description 'This issue Is Replaced by LIC-221' is either not found or this issue (LIC-1) is not inward issue. [D][11:15:12] No attachment to undo for 'Added 10273/ISL_1104.xls'. [D][11:15:12] No attachment to undo for 'Added 10035/Xxxxxxx Quote.29-06-2008.pdf'. [D][11:15:12] Link with description 'This issue Is Replaced by LIC-2' is either not found or this issue (LIC-1) is not inward issue.

LIC-1.json (version 3.0.334) doesn't work

{ "Author": "licensing@ikonscience.com", "Time": "2009-04-27T16:19:20+01:00", "Index": 14, "Fields": [], "Links": [], "Attachments": [], "Commit": null, "AttachmentReferences": false },

Jira Export Log extract:

[D][11:07:28] Link with description 'This issue Is Replaced by LIC-221' is either not found or this issue (LIC-1) is not inward issue. [D][11:07:28] No attachment to undo for 'Added 10273/ISL_1104.xls'. [D][11:07:28] Attachment ISL_1104.xls cannot be migrated because it was deleted. [D][11:07:28] No attachment to undo for 'Added 10035/Xxxxxxx Quote.29-06-2008.pdf'. [D][11:07:28] Attachment Xxxxxxx Quote.29-06-2008.pdf cannot be migrated because it was deleted. [D][11:07:28] Link with description 'This issue Is Replaced by LIC-2' is either not found or this issue (LIC-1) is not inward issue.

[Alexander-Hjelm]

Does the suggested fix by @TKallem work for either of you, by any chance? See https://github.com/solidify/jira-azuredevops-migrator/issues/1049#issuecomment-2158435878

@C-F-Palmer , @nickn00 , @RPainter8West

[RPainter8West]

Sorry @Alexander-Hjelm, I haven't had a need to keep up with this issue. In Oct 2023 I used a fork of this project by @nickn00 that sorted the problem out for me. I only needed to do this transition one time.

[C-F-Palmer]

Hi Alexander,

The suggested fix by TKallem seems to have sorted this problem. I just ran an export on version 3.0.420 with the setting in Jira set to "Show" and the attachments appeared in the Export folder as expected. When I changed this setting back to "Hide" in Jira and run the same export again the "Attachments" folder did not appear in the Export folder and the log stated the error "cannot be migrated as file was deleted".

However this does not explain why the attachments export correctly in version 3.0.85 with the Jira setting at "Hide" and not in any version after this? This may be where the issue was introduced.

Thanks

Chris

[Alexander-Hjelm]

@C-F-Palmer Agreed! In that case it seems strange that v3.0.85 would function just fine even for hidden attachments.

Still, I think that this is a very important finding. Thank you for the help with checking. I will continue investigating on the side, as my time allows!