Getting: KeyError: 'attachment'

[anggegavino17]

Hi Prince,

First of all, thank you for this code. I have tested it on a test project and it is working fine. However, on a specific project with thousands of issues, I keep on getting : KeyError: 'attachment'

I checked my permission on the project on both Server and Cloud already. Do you know the possible reason for this? Thank you

[princenyeche]

Hey @anggegavino17

This behaviour occurs because the attachment key in the payload isn't present in an issue that's supposed to have an attachment. You can fix it using the code below to get all the project attachments.

• Authentication file e.g. config.json

  {
  "user": "youremail",
  "password": "yourpassword",
  "url": "<serverurl>"
  }

• Python file

  
  from jiraone import (
  LOGIN,
  PROJECT,
  endpoint,
  issue_export,
  path_builder,
  file_reader,
  file_writer,
  )
  from jiraone.utils import process_executor
  from datetime import datetime
  import json

CONFIG = json.load(open("config.json")) LOGIN.api = False # if on a Server environment LOGIN(**CONFIG)

if name == "main": jql = 'project = "ABC" AND attachments is NOT EMPTY ORDER BY created DESC' count = 0 jira_issue = LOGIN.get(endpoint.search_issues_jql(query=jql)) attach_list = [] total = 0

def issue_pull():
    global count, jira_issue, total
    while True:
        if jira_issue.status_code < 300:
            issues = jira_issue.json()
            total = issues["total"]
            for issue in issues["issues"]:
                keys = issue["key"]
                attach_names = "attachment_{}.csv".format(datetime.now())
                process_executor(
                    PROJECT.get_attachments_on_projects,
                    data="Attachment",
                    query="key = '{}'".format(keys),
                    attachment_file_name=attach_names,
                    workers=7,
                )
                attach_list.append(attach_names)
        count += 50
        jira_issue = LOGIN.get(
            endpoint.search_issues_jql(query=jql, start_at=count)
        )
        if count > total:
            break

issue_pull()
issue_export(
    folder="Attachment",
    check_auth=False,
    merge_files=attach_list,
    final_file="Export-attachments",
    fields=[],
)

reader = file_reader("Attachment",
                     file_name="Export-attachments.csv")
headers = reader[0]
counter = 0
del reader[0]
for row in reader:
    if row[4].startswith("Total Size"):
        del reader[counter]
    counter += 1

file_writer(
    "Attachment",
    file_name="Export-attachments2.csv",
    data=[headers],
    mark="many",
    mode="w+",
)
file_writer(
    "Attachment",
    file_name="Export-attachments2.csv",
    data=reader,
    mark="many",
    mode="a+",
)

[anggegavino17]

Hi Prince,

I didn't thought that you will really respond on this. Thankfully you seem to be an active user here.

Did I understand it correctly that I need to add 2 files?

1. json file for the server credentials
2. py for getting server attachments

Once done, do I need to run 2 python scripts?

1. py for getting server attachments (one you have just provided)
2. afterwards, the original attachment_transfer script to transfer to cloud

[anggegavino17]

Additionally, Here's the current setup I have:

config.json - contains the:

{ "user": "youremail", #is this supposed to be username "password": "yourpassword", "url": "<serverurl>" }

then attachment_get.py - contains the python script you have added above.

When I run python attachment_get.py I am getting: OSError: [Errno 22] Invalid argument: 'C:\Users\xxx\Downloads\S2C-main - ABC\Attachment\attachment_2024-06-22 05:51:04.280937.csv'

[princenyeche]

Hey @anggegavino17

The login arguments are correct.

I believe that error is referring to your file path. It seems as though it is not valid. I believe it could be because of this space "S2C-main - ABC". Can you make sure that folder doesn't have any space characters?

To your other questions: After you get this data, edit the original Python script to just move the attachment to the destination site. You don't need to run the extraction again. That is, you should comment out the part that runs the extraction.

[anggegavino17]

Thanks @princenyeche

Looks like the issue was due to the fact that windows doesn't allow a filename with ':' So the filename with date and time just had to be reformatted to use '-' instead.

I tried to extract now attachments from a project with 12k issues having attachments. It run for around 5 hours and ended with max retries error / remote disconnected. So I'm planning to narrow down the JQL even further and do it by batch instead. Hopefully it will work that way.

Thanks for all your help in this issue

[caeliablesh]

Hi @princenyeche,

Thank you for this code. I was able to transfer a few of our attachments to Cloud. I'm experiencing the same issue with Angge. I'm having an issue in the get attachment part.

It is creating an attachment file in the Attachment folder but the .csv file is empty. Would you know what is the possible issue on this?

[princenyeche]

Hey @caeliablesh

You seem to be running into the same problem, did you try the code below as a workaround that should fix the problem? Given that the attachment key might not be present, using an asynchronous request, if an exception occurs as keyError then only that single issue is ignored and we know that there isn't any real attachment there. The code below also solves the file access issue with ":" character if you're on windows.

Hey @anggegavino17

This behaviour occurs because the attachment key in the payload isn't present in an issue that's supposed to have an attachment. You can fix it using the code below to get all the project attachments.

• Authentication file e.g. config.json

{
"user": "youremail",
"password": "yourpassword",
"url": "<serverurl>"
}

• Python file e.g myscript.py

from jiraone import (
    LOGIN,
    PROJECT,
    endpoint,
    issue_export,
    path_builder,
    file_reader,
    file_writer,
)
from jiraone.utils import process_executor
from datetime import datetime
import json

CONFIG = json.load(open("config.json"))
LOGIN.api = False # if on a Server environment
LOGIN(**CONFIG)

if __name__ == "__main__":
    jql = 'project = "ABC" AND attachments is NOT EMPTY ORDER BY created DESC'
    count = 0
    jira_issue = LOGIN.get(endpoint.search_issues_jql(query=jql))
    attach_list = []
    total = 0

    def issue_pull():
        global count, jira_issue, total
        while True:
            if jira_issue.status_code < 300:
                issues = jira_issue.json()
                total = issues["total"]
                for issue in issues["issues"]:
                    keys = issue["key"]
                    attach_names = "attachment_{}.csv".format(datetime.now().strftime("%Y-%m-%dT%H-%M-%S.%f%z"))
                    process_executor(
                        PROJECT.get_attachments_on_projects,
                        data="Attachment",
                        query="key = '{}'".format(keys),
                        attachment_file_name=attach_names,
                        workers=7,
                    )
                    attach_list.append(attach_names)
            count += 50
            jira_issue = LOGIN.get(
                endpoint.search_issues_jql(query=jql, start_at=count)
            )
            if count > total:
                break

    issue_pull()
    issue_export(
        folder="Attachment",
        check_auth=False,
        merge_files=attach_list,
        final_file="Export-attachments",
        fields=[],
    )

    reader = file_reader("Attachment",
                         file_name="Export-attachments.csv")
    headers = reader[0]
    counter = 0
    del reader[0]
    for row in reader:
        if row[4].startswith("Total Size"):
            del reader[counter]
        counter += 1

    file_writer(
        "Attachment",
        file_name="Export-attachments2.csv",
        data=[headers],
        mark="many",
        mode="w+",
    )
    file_writer(
        "Attachment",
        file_name="Export-attachments2.csv",
        data=reader,
        mark="many",
        mode="a+",
    )