atlassian-api / atlassian-python-api

Atlassian Python REST API wrapper
https://atlassian-python-api.readthedocs.io
Apache License 2.0
1.34k stars 662 forks source link

endless loop when retrieving projects from the cloud #1310

Closed ebaschiera closed 6 months ago

ebaschiera commented 8 months ago

Hi, we have a pretty large list of projects, and when we retrieve the project list from the cloud, and the list is paginated, the library gets stuck in a loop. It looks like projects["nextPage"] is still the same at every iteration, so the url to download the list from is always the same: https://company.atlassian.net/rest/api/2/project/search?maxResults=50&startAt=50

I was expecting a startAt=100, then startAt=150, but it's always 50.

Am I missing something? I am not a Python expert, I just observed this strange behaviour

Thanks!

ebaschiera commented 8 months ago

Maybe something like this?

        projects = self.paginated_projects(
            included_archived=included_archived,
            expand=expand,
        )
        is_last_page = projects.get("isLast")
        next_page_url = projects.get("nextPage")
        while not is_last_page:
            next_page_projects = self.paginated_projects(
                    included_archived=included_archived,
                    expand=expand,
                    url=next_page_url,
                )
            next_page_url = next_page_projects.get("nextPage")
            is_last_page = next_page_projects.get("isLast")
            projects["values"].extend(
                next_page_projects["values"]
            )
        return projects["values"]
marmur1976 commented 7 months ago

I experienced this behavior too, this is a pretty bad bug. I don't think it is that unusual to have more than 50 projects. Rolled back to previous version for now.

BioSehnsucht commented 7 months ago

I've just run into this myself, we have just a bit over 50 projects currently and trying to get projects from Cloud never returns with the current release.

I've pinned the version before the PR was merged (atlassian-python-api==3.41.3) and was able to retrieve the project list. So far I haven't run into issues with using an older version, but it would be nice if this was fixed to actually work without needing to do this, as it's possible we'll run into a problem at some point.

Spacetown commented 7 months ago

The usage of the paging in Jira is done in a different way than in . There is an issue from Atlassian still open which looks the same and has already a workaround implemented for BitBucket:

https://github.com/atlassian-api/atlassian-python-api/blob/0a5bc5ee72427be0ec16ce5db4f4b091f7a00ddc/atlassian/bitbucket/cloud/base.py#L85-L86

This code should be copied to Jira module and adapted there.

Spacetown commented 6 months ago

Can someone test https://github.com/Spacetown/atlassian-python-api/tree/jira_add_get_paged?